[PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets external tools provide hunks to git's diff and blame pipelines.

Over the past 18 years, git's diff pipeline accumulated many features that
operate on hunks: word diff, function context, color-moved, indent
heuristic, blame. External tools can replace the pipeline entirely
(diff.<driver>.command) or select among builtin algorithms
(diff.<driver>.algorithm), but there is no way for a tool to provide
line-change information into the pipeline. Tools that understand code
structure (tree-sitter parsers, format-aware analyzers, tools like
Difftastic and Mergiraf) must bypass git's pipeline and lose access to
everything downstream.

The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
capability negotiation, one tool invocation per git command. The tool
receives file pairs and returns hunk descriptors that git feeds into the
standard xdiff pipeline. All output features work normally.

Zero hunks with status=success means the tool considers the files
equivalent. git diff shows no output for the file, and git blame skips the
commit, attributing lines to earlier commits.

On error or tool crash, git falls back silently to the builtin diff
algorithm. The feature is opt-in via diff.<driver>.process and
.gitattributes; unconfigured files are unaffected.

The series includes git diff-process-normalize, a built-in tool that
compares files line by line ignoring whitespace (same logic as "git diff -w"
via xdiff_compare_lines):

[diff "cdiff"]
    process = git diff-process-normalize


A whitespace-only boolean flag could serve this specific case. The
subprocess protocol is more general, allowing any tool to participate
without further changes to git.

Michael Montalbo (5):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  diff: add long-running diff process via diff.<driver>.process
  blame: consult diff process for zero-hunk detection
  diff-process-normalize: add built-in whitespace normalizer

 Documentation/config/diff.adoc   |  10 +
 Documentation/gitattributes.adoc |  58 +++++
 Makefile                         |   2 +
 blame.c                          |  43 +++-
 builtin.h                        |   1 +
 builtin/diff-process-normalize.c | 143 ++++++++++
 diff-process.c                   | 203 +++++++++++++++
 diff-process.h                   |  28 ++
 diff.c                           |  25 ++
 git.c                            |   1 +
 t/t4080-diff-process.sh          | 430 +++++++++++++++++++++++++++++++
 userdiff.c                       |   7 +
 userdiff.h                       |   2 +
 xdiff-interface.c                |   7 +-
 xdiff/xdiff.h                    |  13 +
 xdiff/xdiffi.c                   |  98 ++++++-
 16 files changed, 1063 insertions(+), 8 deletions(-)
 create mode 100644 builtin/diff-process-normalize.c
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh


base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
-- 
gitgitgadget

[PATCH 1/5] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 13 +++++++
 xdiff/xdiffi.c    | 98 +++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 114 insertions(+), 4 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index f043330f2a..9542c0bcc2 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..2ee6f1aae3 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based, matching unified diff convention.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +97,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm. */
+	const struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index 5455b4690d..7eca4ab4a1 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,108 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    const struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long total_old = 0, total_new = 0;
+
+	/*
+	 * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
+	 * them via xdl_cleanup_records().  The allocation is nrec + 2
+	 * elements; changed points one past the start (see xprepare.c).
+	 */
+	memset(xe->xdf1.changed - 1, 0,
+	       (xe->xdf1.nrec + 2) * sizeof(bool));
+	memset(xe->xdf2.changed - 1, 0,
+	       (xe->xdf2.nrec + 2) * sizeof(bool));
+
+	for (i = 0; i < nr_hunks; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0)
+			return -1;
+
+		/* Bounds check (1-based line numbers) */
+		if (h->old_count > 0 &&
+		    (h->old_start < 1 ||
+		     h->old_start + h->old_count - 1 > xe->xdf1.nrec))
+			return -1;
+		if (h->new_count > 0 &&
+		    (h->new_start < 1 ||
+		     h->new_start + h->new_count - 1 > xe->xdf2.nrec))
+			return -1;
+
+		/* Zero-count hunks: start must still be in [1, nrec+1] */
+		if (h->old_count == 0 &&
+		    (h->old_start < 1 || h->old_start > xe->xdf1.nrec + 1))
+			return -1;
+		if (h->new_count == 0 &&
+		    (h->new_start < 1 || h->new_start > xe->xdf2.nrec + 1))
+			return -1;
+
+		/* Ordering: no overlap with previous hunk */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+		total_old += h->old_count;
+		total_new += h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 */
+	if ((long)xe->xdf1.nrec - total_old !=
+	    (long)xe->xdf2.nrec - total_new)
+		return -1;
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
-
-		return -1;
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) < 0) {
+			/*
+			 * Invalid external hunks; fall back to the
+			 * builtin diff algorithm.  Re-runs
+			 * xdl_prepare_env() via xdl_do_diff().
+			 */
+			xdl_free_env(&xe);
+			if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
+				return -1;
+		}
+	} else {
+		if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
+			return -1;
 	}
+
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
-- 
gitgitgadget

[PATCH 2/5] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add a new per-driver configuration key that specifies the command
for a long-running diff process.

The name follows filter.<driver>.process: a long-running subprocess
that stays alive across files within a single git invocation.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index fe710a68bf..81c0bebcce 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH 3/5] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool receives file pairs and returns hunks describing which
lines changed.  Git feeds these hunks into the standard xdiff
pipeline, so all output features (word diff, function context,
color) work normally.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet.  On error, git falls back to the builtin diff algorithm.

Zero hunks with status=success means the tool considers the files
equivalent.  Git skips diff output for that file entirely.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   8 +
 Documentation/gitattributes.adoc |  40 ++++
 Makefile                         |   1 +
 diff-process.c                   | 203 +++++++++++++++++++
 diff-process.h                   |  28 +++
 diff.c                           |  25 +++
 t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
 7 files changed, 643 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..4ab5f60df6 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,14 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process.
+	The tool communicates via the pkt-line protocol and returns
+	hunks that are fed into Git's diff and blame pipelines.
+	If the tool returns zero hunks, the file is treated as
+	unchanged for both diff output and blame attribution.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f20041a323..cc724f8c63 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,46 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An external tool can provide content-aware line matching by
+setting `diff.<name>.process` to the command that runs
+the tool.  The tool is a long-running process that communicates via
+the pkt-line protocol (see
+linkgit:gitprotocol-long-running-process[5]).
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+The tool receives file pairs and returns hunk descriptors indicating
+which lines changed.  Git feeds these hunks into its standard diff
+pipeline, so all output features (word diff, function context,
+color) work normally.
+
+If the tool fails or returns an error, Git silently falls back to
+the builtin diff algorithm.  If the tool returns invalid hunks
+(out of bounds, overlapping), Git also falls back silently.
+
+The handshake negotiates `version=1` and `capability=hunks`.
+Per-file requests send `command=hunks` and `pathname=<path>`,
+followed by the old and new file content as packetized data.
+The tool responds with lines of the form
+`hunk <old_start> <old_count> <new_start> <new_count>`
+(1-based line numbers), a flush packet, and `status=success`.
+
+If the tool returns zero hunks with `status=success`, Git treats
+the file as having no changes and produces no diff output.
+
+Tools should ignore unknown keys in the per-file request to
+remain forward-compatible.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index cedc234173..22900368dd 100644
--- a/Makefile
+++ b/Makefile
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..7b0f0e1f7e
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,203 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain content-aware hunks.
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * Zero hunks with status=success means the tool considers the
+ * files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int subprocess_map_initialized;
+static struct hashmap subprocess_map;
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		(struct diff_subprocess *)subprocess;
+
+	/* Uses dying pkt-line variant, same as convert.c filters. */
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *find_or_start_process(const char *cmd)
+{
+	struct diff_subprocess *entry;
+
+	if (!subprocess_map_initialized) {
+		subprocess_map_initialized = 1;
+		hashmap_init(&subprocess_map, cmd2process_cmp, NULL, 0);
+	}
+
+	entry = (struct diff_subprocess *)
+		subprocess_find_entry(&subprocess_map, cmd);
+	if (entry)
+		return entry;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start(&subprocess_map, &entry->subprocess,
+			     cmd, start_diff_process_fn)) {
+		free(entry);
+		return NULL;
+	}
+
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret;
+
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	else
+		ret = 0;
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	hunk->old_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->old_count = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_count = strtol(line, &end, 10);
+	if (end == line || *end != '\0')
+		return -1;
+
+	return 0;
+}
+
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	if (!drv || !drv->process)
+		return -1;
+
+	backend = find_or_start_process(drv->process);
+	if (!backend)
+		return -1;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return -1;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto error;
+
+	if (strcmp(status.buf, "success")) {
+		if (!strcmp(status.buf, "abort"))
+			backend->supported_capabilities &= ~CAP_HUNKS;
+		goto error;
+	}
+
+	*hunks_out = hunks;
+	*nr_hunks_out = nr_hunks;
+	strbuf_release(&status);
+	return 0;
+
+error:
+	free(hunks);
+	strbuf_release(&status);
+	return -1;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..4c84951e02
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,28 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+struct userdiff_driver;
+struct xdl_hunk;
+
+/*
+ * Query a diff process for hunks describing the changes
+ * between old_buf and new_buf.
+ *
+ * The backend is a long-running subprocess configured via
+ * diff.<driver>.process.  It receives file content via
+ * pkt-line and returns hunks with 1-based line numbers.
+ *
+ * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
+ * array (caller must free) and returns 0.
+ *
+ * On failure, returns -1.  The caller should fall back to the
+ * builtin diff algorithm.
+ */
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 397e38b41c..c5e7c329b2 100644
--- a/diff.c
+++ b/diff.c
@@ -25,7 +25,9 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
+#include "trace2.h"
 #include "hashmap.h"
 #include "mem-pool.h"
 #include "merge-ll.h"
@@ -3991,6 +3993,7 @@ static void builtin_diff(const char *name_a,
 		xpparam_t xpp;
 		xdemitconf_t xecfg;
 		struct emit_callback ecbdata;
+		struct xdl_hunk *ext_hunks = NULL;
 		unsigned ws_rule;
 		const struct userdiff_funcname *pe;
 
@@ -4031,6 +4034,27 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (!o->ignore_driver_algorithm &&
+		    one->driver && one->driver->process) {
+			size_t ext_hunks_nr = 0;
+			if (!diff_process_get_hunks(
+				    one->driver, name_a,
+				    mf1.ptr, mf1.size,
+				    mf2.ptr, mf2.size,
+				    &ext_hunks, &ext_hunks_nr)) {
+				if (!ext_hunks_nr)
+					goto free_ab_and_return;
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = ext_hunks_nr;
+			} else {
+				trace2_data_string("diff",
+						   o->repo,
+						   "diff-process-fallback",
+						   name_a);
+			}
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4111,6 +4135,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(ext_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..6f49f4e66b
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,338 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+if test_have_prereq PYTHON
+then
+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+fi
+
+#
+# A single parametric diff process.
+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
+#
+# Modes:
+#   whole-file  - report all lines as changed (default)
+#   fixed-hunk  - always report hunk 5 2 5 2
+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
+#   zero-hunk   - return zero hunks (files considered equivalent)
+#   error       - return status=error for every request
+#   abort       - return status=abort for every request
+#   crash       - read one request then exit without responding
+#
+setup_backend () {
+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
+	import sys, os
+
+	def read_pkt():
+	    hdr = sys.stdin.buffer.read(4)
+	    if len(hdr) < 4: return None
+	    length = int(hdr, 16)
+	    if length == 0: return ""
+	    data = sys.stdin.buffer.read(length - 4)
+	    return data.decode().rstrip("\n")
+
+	def write_pkt(line):
+	    data = (line + "\n").encode()
+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    sys.stdout.buffer.flush()
+
+	def write_flush():
+	    sys.stdout.buffer.write(b"0000")
+	    sys.stdout.buffer.flush()
+
+	def read_content():
+	    chunks = []
+	    while True:
+	        hdr = sys.stdin.buffer.read(4)
+	        if len(hdr) < 4: break
+	        length = int(hdr, 16)
+	        if length == 0: break
+	        chunks.append(sys.stdin.buffer.read(length - 4))
+	    return b"".join(chunks)
+
+	mode = "whole-file"
+	logfile = None
+	for arg in sys.argv[1:]:
+	    if arg.startswith("--mode="):
+	        mode = arg[7:]
+	    elif arg.startswith("--log="):
+	        logfile = open(arg[6:], "a")
+
+	def log(msg):
+	    if logfile:
+	        logfile.write(msg + "\n")
+	        logfile.flush()
+
+	# Handshake
+	assert read_pkt() == "git-diff-client"
+	assert read_pkt() == "version=1"
+	read_pkt()
+	write_pkt("git-diff-server")
+	write_pkt("version=1")
+	write_flush()
+	while True:
+	    p = read_pkt()
+	    if p == "": break
+	write_pkt("capability=hunks")
+	write_flush()
+
+	log("ready")
+
+	while True:
+	    cmd = None
+	    pathname = None
+	    while True:
+	        p = read_pkt()
+	        if p is None: sys.exit(0)
+	        if p == "": break
+	        if p.startswith("command="): cmd = p.split("=",1)[1]
+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
+	    if cmd is None: sys.exit(0)
+	    old = read_content()
+	    new = read_content()
+	    log(f"command={cmd} pathname={pathname}")
+
+	    if mode == "error":
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	        continue
+
+	    if mode == "abort":
+	        write_flush()
+	        write_pkt("status=abort")
+	        write_flush()
+	        continue
+
+	    if mode == "crash":
+	        sys.exit(1)
+
+	    if cmd == "hunks":
+	        if mode == "fixed-hunk":
+	            write_pkt("hunk 5 2 5 2")
+	        elif mode == "bad-hunk":
+	            write_pkt("hunk 999 1 999 1")
+	        elif mode == "zero-hunk":
+	            pass
+	        else:
+	            ol = len(old.split(b"\n"))
+	            nl = len(new.split(b"\n"))
+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	        write_flush()
+	        write_pkt("status=success")
+	        write_flush()
+	    else:
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	PYEOF
+	write_script diff-process-backend <<-SHEOF
+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
+	SHEOF
+}
+
+BACKEND="./diff-process-backend"
+
+test_expect_success PYTHON 'setup' '
+	setup_backend &&
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+	git commit -m "initial"
+'
+
+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+	git commit -m "add boundary.c" &&
+
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	grep "^-OLD5" actual &&
+	grep "^-OLD6" actual &&
+	grep "^+NEW5" actual &&
+	grep "^+NEW6" actual &&
+	! grep "^-OLD9" actual &&
+	! grep "^-OLD10" actual &&
+	! grep "^+NEW9" actual &&
+	! grep "^+NEW10" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool error status' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual &&
+	# Fallback produces the full builtin diff (both change regions).
+	grep "^-OLD5" actual &&
+	grep "^+NEW5" actual &&
+	grep "^-OLD9" actual &&
+	grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	grep "command=hunks pathname=boundary.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual &&
+	grep "^-OLD5" actual &&
+	grep "^+NEW5" actual &&
+	grep "^-OLD9" actual &&
+	grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual &&
+	grep "^-OLD5" actual &&
+	grep "^+NEW5" actual &&
+	grep "^-OLD9" actual &&
+	grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process abort disables for session' '
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add abort1.c abort2.c &&
+	git commit -m "add abort files" &&
+
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- abort1.c abort2.c >actual &&
+	# Both files should still produce diff output via fallback.
+	grep "return 10" actual &&
+	grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool,
+	# so the log should have exactly one entry, not two.
+	grep "command=hunks" backend.log >matches &&
+	test_line_count = 1 matches
+'
+
+test_expect_success PYTHON 'diff process handles multiple files' '
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 1; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 2; }
+	EOF
+	git add multi1.c multi2.c &&
+	git commit -m "add multi files" &&
+
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 10; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- multi1.c multi2.c >actual &&
+	grep "return 10" actual &&
+	grep "return 20" actual &&
+	grep "pathname=multi1.c" backend.log &&
+	grep "pathname=multi2.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process with --word-diff' '
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+	git commit -m "add worddiff.c" &&
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND" \
+		diff --word-diff worddiff.c >actual &&
+	grep "\[-1;-\]" actual &&
+	grep "{+999;+}" actual
+'
+
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process works with git log -p' '
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+	git commit -m "add logtest.c" &&
+
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		log -1 -p -- logtest.c >actual &&
+	grep "return 2" actual &&
+	grep "command=hunks pathname=logtest.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add zerohunk.c &&
+	git commit -m "add zerohunk.c" &&
+
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		diff zerohunk.c >actual &&
+	test_must_be_empty actual
+'
+
+test_done
-- 
gitgitgadget

[PATCH 4/5] blame: consult diff process for zero-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns zero hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc |  3 +++
 blame.c                          | 43 +++++++++++++++++++++++++++++---
 t/t4080-diff-process.sh          | 32 ++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index cc724f8c63..7d66fa3aa1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -857,6 +857,9 @@ The tool responds with lines of the form
 
 If the tool returns zero hunks with `status=success`, Git treats
 the file as having no changes and produces no diff output.
+`git blame` also consults the diff process and skips commits
+where it reports zero hunks, attributing lines to earlier commits
+instead.
 
 Tools should ignore unknown keys in the per-file request to
 remain forward-compatible.
diff --git a/blame.c b/blame.c
index a3c49d132e..8a5f14db7a 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "userdiff.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -315,16 +317,47 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data,
+		      int xdl_opts, struct index_state *istate,
+		      const char *path)
 {
 	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
+	struct xdl_hunk *ext_hunks = NULL;
+	int ret;
 
 	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+
+	if (path && istate) {
+		struct userdiff_driver *drv;
+		drv = userdiff_find_by_path(istate, path);
+		if (drv && drv->process) {
+			size_t nr = 0;
+			if (!diff_process_get_hunks(drv, path,
+						    file_a->ptr, file_a->size,
+						    file_b->ptr, file_b->size,
+						    &ext_hunks, &nr)) {
+				if (!nr) {
+					/*
+					 * Zero hunks: the diff process
+					 * considers these files equivalent.
+					 * Skip so blame looks past this
+					 * commit.
+					 */
+					return 0;
+				}
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = nr;
+			}
+		}
+	}
+
+	ret = xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	free(ext_hunks);
+	return ret;
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1961,7 +1994,8 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts,
+		       sb->revs->diffopt.repo->index, target->path))
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
@@ -2114,7 +2148,8 @@ static void find_copy_in_blob(struct blame_scoreboard *sb,
 	 * file_p partially may match that image.
 	 */
 	memset(split, 0, sizeof(struct blame_entry [3]));
-	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts))
+	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts,
+		       NULL, NULL))
 		die("unable to generate diff (%s)",
 		    oid_to_hex(&parent->commit->object.oid));
 	/* remainder, if any, all match the preimage */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 6f49f4e66b..5ed644b786 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -335,4 +335,36 @@ test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
 	test_must_be_empty actual
 '
 
+test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without zero-hunk mode, blame attributes the change.
+	git blame blame.c >without &&
+	grep "$BLAME_COMMIT" without &&
+
+	# With zero-hunk mode, the process considers the files equivalent
+	# and blame skips the reformat commit.
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		blame blame.c >with &&
+	! grep "$BLAME_COMMIT" with
+'
+
+
 test_done
-- 
gitgitgadget

[PATCH 5/5] diff-process-normalize: add built-in whitespace normalizer

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add git diff-process-normalize, a built-in diff process that
detects whitespace-only changes.  It compares files line by line
using xdiff_compare_lines() with XDF_IGNORE_WHITESPACE (same
logic as "git diff -w").  If all lines match, it returns zero
hunks; otherwise it returns an error so git falls back to the
builtin diff algorithm.

    [diff "cdiff"]
        process = git diff-process-normalize

Update documentation to describe zero-hunk behavior for diff
and blame, and document the built-in normalize tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   2 +
 Documentation/gitattributes.adoc |  15 ++++
 Makefile                         |   1 +
 builtin.h                        |   1 +
 builtin/diff-process-normalize.c | 143 +++++++++++++++++++++++++++++++
 git.c                            |   1 +
 t/t4080-diff-process.sh          |  60 +++++++++++++
 7 files changed, 223 insertions(+)
 create mode 100644 builtin/diff-process-normalize.c

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 4ab5f60df6..475736c6ed 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -224,6 +224,8 @@ endif::git-diff[]
 	hunks that are fed into Git's diff and blame pipelines.
 	If the tool returns zero hunks, the file is treated as
 	unchanged for both diff output and blame attribution.
+	Git provides `git diff-process-normalize` as a built-in
+	tool that detects whitespace-only changes.
 	See linkgit:gitattributes[5] for details.
 
 `diff.indentHeuristic`::
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 7d66fa3aa1..3f1d7affd8 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -861,6 +861,21 @@ the file as having no changes and produces no diff output.
 where it reports zero hunks, attributing lines to earlier commits
 instead.
 
+Git ships with a built-in diff process, `git diff-process-normalize`,
+that detects whitespace-only changes.  Files whose only differences
+are whitespace produce zero hunks; files with non-whitespace changes
+fall back to the builtin diff algorithm.  To use it:
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = git diff-process-normalize
+----------------------------------------------------------------
+
+This is useful after running a code formatter: `git diff` shows
+no output for files that only had whitespace changes,
+`git blame` skips whitespace-only commits automatically without
+requiring a `.git-blame-ignore-revs` file.
+
 Tools should ignore unknown keys in the per-file request to
 remain forward-compatible.
 
diff --git a/Makefile b/Makefile
index 22900368dd..01acfaf7b8 100644
--- a/Makefile
+++ b/Makefile
@@ -1409,6 +1409,7 @@ BUILTIN_OBJS += builtin/diagnose.o
 BUILTIN_OBJS += builtin/diff-files.o
 BUILTIN_OBJS += builtin/diff-index.o
 BUILTIN_OBJS += builtin/diff-pairs.o
+BUILTIN_OBJS += builtin/diff-process-normalize.o
 BUILTIN_OBJS += builtin/diff-tree.o
 BUILTIN_OBJS += builtin/diff.o
 BUILTIN_OBJS += builtin/difftool.o
diff --git a/builtin.h b/builtin.h
index 235c51f30e..c713a0417f 100644
--- a/builtin.h
+++ b/builtin.h
@@ -178,6 +178,7 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repos
 int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);
+int cmd_diff_process_normalize(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);
diff --git a/builtin/diff-process-normalize.c b/builtin/diff-process-normalize.c
new file mode 100644
index 0000000000..1580f6b7d9
--- /dev/null
+++ b/builtin/diff-process-normalize.c
@@ -0,0 +1,143 @@
+/*
+ * Built-in diff process that returns zero hunks for files whose
+ * only differences are whitespace, and status=error otherwise.
+ * See diff-process.c for the protocol and gitattributes(5) for usage.
+ *
+ * Uses xdiff_compare_lines() with XDF_IGNORE_WHITESPACE to compare
+ * lines, giving the same whitespace handling as "git diff -w".
+ */
+
+#include "builtin.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff-interface.h"
+
+/*
+ * Read a single pkt-line.  Returns 1 for data, 0 for flush, -1 for EOF.
+ */
+static int read_pkt(int fd, struct strbuf *line)
+{
+	int len;
+	char *data;
+
+	if (packet_read_line_gently(fd, &len, &data) < 0)
+		return -1;
+	if (!data || !len)
+		return 0; /* flush */
+	strbuf_reset(line);
+	strbuf_add(line, data, len);
+	strbuf_rtrim(line);
+	return 1;
+}
+
+/*
+ * Read packetized content until a flush packet.
+ */
+static int read_content(int fd, struct strbuf *out)
+{
+	strbuf_reset(out);
+	if (read_packetized_to_strbuf(fd, out, PACKET_READ_GENTLE_ON_EOF) < 0)
+		return -1;
+	return 0;
+}
+
+/*
+ * Compare two buffers line by line using xdiff_compare_lines() with
+ * XDF_IGNORE_WHITESPACE (same logic as "git diff -w").
+ * Returns 1 if all lines match, 0 otherwise.
+ */
+static int whitespace_equivalent(const char *a, long size_a,
+				 const char *b, long size_b)
+{
+	const char *ea = a + size_a;
+	const char *eb = b + size_b;
+
+	while (a < ea && b < eb) {
+		const char *eol_a = memchr(a, '\n', ea - a);
+		const char *eol_b = memchr(b, '\n', eb - b);
+		long len_a = (eol_a ? eol_a : ea) - a;
+		long len_b = (eol_b ? eol_b : eb) - b;
+
+		if (!xdiff_compare_lines(a, len_a, b, len_b,
+					 XDF_IGNORE_WHITESPACE))
+			return 0;
+
+		a += len_a + (eol_a ? 1 : 0);
+		b += len_b + (eol_b ? 1 : 0);
+	}
+
+	/* Both sides must be exhausted */
+	return a >= ea && b >= eb;
+}
+
+int cmd_diff_process_normalize(int argc UNUSED, const char **argv UNUSED,
+			       const char *prefix UNUSED,
+			       struct repository *repo UNUSED)
+{
+	struct strbuf line = STRBUF_INIT;
+	struct strbuf old_content = STRBUF_INIT;
+	struct strbuf new_content = STRBUF_INIT;
+	int ret;
+
+	/* Handshake: read client greeting */
+	ret = read_pkt(0, &line);
+	if (ret <= 0 || strcmp(line.buf, "git-diff-client"))
+		return 1;
+	ret = read_pkt(0, &line);
+	if (ret <= 0 || strcmp(line.buf, "version=1"))
+		return 1;
+	read_pkt(0, &line); /* flush */
+
+	/* Send server greeting */
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Read client capabilities until flush */
+	while ((ret = read_pkt(0, &line)) > 0)
+		; /* consume */
+
+	/* Send our capabilities */
+	packet_write_fmt(1, "capability=hunks\n");
+	packet_flush(1);
+
+	/* Main loop: process file pairs */
+	for (;;) {
+		int have_command = 0;
+
+		/* Read request headers until flush */
+		while ((ret = read_pkt(0, &line)) > 0) {
+			if (starts_with(line.buf, "command="))
+				have_command = 1;
+		}
+		if (ret < 0)
+			break; /* EOF: client closed connection */
+		if (!have_command)
+			break;
+
+		/* Read old file content */
+		if (read_content(0, &old_content) < 0)
+			break;
+		/* Read new file content */
+		if (read_content(0, &new_content) < 0)
+			break;
+
+		if (whitespace_equivalent(old_content.buf, old_content.len,
+					  new_content.buf, new_content.len)) {
+			/* Whitespace-only differences */
+			packet_flush(1); /* zero hunks */
+			packet_write_fmt(1, "status=success\n");
+			packet_flush(1);
+		} else {
+			/* Non-whitespace differences: fall back */
+			packet_flush(1);
+			packet_write_fmt(1, "status=error\n");
+			packet_flush(1);
+		}
+	}
+
+	strbuf_release(&line);
+	strbuf_release(&old_content);
+	strbuf_release(&new_content);
+	return 0;
+}
diff --git a/git.c b/git.c
index 5a40eab8a2..6239240b02 100644
--- a/git.c
+++ b/git.c
@@ -568,6 +568,7 @@ static struct cmd_struct commands[] = {
 	{ "diff-files", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },
 	{ "diff-index", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },
 	{ "diff-pairs", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },
+	{ "diff-process-normalize", cmd_diff_process_normalize, NO_PARSEOPT },
 	{ "diff-tree", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },
 	{ "difftool", cmd_difftool, RUN_SETUP_GENTLY },
 	{ "fast-export", cmd_fast_export, RUN_SETUP },
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 5ed644b786..a6fa1df456 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -366,5 +366,65 @@ test_expect_success PYTHON 'blame skips commits with zero hunks from diff proces
 	! grep "$BLAME_COMMIT" with
 '
 
+NORMALIZE="git diff-process-normalize"
+
+test_expect_success 'diff-process-normalize setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+	test_commit normalize-base
+'
+
+test_expect_success 'diff-process-normalize suppresses whitespace-only changes' '
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add ws.c &&
+	git commit -m "add ws.c" &&
+
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+
+	git -c diff.cdiff.process="$NORMALIZE" \
+		diff ws.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success 'diff-process-normalize falls back on non-whitespace changes' '
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+
+	int added_function(void)
+	{
+	    return 99;
+	}
+	EOF
+
+	git -c diff.cdiff.process="$NORMALIZE" \
+		diff ws.c >actual &&
+	grep "added_function" actual
+'
+
+test_expect_success 'diff-process-normalize falls back on mixed whitespace and real changes' '
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	        return 42;
+	}
+	EOF
+
+	git -c diff.cdiff.process="$NORMALIZE" \
+		diff ws.c >actual &&
+	grep "return 42" actual
+'
 
 test_done
-- 
gitgitgadget

Re: [PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> This series adds diff.<driver>.process, a long-running subprocess protocol
> that lets external tools provide hunks to git's diff and blame pipelines.
>
> Over the past 18 years, git's diff pipeline accumulated many features that
> operate on hunks: word diff, function context, color-moved, indent
> heuristic, blame. External tools can replace the pipeline entirely
> (diff.<driver>.command) or select among builtin algorithms
> (diff.<driver>.algorithm), but there is no way for a tool to provide
> line-change information into the pipeline. Tools that understand code
> structure (tree-sitter parsers, format-aware analyzers, tools like
> Difftastic and Mergiraf) must bypass git's pipeline and lose access to
> everything downstream.
>
> The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
> capability negotiation, one tool invocation per git command. The tool
> receives file pairs and returns hunk descriptors that git feeds into the
> standard xdiff pipeline. All output features work normally.
>
> Zero hunks with status=success means the tool considers the files
> equivalent. git diff shows no output for the file, and git blame skips the
> commit, attributing lines to earlier commits.
>
> On error or tool crash, git falls back silently to the builtin diff
> algorithm. The feature is opt-in via diff.<driver>.process and
> .gitattributes; unconfigured files are unaffected.
>
> The series includes git diff-process-normalize, a built-in tool that
> compares files line by line ignoring whitespace (same logic as "git diff -w"
> via xdiff_compare_lines):

Interesting.

If the goal is purely to normalize content before comparison
(e.g. stripping comments or canonicalizing formatting), we already
have the `textconv` mechanism.  While `textconv` is a "one-shot"
per-file process, it is significantly simpler.

I suspect, however, that the primary focus here is to allow external
tools to provide structural alignment (e.g. for AST- aware diffs
like Difftastic or Mergiraf) without losing the original content in
the display.  Unlike `textconv`, which transforms the text the user
sees, this protocol lets the display remain identical to the source
while using a custom engine for the line-matching logic.

If that is the intent, it should be stated more explicitly in the
documentation and commit messages.  The "whitespace-normalize"
demonstration in [PATCH 5/5] is misleading because it's exactly the
case where `textconv` would be sufficient.

I am afraid that the use of a long-running subprocess for every
diff/blame invocation adds significant complexity and overhead.  In
particular, wouldn't the `blame` implementation performs a
round-trip to the subprocess for every commit in the history?  Even
with a persistent process, the overhead of serializing and
deserializing the entire file content twice (old and new) for every
commit could be prohibitive for large files or deep histories.

So, I dunno.

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +/*
> + * Populate the changed[] arrays from externally supplied hunks,
> + * bypassing the diff algorithm.  Validates that hunks are in order,
> + * non-overlapping, and within bounds.
> + *
> + * Returns 0 on success, -1 on validation failure.
> + */
> +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
> +					    const struct xdl_hunk *hunks,
> +					    size_t nr_hunks)
> +{
> +	size_t i;
> +	long j, prev_old_end = 0, prev_new_end = 0;
> +	long total_old = 0, total_new = 0;
> +
> +	/*
> +	 * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
> +	 * them via xdl_cleanup_records().  The allocation is nrec + 2
> +	 * elements; changed points one past the start (see xprepare.c).
> +	 */
> +	memset(xe->xdf1.changed - 1, 0,
> +	       (xe->xdf1.nrec + 2) * sizeof(bool));
> +	memset(xe->xdf2.changed - 1, 0,
> +	       (xe->xdf2.nrec + 2) * sizeof(bool));

This, especially the starting offset of -1, looks horrible.  The
internal layout of xdfenv_t might happen to match the way the above
code expects, which is how xdl_prepare_ctx() may have give you, but
it somehow feels brittle.  I guess the assumption that changed[]
does not point at the beginning of the allocated area (e.g., it is a
no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
it cannot be helped.  Sigh.

>  int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
>  	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
>  	xdchange_t *xscr;
>  	xdfenv_t xe;
>  	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
>  
> -	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
> -
> -		return -1;
> +	if (xpp->external_hunks) {
> +		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
> +			return -1;
> +		if (xdl_populate_hunks_from_external(&xe,
> +						     xpp->external_hunks,
> +						     xpp->external_hunks_nr) < 0) {
> +			/*
> +			 * Invalid external hunks; fall back to the
> +			 * builtin diff algorithm.  Re-runs
> +			 * xdl_prepare_env() via xdl_do_diff().
> +			 */
> +			xdl_free_env(&xe);
> +			if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> +				return -1;

If the external tool keeps sending bogus hunks, silently falling
back to what we would have done if there weren't any external stuff
may be necessary to pleasantly keep using Git, but two and a half
short comments here.

 (1) "What we would have done" is exactly the same as what appears
     in the corresponding "else" block.  Can we make sure that we do
     not have to keep updating both copies in the future with some
     code rearrangement?

 (2) The writer of the external tool may want to see some trace of
     warning under certain flags when a failure of the tool forces
     the receiving end to fallback.

 (3) If the tool throws too many broken replies, perhaps we want to
     disable it automatically?

> +		}
> +	} else {
> +		if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> +			return -1;
>  	}
> +
>  	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
>  	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
>  	    xdl_build_script(&xe, &xscr) < 0) {

Re: [PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo]

On Thu, May 21, 2026 at 10:29 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > This series adds diff.<driver>.process, a long-running subprocess protocol
> > that lets external tools provide hunks to git's diff and blame pipelines.
> >
> > Over the past 18 years, git's diff pipeline accumulated many features that
> > operate on hunks: word diff, function context, color-moved, indent
> > heuristic, blame. External tools can replace the pipeline entirely
> > (diff.<driver>.command) or select among builtin algorithms
> > (diff.<driver>.algorithm), but there is no way for a tool to provide
> > line-change information into the pipeline. Tools that understand code
> > structure (tree-sitter parsers, format-aware analyzers, tools like
> > Difftastic and Mergiraf) must bypass git's pipeline and lose access to
> > everything downstream.
> >
> > The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
> > capability negotiation, one tool invocation per git command. The tool
> > receives file pairs and returns hunk descriptors that git feeds into the
> > standard xdiff pipeline. All output features work normally.
> >
> > Zero hunks with status=success means the tool considers the files
> > equivalent. git diff shows no output for the file, and git blame skips the
> > commit, attributing lines to earlier commits.
> >
> > On error or tool crash, git falls back silently to the builtin diff
> > algorithm. The feature is opt-in via diff.<driver>.process and
> > .gitattributes; unconfigured files are unaffected.
> >
> > The series includes git diff-process-normalize, a built-in tool that
> > compares files line by line ignoring whitespace (same logic as "git diff -w"
> > via xdiff_compare_lines):
>
> Interesting.
>
> If the goal is purely to normalize content before comparison
> (e.g. stripping comments or canonicalizing formatting), we already
> have the `textconv` mechanism.  While `textconv` is a "one-shot"
> per-file process, it is significantly simpler.
>
> I suspect, however, that the primary focus here is to allow external
> tools to provide structural alignment (e.g. for AST- aware diffs
> like Difftastic or Mergiraf) without losing the original content in
> the display.  Unlike `textconv`, which transforms the text the user
> sees, this protocol lets the display remain identical to the source
> while using a custom engine for the line-matching logic.
>
> If that is the intent, it should be stated more explicitly in the
> documentation and commit messages.  The "whitespace-normalize"
> demonstration in [PATCH 5/5] is misleading because it's exactly the
> case where `textconv` would be sufficient.
>

Thank you for looking at this.

Yes, you have correctly identified the primary focus. My intention with the
whitespace normalization example was to provide a kind of "hello world"
diff process that would showcase how such a tool could interact with
the pipeline further down (i.e., blame vs diff output). However, I do agree
that it is a confusing example because it seems to clash with something
`textconv` already provides. I will update messaging across the series to
make the true intention of these changes more clear.

> I am afraid that the use of a long-running subprocess for every
> diff/blame invocation adds significant complexity and overhead.  In
> particular, wouldn't the `blame` implementation performs a
> round-trip to the subprocess for every commit in the history?  Even
> with a persistent process, the overhead of serializing and
> deserializing the entire file content twice (old and new) for every
> commit could be prohibitive for large files or deep histories.
>
> So, I dunno.

I hear you on this point. Anecdotally, my measurement of running blame
on diff.c:

Performance:
  without process:  0.57s
  with process:       0.67s

Blame attribution:
  without process:  726 unique commits
  with process:     721 unique commits (5 whitespace-only skipped)

Skipped commits:
  0ea7d5b6f8 diff.c: fix some recent whitespace style violations
  2775d92c53 diff.c: stylefix
  4b25d091ba Fix a bunch of pointer declarations (codestyle)
  a6080a0a44 War on whitespace
  eeefa7c90e Style fixes, add a space after if/for/while.

I was imagining we could potentially optimize performance by extending the
protocol to enable passing OIDs as a capability so tools could read objects
directly without needing any serialization/deserialization.

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Michael Montalbo]

On Thu, May 21, 2026 at 10:29 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +/*
> > + * Populate the changed[] arrays from externally supplied hunks,
> > + * bypassing the diff algorithm.  Validates that hunks are in order,
> > + * non-overlapping, and within bounds.
> > + *
> > + * Returns 0 on success, -1 on validation failure.
> > + */
> > +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
> > +                                         const struct xdl_hunk *hunks,
> > +                                         size_t nr_hunks)
> > +{
> > +     size_t i;
> > +     long j, prev_old_end = 0, prev_new_end = 0;
> > +     long total_old = 0, total_new = 0;
> > +
> > +     /*
> > +      * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
> > +      * them via xdl_cleanup_records().  The allocation is nrec + 2
> > +      * elements; changed points one past the start (see xprepare.c).
> > +      */
> > +     memset(xe->xdf1.changed - 1, 0,
> > +            (xe->xdf1.nrec + 2) * sizeof(bool));
> > +     memset(xe->xdf2.changed - 1, 0,
> > +            (xe->xdf2.nrec + 2) * sizeof(bool));
>
> This, especially the starting offset of -1, looks horrible.  The
> internal layout of xdfenv_t might happen to match the way the above
> code expects, which is how xdl_prepare_ctx() may have give you, but
> it somehow feels brittle.  I guess the assumption that changed[]
> does not point at the beginning of the allocated area (e.g., it is a
> no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
> it cannot be helped.  Sigh.
>

Agreed it is ugly. I wanted to make sure the entire changed[] including
sentinels were clear as a defensive measure for downstream callers
(xdl_change_compact). I agree this results in something that is ugly
and brittle, but in the end I thought it was superior to relying on the
fact that upstream zeroes the entire changed[] array. Maybe if the
comment was more explicit about why this is happening it would be
helpful?

    /*
     * Clear changed[] arrays including sentinels.
     * xdl_prepare_env() may have dirtied them via
     * xdl_cleanup_records(), and xdl_change_compact() reads
     * the sentinel at changed[-1] during backward scans.
     */

> >  int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
> >            xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
> >       xdchange_t *xscr;
> >       xdfenv_t xe;
> >       emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
> >
> > -     if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
> > -
> > -             return -1;
> > +     if (xpp->external_hunks) {
> > +             if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
> > +                     return -1;
> > +             if (xdl_populate_hunks_from_external(&xe,
> > +                                                  xpp->external_hunks,
> > +                                                  xpp->external_hunks_nr) < 0) {
> > +                     /*
> > +                      * Invalid external hunks; fall back to the
> > +                      * builtin diff algorithm.  Re-runs
> > +                      * xdl_prepare_env() via xdl_do_diff().
> > +                      */
> > +                     xdl_free_env(&xe);
> > +                     if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> > +                             return -1;
>
> If the external tool keeps sending bogus hunks, silently falling
> back to what we would have done if there weren't any external stuff
> may be necessary to pleasantly keep using Git, but two and a half
> short comments here.
>
>  (1) "What we would have done" is exactly the same as what appears
>      in the corresponding "else" block.  Can we make sure that we do
>      not have to keep updating both copies in the future with some
>      code rearrangement?
>

How about something like this:

  if (xpp->external_hunks) {
      if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
          return -1;
      if (xdl_populate_hunks_from_external(&xe,
                       xpp->external_hunks,
                       xpp->external_hunks_nr) == 0)
          goto diff_done;
      xdl_free_env(&xe);
  }

  if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
      return -1;

  diff_done:

>  (2) The writer of the external tool may want to see some trace of
>      warning under certain flags when a failure of the tool forces
>      the receiving end to fallback.
>

In diff.c how about we emit a warning rather than a trace on
fallback:

    warning(_("diff process failed for '%s',"
              " falling back to builtin diff"),
            name_a);

>  (3) If the tool throws too many broken replies, perhaps we want to
>      disable it automatically?
>

For the RFC I wanted to keep it simple, but I definitely agree. A configurable
failure policy makes a lot of sense to me (e.g., disable after N failures).

> > +             }
> > +     } else {
> > +             if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> > +                     return -1;
> >       }
> > +
> >       if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
> >           xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
> >           xdl_build_script(&xe, &xscr) < 0) {

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Junio C Hamano]

Michael Montalbo <mmontalbo@gmail.com> writes:

>> > +      * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
>> > +      * them via xdl_cleanup_records().  The allocation is nrec + 2
>> > +      * elements; changed points one past the start (see xprepare.c).
>> > +      */
>> > +     memset(xe->xdf1.changed - 1, 0,
>> > +            (xe->xdf1.nrec + 2) * sizeof(bool));
>> > +     memset(xe->xdf2.changed - 1, 0,
>> > +            (xe->xdf2.nrec + 2) * sizeof(bool));
>>
>> This, especially the starting offset of -1, looks horrible.  The
>> internal layout of xdfenv_t might happen to match the way the above
>> code expects, which is how xdl_prepare_ctx() may have give you, but
>> it somehow feels brittle.  I guess the assumption that changed[]
>> does not point at the beginning of the allocated area (e.g., it is a
>> no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
>> it cannot be helped.  Sigh.
>>
>
> Agreed it is ugly. I wanted to make sure the entire changed[] including
> sentinels were clear as a defensive measure for downstream callers
> (xdl_change_compact). I agree this results in something that is ugly
> and brittle, but in the end I thought it was superior to relying on the
> fact that upstream zeroes the entire changed[] array. Maybe if the
> comment was more explicit about why this is happening it would be
> helpful?

Perhaps make these memset() into calls to a helper function that is
defined in xdiff/xprepare.c with a descriptive name and placed near
where xdl_prepare_ctx() is.  That way, the patch in question does
not even have to expose the strangeness of changed[] (i.e., it has 2
more elements than it would normally contain to make the memory
region for changed[-1] and changed[N] valid, and freeing it requires
free(changed-1)) to the code path.  It only needs to say "Hey, I am
clearing changed[] arrays because of XXX" without having to say "by
the way, the memory layout of changed[] is strange this way", the
latter of which is not exactly of interest for readers of this code.

>     /*
>      * Clear changed[] arrays including sentinels.
>      * xdl_prepare_env() may have dirtied them via
>      * xdl_cleanup_records(), and xdl_change_compact() reads
>      * the sentinel at changed[-1] during backward scans.
>      */

And this belongs in xdiff/xprepare.c near that new helper function.

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Michael Montalbo]

On Sun, May 24, 2026 at 1:50 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Michael Montalbo <mmontalbo@gmail.com> writes:
>
> >> > +      * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
> >> > +      * them via xdl_cleanup_records().  The allocation is nrec + 2
> >> > +      * elements; changed points one past the start (see xprepare.c).
> >> > +      */
> >> > +     memset(xe->xdf1.changed - 1, 0,
> >> > +            (xe->xdf1.nrec + 2) * sizeof(bool));
> >> > +     memset(xe->xdf2.changed - 1, 0,
> >> > +            (xe->xdf2.nrec + 2) * sizeof(bool));
> >>
> >> This, especially the starting offset of -1, looks horrible.  The
> >> internal layout of xdfenv_t might happen to match the way the above
> >> code expects, which is how xdl_prepare_ctx() may have give you, but
> >> it somehow feels brittle.  I guess the assumption that changed[]
> >> does not point at the beginning of the allocated area (e.g., it is a
> >> no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
> >> it cannot be helped.  Sigh.
> >>
> >
> > Agreed it is ugly. I wanted to make sure the entire changed[] including
> > sentinels were clear as a defensive measure for downstream callers
> > (xdl_change_compact). I agree this results in something that is ugly
> > and brittle, but in the end I thought it was superior to relying on the
> > fact that upstream zeroes the entire changed[] array. Maybe if the
> > comment was more explicit about why this is happening it would be
> > helpful?
>
> Perhaps make these memset() into calls to a helper function that is
> defined in xdiff/xprepare.c with a descriptive name and placed near
> where xdl_prepare_ctx() is.  That way, the patch in question does
> not even have to expose the strangeness of changed[] (i.e., it has 2
> more elements than it would normally contain to make the memory
> region for changed[-1] and changed[N] valid, and freeing it requires
> free(changed-1)) to the code path.  It only needs to say "Hey, I am
> clearing changed[] arrays because of XXX" without having to say "by
> the way, the memory layout of changed[] is strange this way", the
> latter of which is not exactly of interest for readers of this code.
>
> >     /*
> >      * Clear changed[] arrays including sentinels.
> >      * xdl_prepare_env() may have dirtied them via
> >      * xdl_cleanup_records(), and xdl_change_compact() reads
> >      * the sentinel at changed[-1] during backward scans.
> >      */
>
> And this belongs in xdiff/xprepare.c near that new helper function.

That sounds a lot nicer. Will update.

[PATCH v2 0/4] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets external tools provide hunks into git's diff and blame pipelines.

diff.<driver>.process opens the door to integrating content-aware logic that
don't exist inside git: structural diff tools, format-aware analyzers, and
other tools that understand the semantics of the content being tracked.
Where diff.<driver>.command replaces the diff pipeline entirely,
diff.<driver>.process feeds hunks into it, so all downstream features (word
diff, function context, color-moved, stat, blame) work normally.

The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
capability negotiation, one tool invocation per git command. The tool
receives file pairs and returns hunk descriptors that git feeds into the
standard xdiff pipeline.

Zero hunks with status=success means the tool considers the files
equivalent. git diff shows no output for the file, and git blame skips the
commit, attributing lines to earlier commits.

On error or tool crash, git falls back silently to the builtin diff
algorithm. The feature is opt-in via diff.<driver>.process and
.gitattributes; unconfigured files are unaffected.

The blame integration calls the diff process for each commit in the file's
history. The subprocess is long-running (one startup amortized across the
traversal), but per-commit round-trips add latency. A natural follow-up
would be a capability that sends blob OIDs instead of content, allowing
tools that can read the object store directly to avoid the cost of
serializing and deserializing blob content over the pipe for each file pair.

Changes since v1:

 * Dropped the built-in diff-process-normalize tool since it obscured the
   main use case for the RFC and update series messaging accordingly.
 * Encapsulated changed[] memory layout behind xdl_clear_changed() helper in
   xprepare.c.
 * Restructured the external hunks path in xdl_diff() to fall through to the
   regular diff algorithm on validation failure instead of duplicating diff
   logic on fallback then returning.
 * Changed subprocess error reporting from trace to warning.
 * Fixed whitespace issues in the test's embedded Python.
 * Changed instances of grep to test_grep in tests.

Michael Montalbo (4):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  diff: add long-running diff process via diff.<driver>.process
  blame: consult diff process for zero-hunk detection

 Documentation/config/diff.adoc   |   8 +
 Documentation/gitattributes.adoc |  43 ++++
 Makefile                         |   1 +
 blame.c                          |  43 +++-
 diff-process.c                   | 206 +++++++++++++++++
 diff-process.h                   |  28 +++
 diff.c                           |  23 ++
 t/.gitattributes                 |   1 +
 t/t4080-diff-process.sh          | 370 +++++++++++++++++++++++++++++++
 userdiff.c                       |   7 +
 userdiff.h                       |   2 +
 xdiff-interface.c                |   7 +-
 xdiff/xdiff.h                    |  13 ++
 xdiff/xdiffi.c                   |  84 ++++++-
 xdiff/xprepare.c                 |  10 +
 xdiff/xprepare.h                 |   1 +
 16 files changed, 840 insertions(+), 7 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh


base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v1:

 1:  8c0ea0bc07 ! 1:  f887a7e2ba xdiff: support external hunks via xpparam_t
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +	long j, prev_old_end = 0, prev_new_end = 0;
      +	long total_old = 0, total_new = 0;
      +
     -+	/*
     -+	 * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
     -+	 * them via xdl_cleanup_records().  The allocation is nrec + 2
     -+	 * elements; changed points one past the start (see xprepare.c).
     -+	 */
     -+	memset(xe->xdf1.changed - 1, 0,
     -+	       (xe->xdf1.nrec + 2) * sizeof(bool));
     -+	memset(xe->xdf2.changed - 1, 0,
     -+	       (xe->xdf2.nrec + 2) * sizeof(bool));
     ++	xdl_clear_changed(&xe->xdf1);
     ++	xdl_clear_changed(&xe->xdf2);
      +
      +	for (i = 0; i < nr_hunks; i++) {
      +		const struct xdl_hunk *h = &hunks[i];
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +		/* Bounds check (1-based line numbers) */
      +		if (h->old_count > 0 &&
      +		    (h->old_start < 1 ||
     -+		     h->old_start + h->old_count - 1 > xe->xdf1.nrec))
     ++		     h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
      +			return -1;
      +		if (h->new_count > 0 &&
      +		    (h->new_start < 1 ||
     -+		     h->new_start + h->new_count - 1 > xe->xdf2.nrec))
     ++		     h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
      +			return -1;
      +
      +		/* Zero-count hunks: start must still be in [1, nrec+1] */
      +		if (h->old_count == 0 &&
     -+		    (h->old_start < 1 || h->old_start > xe->xdf1.nrec + 1))
     ++		    (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
      +			return -1;
      +		if (h->new_count == 0 &&
     -+		    (h->new_start < 1 || h->new_start > xe->xdf2.nrec + 1))
     ++		    (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
      +			return -1;
      +
      +		/* Ordering: no overlap with previous hunk */
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
       	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
       
      -	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
     --
     --		return -1;
      +	if (xpp->external_hunks) {
      +		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
      +			return -1;
      +		if (xdl_populate_hunks_from_external(&xe,
      +						     xpp->external_hunks,
     -+						     xpp->external_hunks_nr) < 0) {
     -+			/*
     -+			 * Invalid external hunks; fall back to the
     -+			 * builtin diff algorithm.  Re-runs
     -+			 * xdl_prepare_env() via xdl_do_diff().
     -+			 */
     -+			xdl_free_env(&xe);
     -+			if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
     -+				return -1;
     -+		}
     -+	} else {
     -+		if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
     -+			return -1;
     - 	}
     ++						     xpp->external_hunks_nr) == 0)
     ++			goto diff_done;
     ++		xdl_free_env(&xe);
     ++	}
     + 
     ++	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
     + 		return -1;
     +-	}
     ++
     ++diff_done:
      +
       	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
       	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
       	    xdl_build_script(&xe, &xscr) < 0) {
     +
     + ## xdiff/xprepare.c ##
     +@@ xdiff/xprepare.c: int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
     + 
     + 	return 0;
     + }
     ++
     ++/*
     ++ * Reset the changed[] array so that no lines are marked as changed.
     ++ * Also clears the sentinel slots at changed[-1] and changed[nrec]
     ++ * that xdl_change_compact() relies on during backward scans.
     ++ */
     ++void xdl_clear_changed(xdfile_t *xdf)
     ++{
     ++	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
     ++}
     +
     + ## xdiff/xprepare.h ##
     +@@
     + int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
     + 		    xdfenv_t *xe);
     + void xdl_free_env(xdfenv_t *xe);
     ++void xdl_clear_changed(xdfile_t *xdf);
     + 
     + 
     + 
 2:  3bc127c800 = 2:  de6d85f9d7 userdiff: add diff.<driver>.process config
 3:  f9976fc6aa ! 3:  c25647c6e5 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
              [diff "cdiff"]
                  process = /path/to/diff-tool
      
     -    The tool receives file pairs and returns hunks describing which
     -    lines changed.  Git feeds these hunks into the standard xdiff
     -    pipeline, so all output features (word diff, function context,
     -    color) work normally.
     +    The tool provides custom line-matching: it receives file pairs
     +    and returns hunks that reference original line numbers.  Unlike
     +    textconv, which transforms the displayed content, the diff
     +    output shows the actual file while the tool controls which
     +    lines are marked as changed.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
          requests send command=hunks, pathname, and both file contents as
          packetized data.  The tool responds with hunk lines and a status
     -    packet.  On error, git falls back to the builtin diff algorithm.
     +    packet.  On error, git falls back to the builtin diff algorithm
     +    with a warning.
      
     -    Zero hunks with status=success means the tool considers the files
     -    equivalent.  Git skips diff output for that file entirely.
     +    Zero hunks with status=success means the tool considers the
     +    files equivalent.  Git skips diff output for that file.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +An external tool can provide content-aware line matching by
      +setting `diff.<name>.process` to the command that runs
      +the tool.  The tool is a long-running process that communicates via
     -+the pkt-line protocol (see
     -+linkgit:gitprotocol-long-running-process[5]).
     ++the pkt-line protocol (described in
     ++Documentation/technical/long-running-process-protocol.adoc).
      +
      +------------------------
      +*.c diff=cdiff
     @@ diff-process.c (new)
      @@
      +/*
      + * Diff process backend: communicates with a long-running external
     -+ * tool via the pkt-line protocol to obtain content-aware hunks.
     ++ * tool via the pkt-line protocol to obtain custom line-matching
     ++ * results.  Unlike textconv, which transforms the displayed content,
     ++ * hunks from a diff process reference original line numbers and
     ++ * the display shows the actual file content.
      + *
      + * Protocol: pkt-line over stdin/stdout, following the pattern of
      + * the long-running filter process protocol (see convert.c).
     @@ diff.c
       #include "userdiff.h"
      +#include "diff-process.h"
       #include "submodule.h"
     -+#include "trace2.h"
       #include "hashmap.h"
       #include "mem-pool.h"
     - #include "merge-ll.h"
      @@ diff.c: static void builtin_diff(const char *name_a,
       		xpparam_t xpp;
       		xdemitconf_t xecfg;
     @@ diff.c: static void builtin_diff(const char *name_a,
      +				xpp.external_hunks = ext_hunks;
      +				xpp.external_hunks_nr = ext_hunks_nr;
      +			} else {
     -+				trace2_data_string("diff",
     -+						   o->repo,
     -+						   "diff-process-fallback",
     -+						   name_a);
     ++				warning(_("diff process failed for '%s',"
     ++					  " falling back to builtin diff"),
     ++					name_a);
      +			}
      +		}
      +
     @@ diff.c: static void builtin_diff(const char *name_a,
       			free_diff_words_data(&ecbdata);
       		if (textconv_one)
      
     + ## t/.gitattributes ##
     +@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
     + /t8005/*.txt eol=lf
     + /t9*/*.dump eol=lf
     + /t0040*.sh whitespace=-indent-with-non-tab
     ++/t4080-diff-process.sh whitespace=-indent-with-non-tab
     +
       ## t/t4080-diff-process.sh (new) ##
      @@
      +#!/bin/sh
     @@ t/t4080-diff-process.sh (new)
      +	# appear as changed in the output.
      +	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
      +		diff boundary.c >actual &&
     -+	grep "^-OLD5" actual &&
     -+	grep "^-OLD6" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^+NEW6" actual &&
     -+	! grep "^-OLD9" actual &&
     -+	! grep "^-OLD10" actual &&
     -+	! grep "^+NEW9" actual &&
     -+	! grep "^+NEW10" actual
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^-OLD6" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^+NEW6" actual &&
     ++	! test_grep "^-OLD9" actual &&
     ++	! test_grep "^-OLD10" actual &&
     ++	! test_grep "^+NEW9" actual &&
     ++	! test_grep "^+NEW10" actual
      +'
      +
      +test_expect_success PYTHON 'diff process fallback on tool error status' '
     @@ t/t4080-diff-process.sh (new)
      +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
      +		diff boundary.c >actual &&
      +	# Fallback produces the full builtin diff (both change regions).
     -+	grep "^-OLD5" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^-OLD9" actual &&
     -+	grep "^+NEW9" actual &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
      +	# Tool was contacted (it replied with error, not crash).
     -+	grep "command=hunks pathname=boundary.c" backend.log
     ++	test_grep "command=hunks pathname=boundary.c" backend.log
      +'
      +
      +test_expect_success PYTHON 'diff process fallback on bad hunks' '
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
      +		diff boundary.c >actual &&
     -+	grep "^-OLD5" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^-OLD9" actual &&
     -+	grep "^+NEW9" actual
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual
      +'
      +
      +test_expect_success PYTHON 'diff process fallback on tool crash' '
      +	git -c diff.cdiff.process="$BACKEND --mode=crash" \
      +		diff boundary.c >actual &&
     -+	grep "^-OLD5" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^-OLD9" actual &&
     -+	grep "^+NEW9" actual
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual
      +'
      +
      +test_expect_success PYTHON 'diff process abort disables for session' '
     @@ t/t4080-diff-process.sh (new)
      +	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
      +		diff -- abort1.c abort2.c >actual &&
      +	# Both files should still produce diff output via fallback.
     -+	grep "return 10" actual &&
     -+	grep "return 20" actual &&
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
      +	# The tool aborts on the first file and git clears its
      +	# capability.  The second file never contacts the tool,
      +	# so the log should have exactly one entry, not two.
     -+	grep "command=hunks" backend.log >matches &&
     ++	test_grep "command=hunks" backend.log >matches &&
      +	test_line_count = 1 matches
      +'
      +
     @@ t/t4080-diff-process.sh (new)
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- multi1.c multi2.c >actual &&
     -+	grep "return 10" actual &&
     -+	grep "return 20" actual &&
     -+	grep "pathname=multi1.c" backend.log &&
     -+	grep "pathname=multi2.c" backend.log
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
     ++	test_grep "pathname=multi1.c" backend.log &&
     ++	test_grep "pathname=multi2.c" backend.log
      +'
      +
      +test_expect_success PYTHON 'diff process with --word-diff' '
     @@ t/t4080-diff-process.sh (new)
      +
      +	git -c diff.cdiff.process="$BACKEND" \
      +		diff --word-diff worddiff.c >actual &&
     -+	grep "\[-1;-\]" actual &&
     -+	grep "{+999;+}" actual
     ++	test_grep "\[-1;-\]" actual &&
     ++	test_grep "{+999;+}" actual
      +'
      +
      +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --diff-algorithm=patience worddiff.c >actual &&
     -+	grep "return 999" actual &&
     ++	test_grep "return 999" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     @@ t/t4080-diff-process.sh (new)
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		log -1 -p -- logtest.c >actual &&
     -+	grep "return 2" actual &&
     -+	grep "command=hunks pathname=logtest.c" backend.log
     ++	test_grep "return 2" actual &&
     ++	test_grep "command=hunks pathname=logtest.c" backend.log
      +'
      +
      +test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
 4:  4e6ea6d518 ! 4:  39ff53acef blame: consult diff process for zero-hunk detection
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
      +
      +	# Without zero-hunk mode, blame attributes the change.
      +	git blame blame.c >without &&
     -+	grep "$BLAME_COMMIT" without &&
     ++	test_grep "$BLAME_COMMIT" without &&
      +
      +	# With zero-hunk mode, the process considers the files equivalent
      +	# and blame skips the reformat commit.
      +	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
      +		blame blame.c >with &&
     -+	! grep "$BLAME_COMMIT" with
     ++	! test_grep "$BLAME_COMMIT" with
      +'
      +
      +
 5:  8c7359b8a1 < -:  ---------- diff-process-normalize: add built-in whitespace normalizer

-- 
gitgitgadget

[PATCH v2 1/4] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 13 ++++++++
 xdiff/xdiffi.c    | 84 +++++++++++++++++++++++++++++++++++++++++++++--
 xdiff/xprepare.c  | 10 ++++++
 xdiff/xprepare.h  |  1 +
 5 files changed, 112 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index f043330f2a..9542c0bcc2 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..2ee6f1aae3 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based, matching unified diff convention.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +97,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm. */
+	const struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index 5455b4690d..e7d6190d37 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,96 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    const struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long total_old = 0, total_new = 0;
+
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0)
+			return -1;
+
+		/* Bounds check (1-based line numbers) */
+		if (h->old_count > 0 &&
+		    (h->old_start < 1 ||
+		     h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
+			return -1;
+		if (h->new_count > 0 &&
+		    (h->new_start < 1 ||
+		     h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
+			return -1;
+
+		/* Zero-count hunks: start must still be in [1, nrec+1] */
+		if (h->old_count == 0 &&
+		    (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
+			return -1;
+		if (h->new_count == 0 &&
+		    (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
+			return -1;
+
+		/* Ordering: no overlap with previous hunk */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+		total_old += h->old_count;
+		total_new += h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 */
+	if ((long)xe->xdf1.nrec - total_old !=
+	    (long)xe->xdf2.nrec - total_new)
+		return -1;
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
+
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index cd4fc405eb..4645a9a746 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -432,3 +432,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v2 2/4] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add a new per-driver configuration key that specifies the command
for a long-running diff process.

The name follows filter.<driver>.process: a long-running subprocess
that stays alive across files within a single git invocation.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index fe710a68bf..81c0bebcce 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference original line numbers.  Unlike
textconv, which transforms the displayed content, the diff
output shows the actual file while the tool controls which
lines are marked as changed.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet.  On error, git falls back to the builtin diff algorithm
with a warning.

Zero hunks with status=success means the tool considers the
files equivalent.  Git skips diff output for that file.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   8 +
 Documentation/gitattributes.adoc |  40 ++++
 Makefile                         |   1 +
 diff-process.c                   | 206 +++++++++++++++++++
 diff-process.h                   |  28 +++
 diff.c                           |  23 +++
 t/.gitattributes                 |   1 +
 t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
 8 files changed, 645 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..4ab5f60df6 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,14 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process.
+	The tool communicates via the pkt-line protocol and returns
+	hunks that are fed into Git's diff and blame pipelines.
+	If the tool returns zero hunks, the file is treated as
+	unchanged for both diff output and blame attribution.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f20041a323..962896a0b4 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,46 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An external tool can provide content-aware line matching by
+setting `diff.<name>.process` to the command that runs
+the tool.  The tool is a long-running process that communicates via
+the pkt-line protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+The tool receives file pairs and returns hunk descriptors indicating
+which lines changed.  Git feeds these hunks into its standard diff
+pipeline, so all output features (word diff, function context,
+color) work normally.
+
+If the tool fails or returns an error, Git silently falls back to
+the builtin diff algorithm.  If the tool returns invalid hunks
+(out of bounds, overlapping), Git also falls back silently.
+
+The handshake negotiates `version=1` and `capability=hunks`.
+Per-file requests send `command=hunks` and `pathname=<path>`,
+followed by the old and new file content as packetized data.
+The tool responds with lines of the form
+`hunk <old_start> <old_count> <new_start> <new_count>`
+(1-based line numbers), a flush packet, and `status=success`.
+
+If the tool returns zero hunks with `status=success`, Git treats
+the file as having no changes and produces no diff output.
+
+Tools should ignore unknown keys in the per-file request to
+remain forward-compatible.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index cedc234173..22900368dd 100644
--- a/Makefile
+++ b/Makefile
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..801ac9e22e
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,206 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  Unlike textconv, which transforms the displayed content,
+ * hunks from a diff process reference original line numbers and
+ * the display shows the actual file content.
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * Zero hunks with status=success means the tool considers the
+ * files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int subprocess_map_initialized;
+static struct hashmap subprocess_map;
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		(struct diff_subprocess *)subprocess;
+
+	/* Uses dying pkt-line variant, same as convert.c filters. */
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *find_or_start_process(const char *cmd)
+{
+	struct diff_subprocess *entry;
+
+	if (!subprocess_map_initialized) {
+		subprocess_map_initialized = 1;
+		hashmap_init(&subprocess_map, cmd2process_cmp, NULL, 0);
+	}
+
+	entry = (struct diff_subprocess *)
+		subprocess_find_entry(&subprocess_map, cmd);
+	if (entry)
+		return entry;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start(&subprocess_map, &entry->subprocess,
+			     cmd, start_diff_process_fn)) {
+		free(entry);
+		return NULL;
+	}
+
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret;
+
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	else
+		ret = 0;
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	hunk->old_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->old_count = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_count = strtol(line, &end, 10);
+	if (end == line || *end != '\0')
+		return -1;
+
+	return 0;
+}
+
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	if (!drv || !drv->process)
+		return -1;
+
+	backend = find_or_start_process(drv->process);
+	if (!backend)
+		return -1;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return -1;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto error;
+
+	if (strcmp(status.buf, "success")) {
+		if (!strcmp(status.buf, "abort"))
+			backend->supported_capabilities &= ~CAP_HUNKS;
+		goto error;
+	}
+
+	*hunks_out = hunks;
+	*nr_hunks_out = nr_hunks;
+	strbuf_release(&status);
+	return 0;
+
+error:
+	free(hunks);
+	strbuf_release(&status);
+	return -1;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..4c84951e02
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,28 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+struct userdiff_driver;
+struct xdl_hunk;
+
+/*
+ * Query a diff process for hunks describing the changes
+ * between old_buf and new_buf.
+ *
+ * The backend is a long-running subprocess configured via
+ * diff.<driver>.process.  It receives file content via
+ * pkt-line and returns hunks with 1-based line numbers.
+ *
+ * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
+ * array (caller must free) and returns 0.
+ *
+ * On failure, returns -1.  The caller should fall back to the
+ * builtin diff algorithm.
+ */
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 397e38b41c..1aeb0f319e 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -3991,6 +3992,7 @@ static void builtin_diff(const char *name_a,
 		xpparam_t xpp;
 		xdemitconf_t xecfg;
 		struct emit_callback ecbdata;
+		struct xdl_hunk *ext_hunks = NULL;
 		unsigned ws_rule;
 		const struct userdiff_funcname *pe;
 
@@ -4031,6 +4033,26 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (!o->ignore_driver_algorithm &&
+		    one->driver && one->driver->process) {
+			size_t ext_hunks_nr = 0;
+			if (!diff_process_get_hunks(
+				    one->driver, name_a,
+				    mf1.ptr, mf1.size,
+				    mf2.ptr, mf2.size,
+				    &ext_hunks, &ext_hunks_nr)) {
+				if (!ext_hunks_nr)
+					goto free_ab_and_return;
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = ext_hunks_nr;
+			} else {
+				warning(_("diff process failed for '%s',"
+					  " falling back to builtin diff"),
+					name_a);
+			}
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4111,6 +4133,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(ext_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/t/.gitattributes b/t/.gitattributes
index 7664c6e027..de97920cab 100644
--- a/t/.gitattributes
+++ b/t/.gitattributes
@@ -23,3 +23,4 @@ t[0-9][0-9][0-9][0-9]/* -whitespace
 /t8005/*.txt eol=lf
 /t9*/*.dump eol=lf
 /t0040*.sh whitespace=-indent-with-non-tab
+/t4080-diff-process.sh whitespace=-indent-with-non-tab
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..083e48e872
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,338 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+if test_have_prereq PYTHON
+then
+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+fi
+
+#
+# A single parametric diff process.
+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
+#
+# Modes:
+#   whole-file  - report all lines as changed (default)
+#   fixed-hunk  - always report hunk 5 2 5 2
+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
+#   zero-hunk   - return zero hunks (files considered equivalent)
+#   error       - return status=error for every request
+#   abort       - return status=abort for every request
+#   crash       - read one request then exit without responding
+#
+setup_backend () {
+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
+	import sys, os
+
+	def read_pkt():
+	    hdr = sys.stdin.buffer.read(4)
+	    if len(hdr) < 4: return None
+	    length = int(hdr, 16)
+	    if length == 0: return ""
+	    data = sys.stdin.buffer.read(length - 4)
+	    return data.decode().rstrip("\n")
+
+	def write_pkt(line):
+	    data = (line + "\n").encode()
+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    sys.stdout.buffer.flush()
+
+	def write_flush():
+	    sys.stdout.buffer.write(b"0000")
+	    sys.stdout.buffer.flush()
+
+	def read_content():
+	    chunks = []
+	    while True:
+	        hdr = sys.stdin.buffer.read(4)
+	        if len(hdr) < 4: break
+	        length = int(hdr, 16)
+	        if length == 0: break
+	        chunks.append(sys.stdin.buffer.read(length - 4))
+	    return b"".join(chunks)
+
+	mode = "whole-file"
+	logfile = None
+	for arg in sys.argv[1:]:
+	    if arg.startswith("--mode="):
+	        mode = arg[7:]
+	    elif arg.startswith("--log="):
+	        logfile = open(arg[6:], "a")
+
+	def log(msg):
+	    if logfile:
+	        logfile.write(msg + "\n")
+	        logfile.flush()
+
+	# Handshake
+	assert read_pkt() == "git-diff-client"
+	assert read_pkt() == "version=1"
+	read_pkt()
+	write_pkt("git-diff-server")
+	write_pkt("version=1")
+	write_flush()
+	while True:
+	    p = read_pkt()
+	    if p == "": break
+	write_pkt("capability=hunks")
+	write_flush()
+
+	log("ready")
+
+	while True:
+	    cmd = None
+	    pathname = None
+	    while True:
+	        p = read_pkt()
+	        if p is None: sys.exit(0)
+	        if p == "": break
+	        if p.startswith("command="): cmd = p.split("=",1)[1]
+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
+	    if cmd is None: sys.exit(0)
+	    old = read_content()
+	    new = read_content()
+	    log(f"command={cmd} pathname={pathname}")
+
+	    if mode == "error":
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	        continue
+
+	    if mode == "abort":
+	        write_flush()
+	        write_pkt("status=abort")
+	        write_flush()
+	        continue
+
+	    if mode == "crash":
+	        sys.exit(1)
+
+	    if cmd == "hunks":
+	        if mode == "fixed-hunk":
+	            write_pkt("hunk 5 2 5 2")
+	        elif mode == "bad-hunk":
+	            write_pkt("hunk 999 1 999 1")
+	        elif mode == "zero-hunk":
+	            pass
+	        else:
+	            ol = len(old.split(b"\n"))
+	            nl = len(new.split(b"\n"))
+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	        write_flush()
+	        write_pkt("status=success")
+	        write_flush()
+	    else:
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	PYEOF
+	write_script diff-process-backend <<-SHEOF
+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
+	SHEOF
+}
+
+BACKEND="./diff-process-backend"
+
+test_expect_success PYTHON 'setup' '
+	setup_backend &&
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+	git commit -m "initial"
+'
+
+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+	git commit -m "add boundary.c" &&
+
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	! test_grep "^-OLD9" actual &&
+	! test_grep "^-OLD10" actual &&
+	! test_grep "^+NEW9" actual &&
+	! test_grep "^+NEW10" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool error status' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process abort disables for session' '
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add abort1.c abort2.c &&
+	git commit -m "add abort files" &&
+
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- abort1.c abort2.c >actual &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool,
+	# so the log should have exactly one entry, not two.
+	test_grep "command=hunks" backend.log >matches &&
+	test_line_count = 1 matches
+'
+
+test_expect_success PYTHON 'diff process handles multiple files' '
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 1; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 2; }
+	EOF
+	git add multi1.c multi2.c &&
+	git commit -m "add multi files" &&
+
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 10; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- multi1.c multi2.c >actual &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	test_grep "pathname=multi1.c" backend.log &&
+	test_grep "pathname=multi2.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process with --word-diff' '
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+	git commit -m "add worddiff.c" &&
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND" \
+		diff --word-diff worddiff.c >actual &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual
+'
+
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process works with git log -p' '
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+	git commit -m "add logtest.c" &&
+
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		log -1 -p -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add zerohunk.c &&
+	git commit -m "add zerohunk.c" &&
+
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		diff zerohunk.c >actual &&
+	test_must_be_empty actual
+'
+
+test_done
-- 
gitgitgadget

[PATCH v2 4/4] blame: consult diff process for zero-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns zero hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc |  3 +++
 blame.c                          | 43 +++++++++++++++++++++++++++++---
 t/t4080-diff-process.sh          | 32 ++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 962896a0b4..c087b4b265 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -857,6 +857,9 @@ The tool responds with lines of the form
 
 If the tool returns zero hunks with `status=success`, Git treats
 the file as having no changes and produces no diff output.
+`git blame` also consults the diff process and skips commits
+where it reports zero hunks, attributing lines to earlier commits
+instead.
 
 Tools should ignore unknown keys in the per-file request to
 remain forward-compatible.
diff --git a/blame.c b/blame.c
index a3c49d132e..8a5f14db7a 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "userdiff.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -315,16 +317,47 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data,
+		      int xdl_opts, struct index_state *istate,
+		      const char *path)
 {
 	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
+	struct xdl_hunk *ext_hunks = NULL;
+	int ret;
 
 	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+
+	if (path && istate) {
+		struct userdiff_driver *drv;
+		drv = userdiff_find_by_path(istate, path);
+		if (drv && drv->process) {
+			size_t nr = 0;
+			if (!diff_process_get_hunks(drv, path,
+						    file_a->ptr, file_a->size,
+						    file_b->ptr, file_b->size,
+						    &ext_hunks, &nr)) {
+				if (!nr) {
+					/*
+					 * Zero hunks: the diff process
+					 * considers these files equivalent.
+					 * Skip so blame looks past this
+					 * commit.
+					 */
+					return 0;
+				}
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = nr;
+			}
+		}
+	}
+
+	ret = xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	free(ext_hunks);
+	return ret;
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1961,7 +1994,8 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts,
+		       sb->revs->diffopt.repo->index, target->path))
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
@@ -2114,7 +2148,8 @@ static void find_copy_in_blob(struct blame_scoreboard *sb,
 	 * file_p partially may match that image.
 	 */
 	memset(split, 0, sizeof(struct blame_entry [3]));
-	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts))
+	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts,
+		       NULL, NULL))
 		die("unable to generate diff (%s)",
 		    oid_to_hex(&parent->commit->object.oid));
 	/* remainder, if any, all match the preimage */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 083e48e872..50f49a9b02 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -335,4 +335,36 @@ test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
 	test_must_be_empty actual
 '
 
+test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without zero-hunk mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With zero-hunk mode, the process considers the files equivalent
+	# and blame skips the reformat commit.
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		blame blame.c >with &&
+	! test_grep "$BLAME_COMMIT" with
+'
+
+
 test_done
-- 
gitgitgadget

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +struct diff_subprocess {
> +	struct subprocess_entry subprocess;
> +	unsigned int supported_capabilities;
> +};
> +
> +static int subprocess_map_initialized;
> +static struct hashmap subprocess_map;

Can we avoid introducing new global variables like these?  Would
"struct userdiff_driver" or "struct diff_options" be a good place to
hang this hashmap, perhaps?

> +static int send_file_content(int fd, const char *buf, long size)
> +{
> +	int ret;
> +
> +	if (size > 0)
> +		ret = write_packetized_from_buf_no_flush(buf, size, fd);
> +	else
> +		ret = 0;

Shouldn't "size == -24" be flagged as an invalid input?

> +	if (ret)
> +		return ret;
> +	return packet_flush_gently(fd);
> +}

> +static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
> +{
> +...
> +}

This gives a silent error diagnosis, which is good for a lower level
helper.

> +int diff_process_get_hunks(struct userdiff_driver *drv,
> +			   const char *path,
> +			   const char *old_buf, long old_size,
> +			   const char *new_buf, long new_size,
> +			   struct xdl_hunk **hunks_out,
> +			   size_t *nr_hunks_out)
> +{
> +	struct diff_subprocess *backend;
> +	struct child_process *process;
> +	int fd_in, fd_out;
> +	struct strbuf status = STRBUF_INIT;
> +	struct xdl_hunk *hunks = NULL;
> +	struct xdl_hunk hunk;
> +	size_t nr_hunks = 0, alloc_hunks = 0;
> +	int len;
> +	char *line;
> +
> +	if (!drv || !drv->process)
> +		return -1;

A driver that does not define process is not an error; it is
perfectly normal in the current world order where nobody has such an
external process and even fi this patch lands, external processes
are optional.  So here "return -1" does not mean an error, and
silent return is perfectly fine.

> +	backend = find_or_start_process(drv->process);
> +	if (!backend)
> +		return -1;

This is probably an error; the user specified drv->process, we
either tried to find or start the process and failed.  Isn't it an
event that deserves to be reported in an error message?

> +	if (!(backend->supported_capabilities & CAP_HUNKS))
> +		return -1;

Backend started, but the "hunks" feature is not supported.  Perhaps
in a year or two, this external process protocol may have become so
popular that it gained more capabilities, possibly making get_hunks
obsolete.  We may be looking at such an external process that uses
other capabilities but not this one.  This is not an error, so
silent return is perfectly fine.

> +	process = subprocess_get_child_process(&backend->subprocess);
> +	fd_in = process->in;
> +	fd_out = process->out;
> +
> +	/* Send request */
> +	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
> +	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
> +	    packet_flush_gently(fd_in))
> +		goto error;
> +
> +	/* Send old file content */
> +	if (send_file_content(fd_in, old_buf, old_size))
> +		goto error;
> +
> +	/* Send new file content */
> +	if (send_file_content(fd_in, new_buf, new_size))
> +		goto error;
> +
> +	/* Read hunks until flush packet */
> +	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
> +	       line) {
> +		if (parse_hunk_line(line, &hunk) < 0)
> +			goto error;
> +		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
> +		hunks[nr_hunks++] = hunk;
> +	}
> +	if (len < 0)
> +		goto error;
> +
> +	/* Read status */
> +	if (subprocess_read_status(fd_out, &status))
> +		goto error;
> +
> +	if (strcmp(status.buf, "success")) {
> +		if (!strcmp(status.buf, "abort"))
> +			backend->supported_capabilities &= ~CAP_HUNKS;
> +		goto error;
> +	}
> +
> +	*hunks_out = hunks;
> +	*nr_hunks_out = nr_hunks;
> +	strbuf_release(&status);
> +	return 0;
> +
> +error:

All exceptions that lead here look like events that should be
reported to the end-user.

> +	free(hunks);
> +	strbuf_release(&status);
> +	return -1;
> +}

> +/*
> + * Query a diff process for hunks describing the changes
> + * between old_buf and new_buf.
> + *
> + * The backend is a long-running subprocess configured via
> + * diff.<driver>.process.  It receives file content via
> + * pkt-line and returns hunks with 1-based line numbers.
> + *
> + * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
> + * array (caller must free) and returns 0.
> + *
> + * On failure, returns -1.  The caller should fall back to the
> + * builtin diff algorithm.
> + */

I do not agree with this.  If it is a failure, the user should fix
the external process (or disable).  It shouldn't be hidden behind a
fallback.  As I left comments, in this round of implementation,
there are conditions that returns -1 for soemthing that is not an
error (i.e., not configured, or process not supporting the
particular capability) *and* in those cases the caller should fall
back as if nothing happened.  But some error cases, the caller
should't hide them.

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Zero hunks with status=success means the tool considers the
> files equivalent.  Git skips diff output for that file.

Is "zero hunk" a common word or some random string you invented?  If
the latter, which is I am assuming it to be, you should define what
it means at/before the first use.  Here in the proposed log message,
and ...

>
> Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
> ---
>  Documentation/config/diff.adoc   |   8 +
>  Documentation/gitattributes.adoc |  40 ++++
>  Makefile                         |   1 +
>  diff-process.c                   | 206 +++++++++++++++++++
>  diff-process.h                   |  28 +++
>  diff.c                           |  23 +++
>  t/.gitattributes                 |   1 +
>  t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
>  8 files changed, 645 insertions(+)
>  create mode 100644 diff-process.c
>  create mode 100644 diff-process.h
>  create mode 100755 t/t4080-diff-process.sh
>
> diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
> index 1135a62a0a..4ab5f60df6 100644
> --- a/Documentation/config/diff.adoc
> +++ b/Documentation/config/diff.adoc
> @@ -218,6 +218,14 @@ endif::git-diff[]
>  	Set this option to `true` to make the diff driver cache the text
>  	conversion outputs.  See linkgit:gitattributes[5] for details.
>  
> +`diff.<driver>.process`::
> +	The command to run as a long-running diff process.
> +	The tool communicates via the pkt-line protocol and returns
> +	hunks that are fed into Git's diff and blame pipelines.
> +	If the tool returns zero hunks, the file is treated as
> +	unchanged for both diff output and blame attribution.
> +	See linkgit:gitattributes[5] for details.

... also here.

I do not know if you mean "the tool returns no hunks" (there is no
"hunk <old_start> <old_count> <new_start> <new_count>" line passed
from the tool over the protocol) or "the tool returns zero-hunk"
(there is a special "zero-hunk" message to signal this particular
condition sent over the protocol), and this description does not
quite help disambiguating between the two.

If the former, then avoid "zero hunks" as it sounds like a noun with
special meaning.  Yes, we can say "tool returns one hunk", "tool
returns 31 hunks", etc., so "tool returns zero hunks" may logically
be correct, but "when the tool returns no hunks with status=success"
is much less confusing, I think.

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Mon, May 25, 2026 at 6:56 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +struct diff_subprocess {
> > +     struct subprocess_entry subprocess;
> > +     unsigned int supported_capabilities;
> > +};
> > +
> > +static int subprocess_map_initialized;
> > +static struct hashmap subprocess_map;
>
> Can we avoid introducing new global variables like these?  Would
> "struct userdiff_driver" or "struct diff_options" be a good place to
> hang this hashmap, perhaps?
>

Will clean this up.

> > +static int send_file_content(int fd, const char *buf, long size)
> > +{
> > +     int ret;
> > +
> > +     if (size > 0)
> > +             ret = write_packetized_from_buf_no_flush(buf, size, fd);
> > +     else
> > +             ret = 0;
>
> Shouldn't "size == -24" be flagged as an invalid input?
>

Will fix and do a broader audit of input validation and bounds checking.

> > +     if (ret)
> > +             return ret;
> > +     return packet_flush_gently(fd);
> > +}
>
> > +static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
> > +{
> > +...
> > +}
>
> This gives a silent error diagnosis, which is good for a lower level
> helper.
>
> > +int diff_process_get_hunks(struct userdiff_driver *drv,
> > +                        const char *path,
> > +                        const char *old_buf, long old_size,
> > +                        const char *new_buf, long new_size,
> > +                        struct xdl_hunk **hunks_out,
> > +                        size_t *nr_hunks_out)
> > +{
> > +     struct diff_subprocess *backend;
> > +     struct child_process *process;
> > +     int fd_in, fd_out;
> > +     struct strbuf status = STRBUF_INIT;
> > +     struct xdl_hunk *hunks = NULL;
> > +     struct xdl_hunk hunk;
> > +     size_t nr_hunks = 0, alloc_hunks = 0;
> > +     int len;
> > +     char *line;
> > +
> > +     if (!drv || !drv->process)
> > +             return -1;
>
> A driver that does not define process is not an error; it is
> perfectly normal in the current world order where nobody has such an
> external process and even fi this patch lands, external processes
> are optional.  So here "return -1" does not mean an error, and
> silent return is perfectly fine.
>
> > +     backend = find_or_start_process(drv->process);
> > +     if (!backend)
> > +             return -1;
>
> This is probably an error; the user specified drv->process, we
> either tried to find or start the process and failed.  Isn't it an
> event that deserves to be reported in an error message?
>
> > +     if (!(backend->supported_capabilities & CAP_HUNKS))
> > +             return -1;
>
> Backend started, but the "hunks" feature is not supported.  Perhaps
> in a year or two, this external process protocol may have become so
> popular that it gained more capabilities, possibly making get_hunks
> obsolete.  We may be looking at such an external process that uses
> other capabilities but not this one.  This is not an error, so
> silent return is perfectly fine.
>
> > +     process = subprocess_get_child_process(&backend->subprocess);
> > +     fd_in = process->in;
> > +     fd_out = process->out;
> > +
> > +     /* Send request */
> > +     if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
> > +         packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
> > +         packet_flush_gently(fd_in))
> > +             goto error;
> > +
> > +     /* Send old file content */
> > +     if (send_file_content(fd_in, old_buf, old_size))
> > +             goto error;
> > +
> > +     /* Send new file content */
> > +     if (send_file_content(fd_in, new_buf, new_size))
> > +             goto error;
> > +
> > +     /* Read hunks until flush packet */
> > +     while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
> > +            line) {
> > +             if (parse_hunk_line(line, &hunk) < 0)
> > +                     goto error;
> > +             ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
> > +             hunks[nr_hunks++] = hunk;
> > +     }
> > +     if (len < 0)
> > +             goto error;
> > +
> > +     /* Read status */
> > +     if (subprocess_read_status(fd_out, &status))
> > +             goto error;
> > +
> > +     if (strcmp(status.buf, "success")) {
> > +             if (!strcmp(status.buf, "abort"))
> > +                     backend->supported_capabilities &= ~CAP_HUNKS;
> > +             goto error;
> > +     }
> > +
> > +     *hunks_out = hunks;
> > +     *nr_hunks_out = nr_hunks;
> > +     strbuf_release(&status);
> > +     return 0;
> > +
> > +error:
>
> All exceptions that lead here look like events that should be
> reported to the end-user.
>

Agreed on all points. I will restructure things so errors are flagged when
appropriate (i.e., user specified a process but one was not found / couldn't
start and exceptions) and non-errors are treated as they should be.

> > +     free(hunks);
> > +     strbuf_release(&status);
> > +     return -1;
> > +}
>
> > +/*
> > + * Query a diff process for hunks describing the changes
> > + * between old_buf and new_buf.
> > + *
> > + * The backend is a long-running subprocess configured via
> > + * diff.<driver>.process.  It receives file content via
> > + * pkt-line and returns hunks with 1-based line numbers.
> > + *
> > + * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
> > + * array (caller must free) and returns 0.
> > + *
> > + * On failure, returns -1.  The caller should fall back to the
> > + * builtin diff algorithm.
> > + */
>
> I do not agree with this.  If it is a failure, the user should fix
> the external process (or disable).  It shouldn't be hidden behind a
> fallback.  As I left comments, in this round of implementation,
> there are conditions that returns -1 for soemthing that is not an
> error (i.e., not configured, or process not supporting the
> particular capability) *and* in those cases the caller should fall
> back as if nothing happened.  But some error cases, the caller
> should't hide them.

Will address in a follow-up.

Thank you for the feedback!

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Mon, May 25, 2026 at 7:26 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > Zero hunks with status=success means the tool considers the
> > files equivalent.  Git skips diff output for that file.
>
> Is "zero hunk" a common word or some random string you invented?  If
> the latter, which is I am assuming it to be, you should define what
> it means at/before the first use.  Here in the proposed log message,
> and ...
>
> >
> > Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
> > ---
> >  Documentation/config/diff.adoc   |   8 +
> >  Documentation/gitattributes.adoc |  40 ++++
> >  Makefile                         |   1 +
> >  diff-process.c                   | 206 +++++++++++++++++++
> >  diff-process.h                   |  28 +++
> >  diff.c                           |  23 +++
> >  t/.gitattributes                 |   1 +
> >  t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
> >  8 files changed, 645 insertions(+)
> >  create mode 100644 diff-process.c
> >  create mode 100644 diff-process.h
> >  create mode 100755 t/t4080-diff-process.sh
> >
> > diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
> > index 1135a62a0a..4ab5f60df6 100644
> > --- a/Documentation/config/diff.adoc
> > +++ b/Documentation/config/diff.adoc
> > @@ -218,6 +218,14 @@ endif::git-diff[]
> >       Set this option to `true` to make the diff driver cache the text
> >       conversion outputs.  See linkgit:gitattributes[5] for details.
> >
> > +`diff.<driver>.process`::
> > +     The command to run as a long-running diff process.
> > +     The tool communicates via the pkt-line protocol and returns
> > +     hunks that are fed into Git's diff and blame pipelines.
> > +     If the tool returns zero hunks, the file is treated as
> > +     unchanged for both diff output and blame attribution.
> > +     See linkgit:gitattributes[5] for details.
>
> ... also here.
>
> I do not know if you mean "the tool returns no hunks" (there is no
> "hunk <old_start> <old_count> <new_start> <new_count>" line passed
> from the tool over the protocol) or "the tool returns zero-hunk"
> (there is a special "zero-hunk" message to signal this particular
> condition sent over the protocol), and this description does not
> quite help disambiguating between the two.
>
> If the former, then avoid "zero hunks" as it sounds like a noun with
> special meaning.  Yes, we can say "tool returns one hunk", "tool
> returns 31 hunks", etc., so "tool returns zero hunks" may logically
> be correct, but "when the tool returns no hunks with status=success"
> is much less confusing, I think.

Yes, "zero hunks" was my own invention and I see why it's confusing. Will
update the messaging to use "no hunks" instead and do a broader sweep of
the documentation to clarify the protocol and expected tool behavior.

[PATCH v3 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's output entirely, losing downstream
features like word diff, function context, color, and blame.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes.

The first two patches add xdiff plumbing for externally supplied hunks and
the diff.<driver>.process config key. Patch 3 refactors the subprocess API
to separate process lifecycle from hashmap management, since the diff
process stores its subprocess on the userdiff driver rather than in a
hashmap. The main feature lands in patch 4. Patch 5 wires up bypass knobs
(--no-ext-diff, format-patch). Patch 6 integrates with blame so the tool can
declare commits as having no changes.

Changes since v2:

 * Extracted subprocess_start_command() from subprocess_start() so the diff
   process can reuse the startup machinery without a hashmap (new patch 3).
 * Subprocess stored on userdiff_driver, no global variables.
 * Differentiated error handling: tool failure warns (with tool name), "not
   configured" is silent, abort is voluntary.
 * Single public entry point: diff_process_fill_hunks() handles driver
   lookup, flag checks, subprocess management, and error reporting for both
   diff.c and blame.c.
 * Rewrote gitattributes protocol section to match filter process convention
   with packet diagrams and fully specified hunk format.
 * Split bypass knobs (--no-ext-diff, format-patch) into a separate commit.
 * SIGPIPE protection, stricter input validation, const correctness.
 * Changed "zero hunks" to "no hunks" throughout.

Michael Montalbo (6):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 139 +++++
 Makefile                                 |   1 +
 blame.c                                  |  40 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 288 ++++++++++
 diff-process.h                           |  39 ++
 diff.c                                   |  29 +-
 diff.h                                   |   5 +
 meson.build                              |   1 +
 sub-process.c                            |  29 +-
 sub-process.h                            |   9 +-
 t/.gitattributes                         |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 660 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  13 +
 xdiff/xdiffi.c                           |  85 ++-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 24 files changed, 1368 insertions(+), 21 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh


base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v2:

 1:  f887a7e2ba ! 1:  13eb201d63 xdiff: support external hunks via xpparam_t
     @@ Commit message
          numbers, overlapping or out-of-order hunks, negative counts, and
          violations of the synchronization invariant (unchanged line counts
          must match between files).  On validation failure, fall back to
     -    the builtin diff algorithm.
     +    the builtin diff algorithm; this re-runs xdl_prepare_env() since
     +    the first call may have dirtied the changed[] arrays.
      
          Skip trim_common_tail() in xdi_diff() when external hunks are
          present, since external hunks reference line numbers in the
     @@ xdiff/xdiff.h: typedef struct s_xpparam {
       	char **anchors;
       	size_t anchors_nr;
      +
     -+	/* Externally computed hunks: bypass the diff algorithm. */
     -+	const struct xdl_hunk *external_hunks;
     ++	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
     ++	struct xdl_hunk *external_hunks;
      +	size_t external_hunks_nr;
       } xpparam_t;
       
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      + * Returns 0 on success, -1 on validation failure.
      + */
      +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
     -+					    const struct xdl_hunk *hunks,
     ++					    struct xdl_hunk *hunks,
      +					    size_t nr_hunks)
      +{
      +	size_t i;
      +	long j, prev_old_end = 0, prev_new_end = 0;
      +	long total_old = 0, total_new = 0;
      +
     ++	/*
     ++	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
     ++	 * Clear them so only the external hunks are marked.
     ++	 */
      +	xdl_clear_changed(&xe->xdf1);
      +	xdl_clear_changed(&xe->xdf2);
      +
      +	for (i = 0; i < nr_hunks; i++) {
     -+		const struct xdl_hunk *h = &hunks[i];
     ++		struct xdl_hunk *h = &hunks[i];
      +
      +		if (h->old_count < 0 || h->new_count < 0)
      +			return -1;
     -+
     -+		/* Bounds check (1-based line numbers) */
     -+		if (h->old_count > 0 &&
     -+		    (h->old_start < 1 ||
     -+		     h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
     -+			return -1;
     -+		if (h->new_count > 0 &&
     -+		    (h->new_start < 1 ||
     -+		     h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
     ++		if (h->old_start < 1 || h->new_start < 1)
      +			return -1;
      +
     -+		/* Zero-count hunks: start must still be in [1, nrec+1] */
     -+		if (h->old_count == 0 &&
     -+		    (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
     ++		/*
     ++		 * Range must fit: start + count - 1 <= nrec,
     ++		 * rewritten to avoid overflow.  Same for both sides.
     ++		 *
     ++		 * When count is 0 (pure insert/delete) the check
     ++		 * reduces to 0 > nrec - start + 1, which rejects
     ++		 * start > nrec + 1 and allows start == nrec + 1
     ++		 * (the position after the last line).
     ++		 */
     ++		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
      +			return -1;
     -+		if (h->new_count == 0 &&
     -+		    (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
     ++		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
      +			return -1;
      +
     -+		/* Ordering: no overlap with previous hunk */
     ++		/* Ordering: no overlap with previous hunk (adjacent is OK) */
      +		if (h->old_start < prev_old_end ||
      +		    h->new_start < prev_new_end)
      +			return -1;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      -	}
      +
      +diff_done:
     -+
       	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
       	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
       	    xdl_build_script(&xe, &xscr) < 0) {
 2:  de6d85f9d7 ! 2:  58f4763c63 userdiff: add diff.<driver>.process config
     @@ Metadata
       ## Commit message ##
          userdiff: add diff.<driver>.process config
      
     -    Add a new per-driver configuration key that specifies the command
     -    for a long-running diff process.
     -
     -    The name follows filter.<driver>.process: a long-running subprocess
     -    that stays alive across files within a single git invocation.
     +    Add the process field to struct userdiff_driver and teach the
     +    config parser to populate it from diff.<driver>.process.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
 -:  ---------- > 3:  d6c833dd42 sub-process: separate process lifecycle from hashmap management
 3:  c25647c6e5 ! 4:  d044fa0ee5 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
                  process = /path/to/diff-tool
      
          The tool provides custom line-matching: it receives file pairs
     -    and returns hunks that reference original line numbers.  Unlike
     -    textconv, which transforms the displayed content, the diff
     -    output shows the actual file while the tool controls which
     -    lines are marked as changed.
     +    and returns hunks that reference line numbers in the content.
     +    When textconv is also configured, the tool receives the
     +    textconv-transformed content.  The tool controls which lines
     +    are marked as changed while the display shows the file content.
     +    Patch output features (word diff, function context, color) work
     +    normally; summary formats like --stat use their own diff path
     +    and are not affected.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
          requests send command=hunks, pathname, and both file contents as
          packetized data.  The tool responds with hunk lines and a status
     -    packet.  On error, git falls back to the builtin diff algorithm
     -    with a warning.
     +    packet (success, error, or abort).  On error, Git warns and falls
     +    back to the builtin diff algorithm for that file.  On abort, Git
     +    silently falls back for the current file and stops sending further
     +    requests to the tool for the remainder of the session.
      
     -    Zero hunks with status=success means the tool considers the
     -    files equivalent.  Git skips diff output for that file.
     +    When the tool returns no hunks followed by status=success, Git
     +    treats the file as having no changes and produces no diff output.
     +    This also means --exit-code reports no changes for that file.
     +
     +    The subprocess is stored on the userdiff_driver struct and
     +    launched on first use.  If the process fails to start, the
     +    handshake fails, or a communication error occurs mid-stream,
     +    the failure is cached on the driver to avoid retrying and
     +    re-warning on every subsequent file.
     +
     +    diff_process_fill_hunks() is the sole public entry point.  It
     +    handles driver lookup, flag checks, subprocess management, and
     +    error reporting, returning an enum that lets callers distinguish
     +    "hunks populated" from "files equivalent" from "not applicable"
     +    from "tool failure."
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ Documentation/config/diff.adoc: endif::git-diff[]
       	conversion outputs.  See linkgit:gitattributes[5] for details.
       
      +`diff.<driver>.process`::
     -+	The command to run as a long-running diff process.
     -+	The tool communicates via the pkt-line protocol and returns
     -+	hunks that are fed into Git's diff and blame pipelines.
     -+	If the tool returns zero hunks, the file is treated as
     -+	unchanged for both diff output and blame attribution.
     ++	The command to run as a long-running diff process that
     ++	provides hunks to Git's diff pipeline.
      +	See linkgit:gitattributes[5] for details.
      +
       `diff.indentHeuristic`::
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
       algorithm is not passed to the external diff driver.
       
      +Using an external diff process
     -+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     -+
     -+An external tool can provide content-aware line matching by
     -+setting `diff.<name>.process` to the command that runs
     -+the tool.  The tool is a long-running process that communicates via
     -+the pkt-line protocol (described in
     ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ++
     ++If `diff.<name>.process` is defined, Git sends the old and new file
     ++content to an external tool and receives back a list of changed
     ++regions (pairs of line ranges in the old and new file).  Git uses
     ++these instead of its builtin diff algorithm, but still controls
     ++all output formatting, so features like word diff, function context,
     ++color, and blame work normally.  This is achieved by using the
     ++long-running process protocol (described in
      +Documentation/technical/long-running-process-protocol.adoc).
     ++Unlike `diff.<name>.command`, which replaces Git's output entirely,
     ++the diff process feeds results back into the standard pipeline.
     ++
     ++First, in `.gitattributes`, assign the `diff` attribute for paths.
      +
      +------------------------
      +*.c diff=cdiff
      +------------------------
      +
     ++Then, define a "diff.<name>.process" configuration to specify
     ++the diff process command.
     ++
      +----------------------------------------------------------------
      +[diff "cdiff"]
      +  process = /path/to/diff-process-tool
      +----------------------------------------------------------------
      +
     -+The tool receives file pairs and returns hunk descriptors indicating
     -+which lines changed.  Git feeds these hunks into its standard diff
     -+pipeline, so all output features (word diff, function context,
     -+color) work normally.
     -+
     -+If the tool fails or returns an error, Git silently falls back to
     -+the builtin diff algorithm.  If the tool returns invalid hunks
     -+(out of bounds, overlapping), Git also falls back silently.
     -+
     -+The handshake negotiates `version=1` and `capability=hunks`.
     -+Per-file requests send `command=hunks` and `pathname=<path>`,
     -+followed by the old and new file content as packetized data.
     -+The tool responds with lines of the form
     -+`hunk <old_start> <old_count> <new_start> <new_count>`
     -+(1-based line numbers), a flush packet, and `status=success`.
     -+
     -+If the tool returns zero hunks with `status=success`, Git treats
     -+the file as having no changes and produces no diff output.
     -+
     -+Tools should ignore unknown keys in the per-file request to
     -+remain forward-compatible.
     ++When Git encounters the first file that needs to be diffed, it starts
     ++the process and performs the handshake.  In the handshake, the welcome
     ++message sent by Git is "git-diff-client", only version 1 is supported,
     ++and the supported capability is "hunks" (the changed regions
     ++described below).
     ++
     ++For each file, Git sends a list of "key=value" pairs terminated with
     ++a flush packet, followed by the old and new file content as packetized
     ++data, each terminated with a flush packet.  The pathname is relative
     ++to the repository root.  When `diff.<name>.textconv` is also set,
     ++the tool receives the textconv-transformed content rather than the
     ++raw blob.  Git does not send binary files to the diff process.
     ++
     ++-----------------------
     ++packet:          git> command=hunks
     ++packet:          git> pathname=path/file.c
     ++packet:          git> 0000
     ++packet:          git> OLD_CONTENT
     ++packet:          git> 0000
     ++packet:          git> NEW_CONTENT
     ++packet:          git> 0000
     ++-----------------------
     ++
     ++The tool is expected to respond with zero or more hunk lines,
     ++a flush packet, and a status packet terminated with a flush packet.
     ++Each hunk line has the form:
     ++
     ++  `hunk <old_start> <old_count> <new_start> <new_count>`
     ++
     ++where `<old_start>` and `<old_count>` identify a range of lines in
     ++the old file, and `<new_start>` and `<new_count>` identify the
     ++replacement range in the new file.  Start values are 1-based and
     ++counts are non-negative.  Ranges must not extend beyond the end of
     ++the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
     ++at line 3 in the old file were replaced by 4 lines starting at
     ++line 3 in the new file.  An `<old_count>` of 0 means no lines were
     ++removed (pure insertion); a `<new_count>` of 0 means no lines were
     ++added (pure deletion).
     ++
     ++Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
     ++file `"foo\nbar"` both have 2 lines.
     ++
     ++Hunks must be listed in order and must not overlap.  Any line
     ++not covered by a hunk is treated as unchanged, so the total
     ++number of unchanged lines must be the same on both sides.
     ++For example, if the old file has 10 lines and the hunks cover
     ++4 of them (`old_count` values summing to 4), then 6 old lines
     ++are unchanged.  The new file must also have exactly 6 lines
     ++not covered by hunks, so the `new_count` values must sum to
     ++`new_file_lines - 6`.
     ++
     ++-----------------------
     ++packet:          git< hunk 1 3 1 5
     ++packet:          git< hunk 10 2 12 2
     ++packet:          git< 0000
     ++packet:          git< status=success
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++If the tool responds with hunks and "success", Git marks those lines
     ++as changed and feeds them into the standard diff pipeline.  Patch
     ++output features (word diff, function context, color) work normally.
     ++Note that `--stat` and other summary formats use their own diff path
     ++and are not affected by the diff process.
     ++
     ++If no hunk lines precede the flush, followed by "success", Git
     ++treats the files as having no changes: `git diff` produces no output
     ++and `git blame` skips the commit, attributing lines to earlier commits.
     ++
     ++-----------------------
     ++packet:          git< 0000
     ++packet:          git< status=success
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++If the tool returns invalid hunks (out of bounds, overlapping), Git
     ++silently falls back to the builtin diff algorithm.
     ++
     ++In case the tool cannot or does not want to process the content,
     ++it is expected to respond with an "error" status.  Git warns and
     ++falls back to the builtin diff algorithm for this file.  The tool
     ++remains available for subsequent files.
     ++
     ++-----------------------
     ++packet:          git< 0000
     ++packet:          git< status=error
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++In case the tool cannot or does not want to process the content as
     ++well as any future content for the lifetime of the Git process, it
     ++is expected to respond with an "abort" status.  Git silently falls
     ++back to the builtin diff algorithm for this file and does not send
     ++further requests to the tool.
     ++
     ++-----------------------
     ++packet:          git< 0000
     ++packet:          git< status=abort
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++If the tool dies during the communication or does not adhere to the
     ++protocol then Git will stop the process and fall back to the builtin
     ++diff algorithm.  Git warns once and does not restart the process for
     ++subsequent files.
     ++
     ++Tools should ignore unknown keys in the per-file request to remain
     ++forward-compatible.  Future versions of Git may send additional
     ++`command=` values; tools that receive an unrecognized command should
     ++respond with `status=error` rather than terminating.
      +
       Defining a custom hunk-header
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     @@ diff-process.c (new)
      +/*
      + * Diff process backend: communicates with a long-running external
      + * tool via the pkt-line protocol to obtain custom line-matching
     -+ * results.  Unlike textconv, which transforms the displayed content,
     -+ * hunks from a diff process reference original line numbers and
     -+ * the display shows the actual file content.
     ++ * results.  The tool controls which lines are marked as changed
     ++ * while the display shows the file content (after any textconv
     ++ * transformation, if configured).
      + *
      + * Protocol: pkt-line over stdin/stdout, following the pattern of
      + * the long-running filter process protocol (see convert.c).
     @@ diff-process.c (new)
      + *   tool< ... / flush
      + *   tool< status=success / flush
      + *
     -+ * Zero hunks with status=success means the tool considers the
     -+ * files equivalent.  Git will skip the diff for that file.
     ++ * When the tool returns no hunks with status=success, it considers
     ++ * the files equivalent.  Git will skip the diff for that file.
      + */
      +
      +#include "git-compat-util.h"
      +#include "diff-process.h"
     ++#include "diff.h"
     ++#include "gettext.h"
     ++#include "repository.h"
     ++#include "sigchain.h"
      +#include "userdiff.h"
      +#include "sub-process.h"
      +#include "pkt-line.h"
     @@ diff-process.c (new)
      +	unsigned int supported_capabilities;
      +};
      +
     -+static int subprocess_map_initialized;
     -+static struct hashmap subprocess_map;
     -+
      +static int start_diff_process_fn(struct subprocess_entry *subprocess)
      +{
      +	static int versions[] = { 1, 0 };
     @@ diff-process.c (new)
      +		{ NULL, 0 }
      +	};
      +	struct diff_subprocess *entry =
     -+		(struct diff_subprocess *)subprocess;
     ++		container_of(subprocess, struct diff_subprocess, subprocess);
      +
     -+	/* Uses dying pkt-line variant, same as convert.c filters. */
      +	return subprocess_handshake(subprocess, "git-diff",
      +				    versions, NULL,
      +				    capabilities,
      +				    &entry->supported_capabilities);
      +}
      +
     -+static struct diff_subprocess *find_or_start_process(const char *cmd)
     ++static struct diff_subprocess *get_or_launch_process(
     ++		struct userdiff_driver *drv)
      +{
      +	struct diff_subprocess *entry;
      +
     -+	if (!subprocess_map_initialized) {
     -+		subprocess_map_initialized = 1;
     -+		hashmap_init(&subprocess_map, cmd2process_cmp, NULL, 0);
     -+	}
     -+
     -+	entry = (struct diff_subprocess *)
     -+		subprocess_find_entry(&subprocess_map, cmd);
     -+	if (entry)
     -+		return entry;
     ++	if (drv->diff_subprocess)
     ++		return drv->diff_subprocess;
      +
      +	entry = xcalloc(1, sizeof(*entry));
     -+	if (subprocess_start(&subprocess_map, &entry->subprocess,
     -+			     cmd, start_diff_process_fn)) {
     ++	if (subprocess_start_command(&entry->subprocess, drv->process,
     ++				     start_diff_process_fn)) {
      +		free(entry);
     ++		drv->diff_process_failed = 1;
      +		return NULL;
      +	}
      +
     ++	drv->diff_subprocess = entry;
      +	return entry;
      +}
      +
      +static int send_file_content(int fd, const char *buf, long size)
      +{
     -+	int ret;
     ++	int ret = 0;
      +
     ++	if (size < 0)
     ++		return -1;
      +	if (size > 0)
      +		ret = write_packetized_from_buf_no_flush(buf, size, fd);
     -+	else
     -+		ret = 0;
      +	if (ret)
      +		return ret;
      +	return packet_flush_gently(fd);
     @@ diff-process.c (new)
      +{
      +	char *end;
      +
     -+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
     ++	/*
     ++	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
     ++	 * All numbers must be non-negative decimal with no leading
     ++	 * whitespace or sign characters.
     ++	 */
      +	if (!skip_prefix(line, "hunk ", &line))
      +		return -1;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->old_start = strtol(line, &end, 10);
     -+	if (end == line || *end != ' ')
     ++	if (errno || end == line || *end++ != ' ')
      +		return -1;
      +	line = end;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->old_count = strtol(line, &end, 10);
     -+	if (end == line || *end != ' ')
     ++	if (errno || end == line || *end++ != ' ')
      +		return -1;
      +	line = end;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->new_start = strtol(line, &end, 10);
     -+	if (end == line || *end != ' ')
     ++	if (errno || end == line || *end++ != ' ')
      +		return -1;
      +	line = end;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->new_count = strtol(line, &end, 10);
     -+	if (end == line || *end != '\0')
     ++	if (errno || end == line || *end != '\0')
      +		return -1;
      +
      +	return 0;
      +}
      +
     -+int diff_process_get_hunks(struct userdiff_driver *drv,
     -+			   const char *path,
     -+			   const char *old_buf, long old_size,
     -+			   const char *new_buf, long new_size,
     -+			   struct xdl_hunk **hunks_out,
     -+			   size_t *nr_hunks_out)
     ++static enum diff_process_result get_hunks(
     ++		struct userdiff_driver *drv,
     ++		const char *path,
     ++		const char *old_buf, long old_size,
     ++		const char *new_buf, long new_size,
     ++		struct xdl_hunk **hunks_out,
     ++		size_t *nr_hunks_out)
      +{
      +	struct diff_subprocess *backend;
      +	struct child_process *process;
     @@ diff-process.c (new)
      +	int len;
      +	char *line;
      +
     -+	if (!drv || !drv->process)
     -+		return -1;
     -+
     -+	backend = find_or_start_process(drv->process);
     ++	backend = get_or_launch_process(drv);
      +	if (!backend)
     -+		return -1;
     ++		return DIFF_PROCESS_ERROR;
      +
      +	if (!(backend->supported_capabilities & CAP_HUNKS))
     -+		return -1;
     ++		return DIFF_PROCESS_SKIP;
      +
      +	process = subprocess_get_child_process(&backend->subprocess);
      +	fd_in = process->in;
      +	fd_out = process->out;
      +
     ++	sigchain_push(SIGPIPE, SIG_IGN);
     ++
      +	/* Send request */
      +	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
      +	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
      +	    packet_flush_gently(fd_in))
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Send old file content */
      +	if (send_file_content(fd_in, old_buf, old_size))
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Send new file content */
      +	if (send_file_content(fd_in, new_buf, new_size))
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Read hunks until flush packet */
      +	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
      +	       line) {
      +		if (parse_hunk_line(line, &hunk) < 0)
     -+			goto error;
     ++			goto comm_error;
      +		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
      +		hunks[nr_hunks++] = hunk;
      +	}
      +	if (len < 0)
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Read status */
      +	if (subprocess_read_status(fd_out, &status))
     -+		goto error;
     ++		goto comm_error;
     ++
     ++	if (!strcmp(status.buf, "success")) {
     ++		*hunks_out = hunks;
     ++		*nr_hunks_out = nr_hunks;
     ++		strbuf_release(&status);
     ++		sigchain_pop(SIGPIPE);
     ++		return DIFF_PROCESS_OK;
     ++	}
      +
     -+	if (strcmp(status.buf, "success")) {
     -+		if (!strcmp(status.buf, "abort"))
     -+			backend->supported_capabilities &= ~CAP_HUNKS;
     -+		goto error;
     ++	if (!strcmp(status.buf, "abort")) {
     ++		/*
     ++		 * The tool voluntarily withdrew: stop sending requests
     ++		 * but do not warn (this is not a failure).
     ++		 */
     ++		backend->supported_capabilities &= ~CAP_HUNKS;
     ++		free(hunks);
     ++		strbuf_release(&status);
     ++		sigchain_pop(SIGPIPE);
     ++		return DIFF_PROCESS_SKIP;
      +	}
      +
     -+	*hunks_out = hunks;
     -+	*nr_hunks_out = nr_hunks;
     ++	/* status=error or unknown status */
     ++	free(hunks);
      +	strbuf_release(&status);
     -+	return 0;
     -+
     -+error:
     ++	sigchain_pop(SIGPIPE);
     ++	return DIFF_PROCESS_ERROR;
     ++
     ++comm_error:
     ++	/*
     ++	 * Communication failure (broken pipe, malformed response).
     ++	 * Tear down the process and mark as failed so we do not
     ++	 * retry on every subsequent file.
     ++	 */
     ++	drv->diff_process_failed = 1;
     ++	drv->diff_subprocess = NULL;
     ++	subprocess_stop_command(&backend->subprocess);
     ++	free(backend);
      +	free(hunks);
      +	strbuf_release(&status);
     -+	return -1;
     ++	sigchain_pop(SIGPIPE);
     ++	return DIFF_PROCESS_ERROR;
     ++}
     ++
     ++enum diff_process_result diff_process_fill_hunks(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		const mmfile_t *file_a,
     ++		const mmfile_t *file_b,
     ++		xpparam_t *xpp)
     ++{
     ++	struct userdiff_driver *drv;
     ++	struct xdl_hunk *ext_hunks = NULL;
     ++	size_t nr = 0;
     ++	enum diff_process_result res;
     ++
     ++	if (!diffopt || !path)
     ++		return DIFF_PROCESS_SKIP;
     ++	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
     ++		return DIFF_PROCESS_SKIP;
     ++
     ++	drv = userdiff_find_by_path(diffopt->repo->index, path);
     ++	if (!drv || !drv->process)
     ++		return DIFF_PROCESS_SKIP;
     ++	if (drv->diff_process_failed)
     ++		return DIFF_PROCESS_SKIP;
     ++
     ++	res = get_hunks(drv, path,
     ++			file_a->ptr, file_a->size,
     ++			file_b->ptr, file_b->size,
     ++			&ext_hunks, &nr);
     ++	if (res == DIFF_PROCESS_OK) {
     ++		if (!nr) {
     ++			free(ext_hunks);
     ++			return DIFF_PROCESS_EQUIVALENT;
     ++		}
     ++		xpp->external_hunks = ext_hunks;
     ++		xpp->external_hunks_nr = nr;
     ++		return DIFF_PROCESS_OK;
     ++	}
     ++	if (res == DIFF_PROCESS_ERROR) {
     ++		warning(_("diff process '%s' failed for '%s',"
     ++			  " falling back to builtin diff"),
     ++			drv->process, path);
     ++		return DIFF_PROCESS_ERROR;
     ++	}
     ++	return DIFF_PROCESS_SKIP;
      +}
      
       ## diff-process.h (new) ##
     @@ diff-process.h (new)
      +#ifndef DIFF_PROCESS_H
      +#define DIFF_PROCESS_H
      +
     -+struct userdiff_driver;
     -+struct xdl_hunk;
     ++#include "xdiff/xdiff.h"
     ++
     ++struct diff_options;
     ++
     ++enum diff_process_result {
     ++	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
     ++	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
     ++	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
     ++	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
     ++};
      +
      +/*
     -+ * Query a diff process for hunks describing the changes
     -+ * between old_buf and new_buf.
     ++ * Consult the diff process configured for 'path' and populate
     ++ * xpp->external_hunks with the returned hunks.
      + *
     -+ * The backend is a long-running subprocess configured via
     -+ * diff.<driver>.process.  It receives file content via
     -+ * pkt-line and returns hunks with 1-based line numbers.
     ++ * Handles driver lookup, flag checks (--no-ext-diff,
     ++ * --diff-algorithm), subprocess management, and error reporting.
      + *
     -+ * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
     -+ * array (caller must free) and returns 0.
     ++ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
     ++ * The caller owns xpp->external_hunks and must free() it.
      + *
     -+ * On failure, returns -1.  The caller should fall back to the
     -+ * builtin diff algorithm.
     ++ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
     ++ * (files are considered identical); caller should skip diff/blame.
     ++ * Returns DIFF_PROCESS_SKIP when no process applies; caller
     ++ * should use the builtin diff algorithm.
     ++ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
     ++ * caller should fall back to the builtin diff algorithm.
      + */
     -+int diff_process_get_hunks(struct userdiff_driver *drv,
     -+			   const char *path,
     -+			   const char *old_buf, long old_size,
     -+			   const char *new_buf, long new_size,
     -+			   struct xdl_hunk **hunks_out,
     -+			   size_t *nr_hunks_out);
     ++enum diff_process_result diff_process_fill_hunks(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		const mmfile_t *file_a,
     ++		const mmfile_t *file_b,
     ++		xpparam_t *xpp);
      +
      +#endif /* DIFF_PROCESS_H */
      
     @@ diff.c
       #include "submodule.h"
       #include "hashmap.h"
       #include "mem-pool.h"
     -@@ diff.c: static void builtin_diff(const char *name_a,
     - 		xpparam_t xpp;
     - 		xdemitconf_t xecfg;
     - 		struct emit_callback ecbdata;
     -+		struct xdl_hunk *ext_hunks = NULL;
     - 		unsigned ws_rule;
     - 		const struct userdiff_funcname *pe;
     - 
      @@ diff.c: static void builtin_diff(const char *name_a,
       		xpp.ignore_regex_nr = o->ignore_regex_nr;
       		xpp.anchors = o->anchors;
       		xpp.anchors_nr = o->anchors_nr;
      +
     -+		if (!o->ignore_driver_algorithm &&
     -+		    one->driver && one->driver->process) {
     -+			size_t ext_hunks_nr = 0;
     -+			if (!diff_process_get_hunks(
     -+				    one->driver, name_a,
     -+				    mf1.ptr, mf1.size,
     -+				    mf2.ptr, mf2.size,
     -+				    &ext_hunks, &ext_hunks_nr)) {
     -+				if (!ext_hunks_nr)
     -+					goto free_ab_and_return;
     -+				xpp.external_hunks = ext_hunks;
     -+				xpp.external_hunks_nr = ext_hunks_nr;
     -+			} else {
     -+				warning(_("diff process failed for '%s',"
     -+					  " falling back to builtin diff"),
     -+					name_a);
     -+			}
     ++		if (diff_process_fill_hunks(o, name_a,
     ++					    &mf1, &mf2, &xpp)
     ++		    == DIFF_PROCESS_EQUIVALENT) {
     ++			if (textconv_one)
     ++				free(mf1.ptr);
     ++			if (textconv_two)
     ++				free(mf2.ptr);
     ++			goto free_ab_and_return;
      +		}
      +
       		xecfg.ctxlen = o->context;
     @@ diff.c: static void builtin_diff(const char *name_a,
       		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
       					 &ecbdata, &xpp, &xecfg))
       			die("unable to generate diff for %s", one->path);
     -+		free(ext_hunks);
     ++		free(xpp.external_hunks);
       		if (o->word_diff)
       			free_diff_words_data(&ecbdata);
       		if (textconv_one)
      
     + ## diff.h ##
     +@@ diff.h: struct diff_flags {
     + 	 */
     + 	unsigned allow_external;
     + 
     ++	/** Disables diff.<driver>.process. */
     ++	unsigned no_diff_process;
     ++
     + 	/**
     + 	 * For communication between the calling program and the options parser;
     + 	 * tell the calling program to signal the presence of difference using
     +
     + ## meson.build ##
     +@@ meson.build: libgit_sources = [
     +   'diff-merges.c',
     +   'diff-lib.c',
     +   'diff-no-index.c',
     ++  'diff-process.c',
     +   'diff.c',
     +   'diffcore-break.c',
     +   'diffcore-delta.c',
     +
       ## t/.gitattributes ##
      @@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
       /t8005/*.txt eol=lf
     @@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
       /t0040*.sh whitespace=-indent-with-non-tab
      +/t4080-diff-process.sh whitespace=-indent-with-non-tab
      
     + ## t/meson.build ##
     +@@ t/meson.build: integration_tests = [
     +   't4072-diff-max-depth.sh',
     +   't4073-diff-stat-name-width.sh',
     +   't4074-diff-shifted-matched-group.sh',
     ++  't4080-diff-process.sh',
     +   't4100-apply-stat.sh',
     +   't4101-apply-nonl.sh',
     +   't4102-apply-rename.sh',
     +
       ## t/t4080-diff-process.sh (new) ##
      @@
      +#!/bin/sh
     @@ t/t4080-diff-process.sh (new)
      +#   whole-file  - report all lines as changed (default)
      +#   fixed-hunk  - always report hunk 5 2 5 2
      +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
     -+#   zero-hunk   - return zero hunks (files considered equivalent)
     ++#   bad-sync    - report hunk with mismatched unchanged totals
     ++#   overlap     - report two overlapping hunks
     ++#   no-hunks   - return no hunks (files considered equivalent)
      +#   error       - return status=error for every request
      +#   abort       - return status=abort for every request
      +#   crash       - read one request then exit without responding
     @@ t/t4080-diff-process.sh (new)
      +	    if cmd is None: sys.exit(0)
      +	    old = read_content()
      +	    new = read_content()
     -+	    log(f"command={cmd} pathname={pathname}")
     ++	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
     ++	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
     ++	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
      +
      +	    if mode == "error":
      +	        write_flush()
     @@ t/t4080-diff-process.sh (new)
      +	            write_pkt("hunk 5 2 5 2")
      +	        elif mode == "bad-hunk":
      +	            write_pkt("hunk 999 1 999 1")
     -+	        elif mode == "zero-hunk":
     ++	        elif mode == "bad-sync":
     ++	            write_pkt("hunk 1 2 1 1")
     ++	        elif mode == "overlap":
     ++	            write_pkt("hunk 1 5 1 5")
     ++	            write_pkt("hunk 3 2 3 2")
     ++	        elif mode == "no-hunks":
      +	            pass
      +	        else:
     -+	            ol = len(old.split(b"\n"))
     -+	            nl = len(new.split(b"\n"))
     ++	            ol = old.count(b"\n")
     ++	            nl = new.count(b"\n")
      +	            write_pkt(f"hunk 1 {ol} 1 {nl}")
      +	        write_flush()
      +	        write_pkt("status=success")
     @@ t/t4080-diff-process.sh (new)
      +	setup_backend &&
      +	echo "*.c diff=cdiff" >.gitattributes &&
      +	git add .gitattributes &&
     -+	git commit -m "initial"
     -+'
      +
     -+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
     ++	# boundary.c: 10 lines, changes at 5-6 and 9-10.
     ++	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
      +	cat >boundary.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh (new)
      +	OLD10
      +	EOF
      +	git add boundary.c &&
     -+	git commit -m "add boundary.c" &&
      +
     ++	# worddiff.c: single-line function, value changes 1 -> 999.
     ++	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
     ++	cat >worddiff.c <<-\EOF &&
     ++	int value(void) { return 1; }
     ++	EOF
     ++	git add worddiff.c &&
     ++
     ++	# newfile.c: single-line function, value changes 42 -> 99.
     ++	# Used by: new file, --exit-code, multiple drivers.
     ++	cat >newfile.c <<-\EOF &&
     ++	int new_func(void) { return 42; }
     ++	EOF
     ++	git add newfile.c &&
     ++
     ++	# logtest.c: single-line function for log/format-patch tests.
     ++	# Needs two commits so log -1 has a diff.
     ++	cat >logtest.c <<-\EOF &&
     ++	int logfunc(void) { return 1; }
     ++	EOF
     ++	git add logtest.c &&
     ++
     ++	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
     ++	cat >one.c <<-\EOF &&
     ++	int first(void) { return 1; }
     ++	EOF
     ++	cat >two.c <<-\EOF &&
     ++	int second(void) { return 2; }
     ++	EOF
     ++	git add one.c two.c &&
     ++
     ++	git commit -m "initial" &&
     ++
     ++	# Second commit for logtest.c (so log -1 has something to show).
     ++	cat >logtest.c <<-\EOF &&
     ++	int logfunc(void) { return 2; }
     ++	EOF
     ++	git add logtest.c &&
     ++	git commit -m "change logtest.c" &&
     ++
     ++	# Working tree modifications (not committed).
      +	cat >boundary.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh (new)
      +	NEW10
      +	EOF
      +
     ++	cat >worddiff.c <<-\EOF &&
     ++	int value(void) { return 999; }
     ++	EOF
     ++
     ++	cat >newfile.c <<-\EOF &&
     ++	int new_func(void) { return 99; }
     ++	EOF
     ++
     ++	cat >one.c <<-\EOF &&
     ++	int first(void) { return 10; }
     ++	EOF
     ++
     ++	cat >two.c <<-\EOF
     ++	int second(void) { return 20; }
     ++	EOF
     ++'
     ++
     ++#
     ++# Core behavior: the tool controls which lines are marked as changed.
     ++#
     ++
     ++test_expect_success PYTHON 'diff process hunk boundaries affect output' '
      +	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
      +	# only reports lines 5-6 as changed.  Lines 9-10 should not
      +	# appear as changed in the output.
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "^-OLD6" actual &&
      +	test_grep "^+NEW5" actual &&
      +	test_grep "^+NEW6" actual &&
     -+	! test_grep "^-OLD9" actual &&
     -+	! test_grep "^-OLD10" actual &&
     -+	! test_grep "^+NEW9" actual &&
     -+	! test_grep "^+NEW10" actual
     ++	test_grep ! "^-OLD9" actual &&
     ++	test_grep ! "^-OLD10" actual &&
     ++	test_grep ! "^+NEW9" actual &&
     ++	test_grep ! "^+NEW10" actual
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on tool error status' '
     ++test_expect_success PYTHON 'diff process works with new file' '
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
     -+		diff boundary.c >actual &&
     -+	# Fallback produces the full builtin diff (both change regions).
     -+	test_grep "^-OLD5" actual &&
     -+	test_grep "^+NEW5" actual &&
     -+	test_grep "^-OLD9" actual &&
     -+	test_grep "^+NEW9" actual &&
     -+	# Tool was contacted (it replied with error, not crash).
     -+	test_grep "command=hunks pathname=boundary.c" backend.log
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- newfile.c >actual 2>stderr &&
     ++	test_grep "return 99" actual &&
     ++	test_grep "pathname=newfile.c" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on bad hunks' '
     -+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
     -+		diff boundary.c >actual &&
     -+	test_grep "^-OLD5" actual &&
     -+	test_grep "^+NEW5" actual &&
     -+	test_grep "^-OLD9" actual &&
     -+	test_grep "^+NEW9" actual
     ++test_expect_success PYTHON 'diff process works with added file (empty old side)' '
     ++	cat >added.c <<-\EOF &&
     ++	int added(void) { return 1; }
     ++	EOF
     ++	git add added.c &&
     ++
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --cached -- added.c >actual 2>stderr &&
     ++	test_grep "added" actual &&
     ++	test_grep "pathname=added.c" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on tool crash' '
     -+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
     -+		diff boundary.c >actual &&
     -+	test_grep "^-OLD5" actual &&
     -+	test_grep "^+NEW5" actual &&
     -+	test_grep "^-OLD9" actual &&
     -+	test_grep "^+NEW9" actual
     ++test_expect_success PYTHON 'diff process skipped for binary files' '
     ++	printf "\\0binary" >binary.c &&
     ++	git add binary.c &&
     ++	git commit -m "add binary" &&
     ++	printf "\\0changed" >binary.c &&
     ++
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- binary.c >actual &&
     ++	test_grep "Binary files" actual &&
     ++	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process abort disables for session' '
     -+	cat >abort1.c <<-\EOF &&
     -+	int first(void) { return 1; }
     -+	EOF
     -+	cat >abort2.c <<-\EOF &&
     -+	int second(void) { return 2; }
     -+	EOF
     -+	git add abort1.c abort2.c &&
     -+	git commit -m "add abort files" &&
     ++test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
     ++	echo "not tracked by cdiff" >unmatched.txt &&
     ++	git add unmatched.txt &&
     ++	git commit -m "add unmatched.txt" &&
      +
     -+	cat >abort1.c <<-\EOF &&
     -+	int first(void) { return 10; }
     -+	EOF
     -+	cat >abort2.c <<-\EOF &&
     -+	int second(void) { return 20; }
     -+	EOF
     ++	echo "modified" >unmatched.txt &&
      +
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
     -+		diff -- abort1.c abort2.c >actual &&
     -+	# Both files should still produce diff output via fallback.
     -+	test_grep "return 10" actual &&
     -+	test_grep "return 20" actual &&
     -+	# The tool aborts on the first file and git clears its
     -+	# capability.  The second file never contacts the tool,
     -+	# so the log should have exactly one entry, not two.
     -+	test_grep "command=hunks" backend.log >matches &&
     -+	test_line_count = 1 matches
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- unmatched.txt >actual &&
     ++	test_grep "modified" actual &&
     ++	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process handles multiple files' '
     -+	cat >multi1.c <<-\EOF &&
     -+	int one(void) { return 1; }
     ++test_expect_success PYTHON 'multiple drivers use separate processes' '
     ++	echo "*.h diff=hdiff" >>.gitattributes &&
     ++	git add .gitattributes &&
     ++
     ++	cat >multi.h <<-\EOF &&
     ++	int header(void) { return 1; }
      +	EOF
     -+	cat >multi2.c <<-\EOF &&
     -+	int two(void) { return 2; }
     ++	git add multi.h &&
     ++	git commit -m "add multi.h" &&
     ++
     ++	cat >multi.h <<-\EOF &&
     ++	int header(void) { return 2; }
      +	EOF
     -+	git add multi1.c multi2.c &&
     -+	git commit -m "add multi files" &&
      +
     -+	cat >multi1.c <<-\EOF &&
     -+	int one(void) { return 10; }
     ++	rm -f backend-c.log backend-h.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
     ++	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
     ++		diff -- newfile.c multi.h >actual 2>stderr &&
     ++	test_grep "pathname=newfile.c" backend-c.log &&
     ++	test_grep "pathname=multi.h" backend-h.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process works alongside textconv' '
     ++	write_script uppercase-filter <<-\EOF &&
     ++	tr "a-z" "A-Z" <"$1"
      +	EOF
     -+	cat >multi2.c <<-\EOF &&
     -+	int two(void) { return 20; }
     ++
     ++	cat >textconv.c <<-\EOF &&
     ++	hello world
     ++	EOF
     ++	git add textconv.c &&
     ++	git commit -m "add textconv.c" &&
     ++
     ++	cat >textconv.c <<-\EOF &&
     ++	goodbye world
      +	EOF
      +
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     -+		diff -- multi1.c multi2.c >actual &&
     -+	test_grep "return 10" actual &&
     -+	test_grep "return 20" actual &&
     -+	test_grep "pathname=multi1.c" backend.log &&
     -+	test_grep "pathname=multi2.c" backend.log
     ++	git -c diff.cdiff.textconv="./uppercase-filter" \
     ++	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- textconv.c >actual 2>stderr &&
     ++	# The diff process receives textconv-transformed (uppercase) content.
     ++	test_grep "pathname=textconv.c" backend.log &&
     ++	test_grep "old=HELLO WORLD" backend.log &&
     ++	test_grep "new=GOODBYE WORLD" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     ++#
     ++# Downstream features: word diff, log, equivalent files, exit code.
     ++#
     ++
      +test_expect_success PYTHON 'diff process with --word-diff' '
     -+	cat >worddiff.c <<-\EOF &&
     -+	int value(void) { return 1; }
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --word-diff worddiff.c >actual 2>stderr &&
     ++	test_grep "\[-1;-\]" actual &&
     ++	test_grep "{+999;+}" actual &&
     ++	test_grep "pathname=worddiff.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process works with git log -p' '
     ++	# With no-hunks mode, the tool says the files are equivalent,
     ++	# so log -p should show the commit but no diff content.
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		log -1 -p -- logtest.c >actual 2>stderr &&
     ++	test_grep "change logtest.c" actual &&
     ++	test_grep ! "return 2" actual &&
     ++	test_grep "command=hunks pathname=logtest.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
     ++	cat >nohunks.c <<-\EOF &&
     ++	int zero(void) { return 0; }
      +	EOF
     -+	git add worddiff.c &&
     -+	git commit -m "add worddiff.c" &&
     ++	git add nohunks.c &&
     ++	git commit -m "add nohunks.c" &&
      +
     -+	cat >worddiff.c <<-\EOF &&
     -+	int value(void) { return 999; }
     ++	cat >nohunks.c <<-\EOF &&
     ++	int zero(void) { return 999; }
      +	EOF
      +
     -+	git -c diff.cdiff.process="$BACKEND" \
     -+		diff --word-diff worddiff.c >actual &&
     -+	test_grep "\[-1;-\]" actual &&
     -+	test_grep "{+999;+}" actual
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff nohunks.c >actual &&
     ++	test_must_be_empty actual
     ++'
     ++
     ++test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff --exit-code nohunks.c
     ++'
     ++
     ++test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
     ++	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
     ++		diff --exit-code newfile.c
      +'
      +
     ++#
     ++# Bypass mechanisms: flags and commands that skip the diff process.
     ++#
     ++
      +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     @@ t/t4080-diff-process.sh (new)
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process works with git log -p' '
     -+	cat >logtest.c <<-\EOF &&
     -+	int logfunc(void) { return 1; }
     -+	EOF
     -+	git add logtest.c &&
     -+	git commit -m "add logtest.c" &&
     ++test_expect_success PYTHON 'diff process not used by --stat' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --stat worddiff.c >actual &&
     ++	test_grep "worddiff.c" actual &&
     ++	test_path_is_missing backend.log
     ++'
      +
     -+	cat >logtest.c <<-\EOF &&
     -+	int logfunc(void) { return 2; }
     -+	EOF
     -+	git add logtest.c &&
     -+	git commit -m "change logtest.c" &&
     ++#
     ++# Error handling and fallback.
     ++#
      +
     ++test_expect_success PYTHON 'diff process fallback on tool error status' '
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     -+		log -1 -p -- logtest.c >actual &&
     -+	test_grep "return 2" actual &&
     -+	test_grep "command=hunks pathname=logtest.c" backend.log
     ++	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	# Fallback produces the full builtin diff (both change regions).
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	# Tool was contacted (it replied with error, not crash).
     ++	test_grep "command=hunks pathname=boundary.c" backend.log &&
     ++	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
     -+	cat >zerohunk.c <<-\EOF &&
     -+	int zero(void) { return 0; }
     ++test_expect_success PYTHON 'diff process error keeps tool available for next file' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
     ++		diff -- one.c two.c >actual 2>stderr &&
     ++	# Unlike abort, error keeps the tool available: both files
     ++	# are sent to the tool (and both fall back).
     ++	test_grep "pathname=one.c" backend.log &&
     ++	test_grep "pathname=two.c" backend.log &&
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual
     ++'
     ++
     ++test_expect_success PYTHON 'diff process abort disables for session' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
     ++		diff -- one.c two.c >actual &&
     ++	# Both files should still produce diff output via fallback.
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
     ++	# The tool aborts on the first file and git clears its
     ++	# capability.  The second file never contacts the tool.
     ++	test_grep "pathname=one.c" backend.log &&
     ++	test_grep ! "pathname=two.c" backend.log
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on tool crash' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=crash" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	# Crash is a communication failure, so a warning is emitted.
     ++	test_grep "diff process.*failed" stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process startup failure only warns once' '
     ++	git -c diff.cdiff.process="/nonexistent/tool" \
     ++		diff -- one.c two.c >actual 2>stderr &&
     ++	# Both files produce diff output via fallback.
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
     ++	# Sentinel prevents repeated warnings: only one, not one per file.
     ++	test_grep "diff process.*failed" stderr >warnings &&
     ++	test_line_count = 1 warnings
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on bad hunks' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	# Invalid hunks are caught by xdiff validation, not the
     ++	# protocol layer, so no warning is emitted.
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
     ++	cat >synctest.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
      +	EOF
     -+	git add zerohunk.c &&
     -+	git commit -m "add zerohunk.c" &&
     ++	git add synctest.c &&
     ++	git commit -m "add synctest.c" &&
      +
     -+	cat >zerohunk.c <<-\EOF &&
     -+	int zero(void) { return 999; }
     ++	cat >synctest.c <<-\EOF &&
     ++	line1
     ++	changed
     ++	line3
      +	EOF
      +
     -+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
     -+		diff zerohunk.c >actual &&
     -+	test_must_be_empty actual
     ++	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
     ++	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
     ++	# The synchronization invariant fails and git falls back.
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
     ++		diff synctest.c >actual 2>stderr &&
     ++	test_grep "changed" actual
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     ++	# boundary.c has 10 lines, so both hunks are in bounds
     ++	# but they overlap at lines 3-5, triggering the ordering check.
     ++	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "NEW5" actual
      +'
      +
      +test_done
     +
     + ## userdiff.h ##
     +@@
     + 
     + #include "notes-cache.h"
     + 
     ++struct diff_subprocess;
     + struct index_state;
     + struct repository;
     + 
     +@@ userdiff.h: struct userdiff_driver {
     + 	int textconv_want_cache;
     + 	const char *process;
     + 	char *process_owned;
     ++	struct diff_subprocess *diff_subprocess;
     ++	unsigned diff_process_failed : 1;
     + };
     + enum userdiff_driver_type {
     + 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
 -:  ---------- > 5:  f4fd9aa682 diff: bypass diff process with --no-ext-diff and in format-patch
 4:  39ff53acef ! 6:  370e766978 blame: consult diff process for zero-hunk detection
     @@ Metadata
      Author: Michael Montalbo <mmontalbo@gmail.com>
      
       ## Commit message ##
     -    blame: consult diff process for zero-hunk detection
     +    blame: consult diff process for no-hunk detection
      
          When a diff process is configured via diff.<driver>.process,
          consult it during blame's per-commit diffing.  If the process
     -    returns zero hunks for a commit's changes to a file, treat the
     +    returns no hunks for a commit's changes to a file, treat the
          commit as having no changes, causing blame to attribute lines
          to earlier commits.
      
     +    The consultation happens at the pass_blame_to_parent() callsite
     +    using diff_process_fill_hunks(), matching how builtin_diff() in
     +    diff.c uses the same function.  A new diff_hunks_xpp() variant
     +    accepts a pre-populated xpparam_t for this callsite, while the
     +    existing diff_hunks() retains its original signature and behavior.
     +    The copy-detection callsite is unaffected since it does not use
     +    the diff process.
     +
          The subprocess is long-running (one startup cost amortized
          across the blame traversal), but each commit in the file's
          history incurs a round-trip to the tool.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     - ## Documentation/gitattributes.adoc ##
     -@@ Documentation/gitattributes.adoc: The tool responds with lines of the form
     - 
     - If the tool returns zero hunks with `status=success`, Git treats
     - the file as having no changes and produces no diff output.
     -+`git blame` also consults the diff process and skips commits
     -+where it reports zero hunks, attributing lines to earlier commits
     -+instead.
     - 
     - Tools should ignore unknown keys in the per-file request to
     - remain forward-compatible.
     -
       ## blame.c ##
      @@
       #include "tag.h"
       #include "trace2.h"
       #include "blame.h"
      +#include "diff-process.h"
     -+#include "userdiff.h"
     ++#include "xdiff-interface.h"
       #include "alloc.h"
       #include "commit-slab.h"
       #include "bloom.h"
      @@ blame.c: static struct commit *fake_working_tree_commit(struct repository *r,
       
       
     - static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     + 
     +-static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
      -		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     -+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data,
     -+		      int xdl_opts, struct index_state *istate,
     -+		      const char *path)
     ++static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
     ++			  xdl_emit_hunk_consume_func_t hunk_func,
     ++			  void *cb_data, xpparam_t *xpp)
       {
     - 	xpparam_t xpp = {0};
     +-	xpparam_t xpp = {0};
       	xdemitconf_t xecfg = {0};
       	xdemitcb_t ecb = {NULL};
     -+	struct xdl_hunk *ext_hunks = NULL;
     -+	int ret;
       
     - 	xpp.flags = xdl_opts;
     +-	xpp.flags = xdl_opts;
       	xecfg.hunk_func = hunk_func;
       	ecb.priv = cb_data;
      -	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
     ++	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
     ++}
      +
     -+	if (path && istate) {
     -+		struct userdiff_driver *drv;
     -+		drv = userdiff_find_by_path(istate, path);
     -+		if (drv && drv->process) {
     -+			size_t nr = 0;
     -+			if (!diff_process_get_hunks(drv, path,
     -+						    file_a->ptr, file_a->size,
     -+						    file_b->ptr, file_b->size,
     -+						    &ext_hunks, &nr)) {
     -+				if (!nr) {
     -+					/*
     -+					 * Zero hunks: the diff process
     -+					 * considers these files equivalent.
     -+					 * Skip so blame looks past this
     -+					 * commit.
     -+					 */
     -+					return 0;
     -+				}
     -+				xpp.external_hunks = ext_hunks;
     -+				xpp.external_hunks_nr = nr;
     -+			}
     -+		}
     -+	}
     ++static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     ++		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     ++{
     ++	xpparam_t xpp = {0};
      +
     -+	ret = xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
     -+	free(ext_hunks);
     -+	return ret;
     ++	xpp.flags = xdl_opts;
     ++	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
       }
       
       static const char *get_next_line(const char *start, const char *end)
     +@@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
     + 				 struct blame_origin *parent, int ignore_diffs)
     + {
     + 	mmfile_t file_p, file_o;
     ++	xpparam_t xpp = {0};
     + 	struct blame_chunk_cb_data d;
     + 	struct blame_entry *newdest = NULL;
     + 
      @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       			 &sb->num_read_blob, ignore_diffs);
       	sb->num_get_patch++;
       
      -	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
     -+	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts,
     -+		       sb->revs->diffopt.repo->index, target->path))
     - 		die("unable to generate diff (%s -> %s)",
     - 		    oid_to_hex(&parent->commit->object.oid),
     - 		    oid_to_hex(&target->commit->object.oid));
     -@@ blame.c: static void find_copy_in_blob(struct blame_scoreboard *sb,
     - 	 * file_p partially may match that image.
     - 	 */
     - 	memset(split, 0, sizeof(struct blame_entry [3]));
     --	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts))
     -+	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts,
     -+		       NULL, NULL))
     - 		die("unable to generate diff (%s)",
     - 		    oid_to_hex(&parent->commit->object.oid));
     - 	/* remainder, if any, all match the preimage */
     +-		die("unable to generate diff (%s -> %s)",
     +-		    oid_to_hex(&parent->commit->object.oid),
     +-		    oid_to_hex(&target->commit->object.oid));
     ++	xpp.flags = sb->xdl_opts;
     ++	/*
     ++	 * If the diff process considers the files equivalent,
     ++	 * skip the diff so blame looks past this commit.
     ++	 */
     ++	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
     ++				    &file_p, &file_o, &xpp)
     ++	    != DIFF_PROCESS_EQUIVALENT) {
     ++		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
     ++				   &d, &xpp))
     ++			die("unable to generate diff (%s -> %s)",
     ++			    oid_to_hex(&parent->commit->object.oid),
     ++			    oid_to_hex(&target->commit->object.oid));
     ++	}
     ++	free(xpp.external_hunks);
     + 	/* The rest are the same as the parent */
     + 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
     + 		    parent, target, 0);
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
     - 	test_must_be_empty actual
     +@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     + 	test_grep "NEW5" actual
       '
       
     -+test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' '
     ++#
     ++# Blame integration.
     ++#
     ++
     ++test_expect_success PYTHON 'blame uses tool-provided hunks' '
     ++	cat >blame-hunk.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
     ++	line4
     ++	original5
     ++	original6
     ++	line7
     ++	line8
     ++	line9
     ++	line10
     ++	EOF
     ++	git add blame-hunk.c &&
     ++	git commit -m "add blame-hunk.c" &&
     ++	ORIG=$(git rev-parse --short HEAD) &&
     ++
     ++	cat >blame-hunk.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
     ++	line4
     ++	changed5
     ++	changed6
     ++	line7
     ++	line8
     ++	changed9
     ++	changed10
     ++	EOF
     ++	git add blame-hunk.c &&
     ++	git commit -m "change blame-hunk.c" &&
     ++	CHANGE=$(git rev-parse --short HEAD) &&
     ++
     ++	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
     ++	# so blame should attribute lines 9-10 to the original commit
     ++	# even though the builtin diff would show them as changed.
     ++	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
     ++		blame blame-hunk.c >actual &&
     ++	sed -n "9p" actual >line9 &&
     ++	sed -n "10p" actual >line10 &&
     ++	test_grep "$ORIG" line9 &&
     ++	test_grep "$ORIG" line10 &&
     ++	sed -n "5p" actual >line5 &&
     ++	sed -n "6p" actual >line6 &&
     ++	test_grep "$CHANGE" line5 &&
     ++	test_grep "$CHANGE" line6
     ++'
     ++
     ++test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
      +	cat >blame.c <<-\EOF &&
      +	int main(void)
      +	{
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
      +	EOF
      +	git add blame.c &&
      +	git commit -m "add blame.c" &&
     ++	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
      +
      +	cat >blame.c <<-\EOF &&
      +	int main(void)
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
      +	git commit -m "reformat blame.c" &&
      +	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
      +
     -+	# Without zero-hunk mode, blame attributes the change.
     ++	# Without no-hunks mode, blame attributes the change.
      +	git blame blame.c >without &&
      +	test_grep "$BLAME_COMMIT" without &&
      +
     -+	# With zero-hunk mode, the process considers the files equivalent
     -+	# and blame skips the reformat commit.
     -+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
     ++	# With no-hunks mode, the process considers the files equivalent
     ++	# and blame skips the reformat commit, attributing to the original.
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
      +		blame blame.c >with &&
     -+	! test_grep "$BLAME_COMMIT" with
     ++	test_grep ! "$BLAME_COMMIT" with &&
     ++	test_grep "$ORIG_COMMIT" with
      +'
      +
     ++test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		blame --no-ext-diff blame.c >actual &&
     ++	# Without the process, blame attributes the reformat commit normally.
     ++	test_grep "$BLAME_COMMIT" actual &&
     ++	test_path_is_missing backend.log
     ++'
     ++
     ++test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
     ++	# fixed-hunk mode would narrow blame to lines 5-6, but
     ++	# --no-ext-diff should bypass it and use the builtin diff.
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
     ++		blame --no-ext-diff blame-hunk.c >actual &&
     ++	# Builtin diff attributes lines 9-10 to the change commit.
     ++	sed -n "9p" actual >line9 &&
     ++	test_grep "$CHANGE" line9 &&
     ++	test_path_is_missing backend.log
     ++'
      +
       test_done

-- 
gitgitgadget

[PATCH v3 1/6] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm; this re-runs xdl_prepare_env() since
the first call may have dirtied the changed[] arrays.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 13 ++++++++
 xdiff/xdiffi.c    | 85 +++++++++++++++++++++++++++++++++++++++++++++--
 xdiff/xprepare.c  | 10 ++++++
 xdiff/xprepare.h  |  1 +
 5 files changed, 113 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index f043330f2a..9542c0bcc2 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..a7e8502e4c 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based, matching unified diff convention.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +97,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
+	struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index 5455b4690d..b0a01ca856 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,97 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long total_old = 0, total_new = 0;
+
+	/*
+	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
+	 * Clear them so only the external hunks are marked.
+	 */
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0)
+			return -1;
+		if (h->old_start < 1 || h->new_start < 1)
+			return -1;
+
+		/*
+		 * Range must fit: start + count - 1 <= nrec,
+		 * rewritten to avoid overflow.  Same for both sides.
+		 *
+		 * When count is 0 (pure insert/delete) the check
+		 * reduces to 0 > nrec - start + 1, which rejects
+		 * start > nrec + 1 and allows start == nrec + 1
+		 * (the position after the last line).
+		 */
+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
+			return -1;
+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
+			return -1;
+
+		/* Ordering: no overlap with previous hunk (adjacent is OK) */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+		total_old += h->old_count;
+		total_new += h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 */
+	if ((long)xe->xdf1.nrec - total_old !=
+	    (long)xe->xdf2.nrec - total_new)
+		return -1;
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index cd4fc405eb..4645a9a746 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -432,3 +432,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v3 2/6] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index fe710a68bf..81c0bebcce 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v3 3/6] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and
managing a hashmap that indexes running processes by command
string.  The hashmap suits callers like convert.c where many
files may share one filter process looked up by name, but
callers that manage process lifetime through their own data
structures do not need it.

Extract subprocess_start_command() and subprocess_stop_command()
so callers can reuse the child process setup and handshake
machinery without maintaining a hashmap.  subprocess_start()
and subprocess_stop() become thin wrappers that add hashmap
operations on top.

No functional change for existing callers.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 29 ++++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 83bf0a0e82..33b0bbc831 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
+
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
 
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,27 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err) {
+		return err;
+	}
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
gitgitgadget

[PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference line numbers in the content.
When textconv is also configured, the tool receives the
textconv-transformed content.  The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
normally; summary formats like --stat use their own diff path
and are not affected.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet (success, error, or abort).  On error, Git warns and falls
back to the builtin diff algorithm for that file.  On abort, Git
silently falls back for the current file and stops sending further
requests to the tool for the remainder of the session.

When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
This also means --exit-code reports no changes for that file.

The subprocess is stored on the userdiff_driver struct and
launched on first use.  If the process fails to start, the
handshake fails, or a communication error occurs mid-stream,
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.

diff_process_fill_hunks() is the sole public entry point.  It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   5 +
 Documentation/gitattributes.adoc | 139 ++++++++
 Makefile                         |   1 +
 diff-process.c                   | 288 +++++++++++++++++
 diff-process.h                   |  39 +++
 diff.c                           |  13 +
 diff.h                           |   3 +
 meson.build                      |   1 +
 t/.gitattributes                 |   1 +
 t/meson.build                    |   1 +
 t/t4080-diff-process.sh          | 538 +++++++++++++++++++++++++++++++
 userdiff.h                       |   3 +
 12 files changed, 1032 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..ac0635bb3b 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f20041a323..1700bd8e97 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,145 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If `diff.<name>.process` is defined, Git sends the old and new file
+content to an external tool and receives back a list of changed
+regions (pairs of line ranges in the old and new file).  Git uses
+these instead of its builtin diff algorithm, but still controls
+all output formatting, so features like word diff, function context,
+color, and blame work normally.  This is achieved by using the
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
+the diff process feeds results back into the standard pipeline.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+Then, define a "diff.<name>.process" configuration to specify
+the diff process command.
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+When Git encounters the first file that needs to be diffed, it starts
+the process and performs the handshake.  In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
+described below).
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
+data, each terminated with a flush packet.  The pathname is relative
+to the repository root.  When `diff.<name>.textconv` is also set,
+the tool receives the textconv-transformed content rather than the
+raw blob.  Git does not send binary files to the diff process.
+
+-----------------------
+packet:          git> command=hunks
+packet:          git> pathname=path/file.c
+packet:          git> 0000
+packet:          git> OLD_CONTENT
+packet:          git> 0000
+packet:          git> NEW_CONTENT
+packet:          git> 0000
+-----------------------
+
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
+
+  `hunk <old_start> <old_count> <new_start> <new_count>`
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
+replacement range in the new file.  Start values are 1-based and
+counts are non-negative.  Ranges must not extend beyond the end of
+the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
+at line 3 in the old file were replaced by 4 lines starting at
+line 3 in the new file.  An `<old_count>` of 0 means no lines were
+removed (pure insertion); a `<new_count>` of 0 means no lines were
+added (pure deletion).
+
+Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
+Hunks must be listed in order and must not overlap.  Any line
+not covered by a hunk is treated as unchanged, so the total
+number of unchanged lines must be the same on both sides.
+For example, if the old file has 10 lines and the hunks cover
+4 of them (`old_count` values summing to 4), then 6 old lines
+are unchanged.  The new file must also have exactly 6 lines
+not covered by hunks, so the `new_count` values must sum to
+`new_file_lines - 6`.
+
+-----------------------
+packet:          git< hunk 1 3 1 5
+packet:          git< hunk 10 2 12 2
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
+as changed and feeds them into the standard diff pipeline.  Patch
+output features (word diff, function context, color) work normally.
+Note that `--stat` and other summary formats use their own diff path
+and are not affected by the diff process.
+
+If no hunk lines precede the flush, followed by "success", Git
+treats the files as having no changes: `git diff` produces no output
+and `git blame` skips the commit, attributing lines to earlier commits.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool returns invalid hunks (out of bounds, overlapping), Git
+silently falls back to the builtin diff algorithm.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status.  Git warns and
+falls back to the builtin diff algorithm for this file.  The tool
+remains available for subsequent files.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=error
+packet:          git< 0000
+-----------------------
+
+In case the tool cannot or does not want to process the content as
+well as any future content for the lifetime of the Git process, it
+is expected to respond with an "abort" status.  Git silently falls
+back to the builtin diff algorithm for this file and does not send
+further requests to the tool.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=abort
+packet:          git< 0000
+-----------------------
+
+If the tool dies during the communication or does not adhere to the
+protocol then Git will stop the process and fall back to the builtin
+diff algorithm.  Git warns once and does not restart the process for
+subsequent files.
+
+Tools should ignore unknown keys in the per-file request to remain
+forward-compatible.  Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index cedc234173..22900368dd 100644
--- a/Makefile
+++ b/Makefile
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..d2ef9463d7
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,288 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  The tool controls which lines are marked as changed
+ * while the display shows the file content (after any textconv
+ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * When the tool returns no hunks with status=success, it considers
+ * the files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *get_or_launch_process(
+		struct userdiff_driver *drv)
+{
+	struct diff_subprocess *entry;
+
+	if (drv->diff_subprocess)
+		return drv->diff_subprocess;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start_command(&entry->subprocess, drv->process,
+				     start_diff_process_fn)) {
+		free(entry);
+		drv->diff_process_failed = 1;
+		return NULL;
+	}
+
+	drv->diff_subprocess = entry;
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret = 0;
+
+	if (size < 0)
+		return -1;
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/*
+	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
+	 * All numbers must be non-negative decimal with no leading
+	 * whitespace or sign characters.
+	 */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_count = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_count = strtol(line, &end, 10);
+	if (errno || end == line || *end != '\0')
+		return -1;
+
+	return 0;
+}
+
+static enum diff_process_result get_hunks(
+		struct userdiff_driver *drv,
+		const char *path,
+		const char *old_buf, long old_size,
+		const char *new_buf, long new_size,
+		struct xdl_hunk **hunks_out,
+		size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	backend = get_or_launch_process(drv);
+	if (!backend)
+		return DIFF_PROCESS_ERROR;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto comm_error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto comm_error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto comm_error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto comm_error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto comm_error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		*hunks_out = hunks;
+		*nr_hunks_out = nr_hunks;
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_OK;
+	}
+
+	if (!strcmp(status.buf, "abort")) {
+		/*
+		 * The tool voluntarily withdrew: stop sending requests
+		 * but do not warn (this is not a failure).
+		 */
+		backend->supported_capabilities &= ~CAP_HUNKS;
+		free(hunks);
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_SKIP;
+	}
+
+	/* status=error or unknown status */
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+
+comm_error:
+	/*
+	 * Communication failure (broken pipe, malformed response).
+	 * Tear down the process and mark as failed so we do not
+	 * retry on every subsequent file.
+	 */
+	drv->diff_process_failed = 1;
+	drv->diff_subprocess = NULL;
+	subprocess_stop_command(&backend->subprocess);
+	free(backend);
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+	struct xdl_hunk *ext_hunks = NULL;
+	size_t nr = 0;
+	enum diff_process_result res;
+
+	if (!diffopt || !path)
+		return DIFF_PROCESS_SKIP;
+	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+		return DIFF_PROCESS_SKIP;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return DIFF_PROCESS_SKIP;
+	if (drv->diff_process_failed)
+		return DIFF_PROCESS_SKIP;
+
+	res = get_hunks(drv, path,
+			file_a->ptr, file_a->size,
+			file_b->ptr, file_b->size,
+			&ext_hunks, &nr);
+	if (res == DIFF_PROCESS_OK) {
+		if (!nr) {
+			free(ext_hunks);
+			return DIFF_PROCESS_EQUIVALENT;
+		}
+		xpp->external_hunks = ext_hunks;
+		xpp->external_hunks_nr = nr;
+		return DIFF_PROCESS_OK;
+	}
+	if (res == DIFF_PROCESS_ERROR) {
+		warning(_("diff process '%s' failed for '%s',"
+			  " falling back to builtin diff"),
+			drv->process, path);
+		return DIFF_PROCESS_ERROR;
+	}
+	return DIFF_PROCESS_SKIP;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..d34b42f811
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,39 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+#include "xdiff/xdiff.h"
+
+struct diff_options;
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
+	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
+	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
+/*
+ * Consult the diff process configured for 'path' and populate
+ * xpp->external_hunks with the returned hunks.
+ *
+ * Handles driver lookup, flag checks (--no-ext-diff,
+ * --diff-algorithm), subprocess management, and error reporting.
+ *
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
+ * (files are considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
+ */
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 397e38b41c..2d5ed6ea8c 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -4031,6 +4032,17 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (diff_process_fill_hunks(o, name_a,
+					    &mf1, &mf2, &xpp)
+		    == DIFF_PROCESS_EQUIVALENT) {
+			if (textconv_one)
+				free(mf1.ptr);
+			if (textconv_two)
+				free(mf2.ptr);
+			goto free_ab_and_return;
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4111,6 +4123,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(xpp.external_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/diff.h b/diff.h
index 7eb84aadf4..d1e5a13e9e 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/** Disables diff.<driver>.process. */
+	unsigned no_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 11488623bf..8a7370b38f 100644
--- a/meson.build
+++ b/meson.build
@@ -328,6 +328,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/t/.gitattributes b/t/.gitattributes
index 7664c6e027..de97920cab 100644
--- a/t/.gitattributes
+++ b/t/.gitattributes
@@ -23,3 +23,4 @@ t[0-9][0-9][0-9][0-9]/* -whitespace
 /t8005/*.txt eol=lf
 /t9*/*.dump eol=lf
 /t0040*.sh whitespace=-indent-with-non-tab
+/t4080-diff-process.sh whitespace=-indent-with-non-tab
diff --git a/t/meson.build b/t/meson.build
index 7528e5cda5..f67208d7ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -510,6 +510,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..f159cd86d8
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,538 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+if test_have_prereq PYTHON
+then
+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+fi
+
+#
+# A single parametric diff process.
+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
+#
+# Modes:
+#   whole-file  - report all lines as changed (default)
+#   fixed-hunk  - always report hunk 5 2 5 2
+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
+#   bad-sync    - report hunk with mismatched unchanged totals
+#   overlap     - report two overlapping hunks
+#   no-hunks   - return no hunks (files considered equivalent)
+#   error       - return status=error for every request
+#   abort       - return status=abort for every request
+#   crash       - read one request then exit without responding
+#
+setup_backend () {
+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
+	import sys, os
+
+	def read_pkt():
+	    hdr = sys.stdin.buffer.read(4)
+	    if len(hdr) < 4: return None
+	    length = int(hdr, 16)
+	    if length == 0: return ""
+	    data = sys.stdin.buffer.read(length - 4)
+	    return data.decode().rstrip("\n")
+
+	def write_pkt(line):
+	    data = (line + "\n").encode()
+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    sys.stdout.buffer.flush()
+
+	def write_flush():
+	    sys.stdout.buffer.write(b"0000")
+	    sys.stdout.buffer.flush()
+
+	def read_content():
+	    chunks = []
+	    while True:
+	        hdr = sys.stdin.buffer.read(4)
+	        if len(hdr) < 4: break
+	        length = int(hdr, 16)
+	        if length == 0: break
+	        chunks.append(sys.stdin.buffer.read(length - 4))
+	    return b"".join(chunks)
+
+	mode = "whole-file"
+	logfile = None
+	for arg in sys.argv[1:]:
+	    if arg.startswith("--mode="):
+	        mode = arg[7:]
+	    elif arg.startswith("--log="):
+	        logfile = open(arg[6:], "a")
+
+	def log(msg):
+	    if logfile:
+	        logfile.write(msg + "\n")
+	        logfile.flush()
+
+	# Handshake
+	assert read_pkt() == "git-diff-client"
+	assert read_pkt() == "version=1"
+	read_pkt()
+	write_pkt("git-diff-server")
+	write_pkt("version=1")
+	write_flush()
+	while True:
+	    p = read_pkt()
+	    if p == "": break
+	write_pkt("capability=hunks")
+	write_flush()
+
+	log("ready")
+
+	while True:
+	    cmd = None
+	    pathname = None
+	    while True:
+	        p = read_pkt()
+	        if p is None: sys.exit(0)
+	        if p == "": break
+	        if p.startswith("command="): cmd = p.split("=",1)[1]
+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
+	    if cmd is None: sys.exit(0)
+	    old = read_content()
+	    new = read_content()
+	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
+	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
+	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
+
+	    if mode == "error":
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	        continue
+
+	    if mode == "abort":
+	        write_flush()
+	        write_pkt("status=abort")
+	        write_flush()
+	        continue
+
+	    if mode == "crash":
+	        sys.exit(1)
+
+	    if cmd == "hunks":
+	        if mode == "fixed-hunk":
+	            write_pkt("hunk 5 2 5 2")
+	        elif mode == "bad-hunk":
+	            write_pkt("hunk 999 1 999 1")
+	        elif mode == "bad-sync":
+	            write_pkt("hunk 1 2 1 1")
+	        elif mode == "overlap":
+	            write_pkt("hunk 1 5 1 5")
+	            write_pkt("hunk 3 2 3 2")
+	        elif mode == "no-hunks":
+	            pass
+	        else:
+	            ol = old.count(b"\n")
+	            nl = new.count(b"\n")
+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	        write_flush()
+	        write_pkt("status=success")
+	        write_flush()
+	    else:
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	PYEOF
+	write_script diff-process-backend <<-SHEOF
+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
+	SHEOF
+}
+
+BACKEND="./diff-process-backend"
+
+test_expect_success PYTHON 'setup' '
+	setup_backend &&
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# boundary.c: 10 lines, changes at 5-6 and 9-10.
+	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+
+	# worddiff.c: single-line function, value changes 1 -> 999.
+	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+
+	# newfile.c: single-line function, value changes 42 -> 99.
+	# Used by: new file, --exit-code, multiple drivers.
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 42; }
+	EOF
+	git add newfile.c &&
+
+	# logtest.c: single-line function for log/format-patch tests.
+	# Needs two commits so log -1 has a diff.
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+
+	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
+	cat >one.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >two.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add one.c two.c &&
+
+	git commit -m "initial" &&
+
+	# Second commit for logtest.c (so log -1 has something to show).
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	# Working tree modifications (not committed).
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 99; }
+	EOF
+
+	cat >one.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+
+	cat >two.c <<-\EOF
+	int second(void) { return 20; }
+	EOF
+'
+
+#
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	test_grep ! "^-OLD9" actual &&
+	test_grep ! "^-OLD10" actual &&
+	test_grep ! "^+NEW9" actual &&
+	test_grep ! "^+NEW10" actual
+'
+
+test_expect_success PYTHON 'diff process works with new file' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- newfile.c >actual 2>stderr &&
+	test_grep "return 99" actual &&
+	test_grep "pathname=newfile.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process works with added file (empty old side)' '
+	cat >added.c <<-\EOF &&
+	int added(void) { return 1; }
+	EOF
+	git add added.c &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "added" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process skipped for binary files' '
+	printf "\\0binary" >binary.c &&
+	git add binary.c &&
+	git commit -m "add binary" &&
+	printf "\\0changed" >binary.c &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- binary.c >actual &&
+	test_grep "Binary files" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
+	echo "not tracked by cdiff" >unmatched.txt &&
+	git add unmatched.txt &&
+	git commit -m "add unmatched.txt" &&
+
+	echo "modified" >unmatched.txt &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- unmatched.txt >actual &&
+	test_grep "modified" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'multiple drivers use separate processes' '
+	echo "*.h diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 1; }
+	EOF
+	git add multi.h &&
+	git commit -m "add multi.h" &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 2; }
+	EOF
+
+	rm -f backend-c.log backend-h.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+		diff -- newfile.c multi.h >actual 2>stderr &&
+	test_grep "pathname=newfile.c" backend-c.log &&
+	test_grep "pathname=multi.h" backend-h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process works alongside textconv' '
+	write_script uppercase-filter <<-\EOF &&
+	tr "a-z" "A-Z" <"$1"
+	EOF
+
+	cat >textconv.c <<-\EOF &&
+	hello world
+	EOF
+	git add textconv.c &&
+	git commit -m "add textconv.c" &&
+
+	cat >textconv.c <<-\EOF &&
+	goodbye world
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- textconv.c >actual 2>stderr &&
+	# The diff process receives textconv-transformed (uppercase) content.
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=HELLO WORLD" backend.log &&
+	test_grep "new=GOODBYE WORLD" backend.log &&
+	test_must_be_empty stderr
+'
+
+#
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
+test_expect_success PYTHON 'diff process with --word-diff' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --word-diff worddiff.c >actual 2>stderr &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual &&
+	test_grep "pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process works with git log -p' '
+	# With no-hunks mode, the tool says the files are equivalent,
+	# so log -p should show the commit but no diff content.
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -1 -p -- logtest.c >actual 2>stderr &&
+	test_grep "change logtest.c" actual &&
+	test_grep ! "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add nohunks.c &&
+	git commit -m "add nohunks.c" &&
+
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff nohunks.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code nohunks.c
+'
+
+test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
+	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+		diff --exit-code newfile.c
+'
+
+#
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process not used by --stat' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --stat worddiff.c >actual &&
+	test_grep "worddiff.c" actual &&
+	test_path_is_missing backend.log
+'
+
+#
+# Error handling and fallback.
+#
+
+test_expect_success PYTHON 'diff process fallback on tool error status' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success PYTHON 'diff process error keeps tool available for next file' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Unlike abort, error keeps the tool available: both files
+	# are sent to the tool (and both fall back).
+	test_grep "pathname=one.c" backend.log &&
+	test_grep "pathname=two.c" backend.log &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual
+'
+
+test_expect_success PYTHON 'diff process abort disables for session' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- one.c two.c >actual &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool.
+	test_grep "pathname=one.c" backend.log &&
+	test_grep ! "pathname=two.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Crash is a communication failure, so a warning is emitted.
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success PYTHON 'diff process startup failure only warns once' '
+	git -c diff.cdiff.process="/nonexistent/tool" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# Sentinel prevents repeated warnings: only one, not one per file.
+	test_grep "diff process.*failed" stderr >warnings &&
+	test_line_count = 1 warnings
+'
+
+test_expect_success PYTHON 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Invalid hunks are caught by xdiff validation, not the
+	# protocol layer, so no warning is emitted.
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
+	cat >synctest.c <<-\EOF &&
+	line1
+	line2
+	line3
+	EOF
+	git add synctest.c &&
+	git commit -m "add synctest.c" &&
+
+	cat >synctest.c <<-\EOF &&
+	line1
+	changed
+	line3
+	EOF
+
+	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
+	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
+	# The synchronization invariant fails and git falls back.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+		diff synctest.c >actual 2>stderr &&
+	test_grep "changed" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
+	# boundary.c has 10 lines, so both hunks are in bounds
+	# but they overlap at lines 3-5, triggering the ordering check.
+	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "NEW5" actual
+'
+
+test_done
diff --git a/userdiff.h b/userdiff.h
index 51c26e0d41..a98eabe377 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -3,6 +3,7 @@
 
 #include "notes-cache.h"
 
+struct diff_subprocess;
 struct index_state;
 struct repository;
 
@@ -33,6 +34,8 @@ struct userdiff_driver {
 	int textconv_want_cache;
 	const char *process;
 	char *process_owned;
+	struct diff_subprocess *diff_subprocess;
+	unsigned diff_process_failed : 1;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v3 5/6] diff: bypass diff process with --no-ext-diff and in format-patch

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Make --no-ext-diff disable diff.<driver>.process in addition to
diff.<driver>.command.  Although the two mechanisms work differently
(command replaces Git's output, process feeds hunks back into the
pipeline), both invoke external tools and --no-ext-diff means
"no external tools."

Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that
sets both allow_external and no_diff_process, so a single option
controls both.  Passing --ext-diff explicitly clears
no_diff_process, so a later --ext-diff overrides an earlier
--no-ext-diff.

Disable the diff process unconditionally in format-patch so that
generated patches are always based on the builtin diff algorithm
and can be applied reliably by recipients who do not have the
external tool.

Document that --diff-algorithm also bypasses the diff process,
since it sets ignore_driver_algorithm which diff_process_fill_hunks
already checks.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/diff-algorithm-option.adoc |  3 +++
 Documentation/diff-options.adoc          |  4 +++-
 builtin/log.c                            |  7 +++++++
 diff.c                                   | 16 ++++++++++++++--
 diff.h                                   |  4 +++-
 t/t4080-diff-process.sh                  | 16 ++++++++++++++++
 6 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..4d7e2ec35f 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index 8a63b5e164..18b8b0ed24 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -825,7 +825,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::
diff --git a/builtin/log.c b/builtin/log.c
index 8c0939dd42..1ea520c12d 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2213,6 +2213,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * Disable diff.<driver>.process so that patches generated by
+	 * format-patch are always based on the builtin diff algorithm
+	 * and can be applied reliably.
+	 */
+	rev.diffopt.flags.no_diff_process = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff.c b/diff.c
index 2d5ed6ea8c..38932db084 100644
--- a/diff.c
+++ b/diff.c
@@ -5913,6 +5913,17 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.no_diff_process = unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6241,8 +6252,9 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index d1e5a13e9e..71ba389f03 100644
--- a/diff.h
+++ b/diff.h
@@ -173,7 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
-	/** Disables diff.<driver>.process. */
+	/**
+	 * Disables diff.<driver>.process.  Set by --no-ext-diff.
+	 */
 	unsigned no_diff_process;
 
 	/**
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index f159cd86d8..ee0c306abd 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -419,6 +419,22 @@ test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
 	test_path_is_missing backend.log
 '
 
+test_expect_success PYTHON 'diff process bypassed by --no-ext-diff' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --no-ext-diff worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process not used by format-patch' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		format-patch -1 --stdout -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_path_is_missing backend.log
+'
+
 test_expect_success PYTHON 'diff process not used by --stat' '
 	rm -f backend.log &&
 	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
-- 
gitgitgadget

[PATCH v3 6/6] blame: consult diff process for no-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The consultation happens at the pass_blame_to_parent() callsite
using diff_process_fill_hunks(), matching how builtin_diff() in
diff.c uses the same function.  A new diff_hunks_xpp() variant
accepts a pre-populated xpparam_t for this callsite, while the
existing diff_hunks() retains its original signature and behavior.
The copy-detection callsite is unaffected since it does not use
the diff process.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                 |  40 +++++++++++----
 t/t4080-diff-process.sh | 106 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+), 9 deletions(-)

diff --git a/blame.c b/blame.c
index a3c49d132e..79f02735a4 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "xdiff-interface.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -314,17 +316,25 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 
-static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
+			  xdl_emit_hunk_consume_func_t hunk_func,
+			  void *cb_data, xpparam_t *xpp)
 {
-	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
 
-	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
+}
+
+static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+{
+	xpparam_t xpp = {0};
+
+	xpp.flags = xdl_opts;
+	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1943,6 +1953,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *parent, int ignore_diffs)
 {
 	mmfile_t file_p, file_o;
+	xpparam_t xpp = {0};
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
 
@@ -1961,10 +1972,21 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
-		die("unable to generate diff (%s -> %s)",
-		    oid_to_hex(&parent->commit->object.oid),
-		    oid_to_hex(&target->commit->object.oid));
+	xpp.flags = sb->xdl_opts;
+	/*
+	 * If the diff process considers the files equivalent,
+	 * skip the diff so blame looks past this commit.
+	 */
+	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
+				    &file_p, &file_o, &xpp)
+	    != DIFF_PROCESS_EQUIVALENT) {
+		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
+				   &d, &xpp))
+			die("unable to generate diff (%s -> %s)",
+			    oid_to_hex(&parent->commit->object.oid),
+			    oid_to_hex(&target->commit->object.oid));
+	}
+	free(xpp.external_hunks);
 	/* The rest are the same as the parent */
 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
 		    parent, target, 0);
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index ee0c306abd..fdf6da1c34 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -551,4 +551,110 @@ test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
 	test_grep "NEW5" actual
 '
 
+#
+# Blame integration.
+#
+
+test_expect_success PYTHON 'blame uses tool-provided hunks' '
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "add blame-hunk.c" &&
+	ORIG=$(git rev-parse --short HEAD) &&
+
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "change blame-hunk.c" &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+
+	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
+	# so blame should attribute lines 9-10 to the original commit
+	# even though the builtin diff would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		blame blame-hunk.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	sed -n "6p" actual >line6 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "$CHANGE" line6
+'
+
+test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without no-hunks mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With no-hunks mode, the process considers the files equivalent
+	# and blame skips the reformat commit, attributing to the original.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		blame blame.c >with &&
+	test_grep ! "$BLAME_COMMIT" with &&
+	test_grep "$ORIG_COMMIT" with
+'
+
+test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		blame --no-ext-diff blame.c >actual &&
+	# Without the process, blame attributes the reformat commit normally.
+	test_grep "$BLAME_COMMIT" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
+	# fixed-hunk mode would narrow blame to lines 5-6, but
+	# --no-ext-diff should bypass it and use the builtin diff.
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		blame --no-ext-diff blame-hunk.c >actual &&
+	# Builtin diff attributes lines 9-10 to the change commit.
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v3 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
> can produce better line matching than Git's builtin diff algorithm, but
> diff.<driver>.command replaces Git's output entirely, losing downstream
> features like word diff, function context, color, and blame.

This seems to break CI on Windows; take a look at

  https://github.com/git/git/actions/runs/26709491830/job/78717295153

for an example.

Thanks.

Re: [PATCH v3 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo]

On Sun, May 31, 2026 at 3:44 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
> > can produce better line matching than Git's builtin diff algorithm, but
> > diff.<driver>.command replaces Git's output entirely, losing downstream
> > features like word diff, function context, color, and blame.
>
> This seems to break CI on Windows; take a look at
>
>   https://github.com/git/git/actions/runs/26709491830/job/78717295153
>
> for an example.
>
> Thanks.

Thanks for the heads up. Interestingly, I think Windows exposed a latent issue
with sub-process startup dying instead of erroring out when there is a space in
the path.

I've submitted https://lore.kernel.org/git/pull.2133.git.1780287309846.gitgitgadget@gmail.com/T/#u
which I believe addresses the issue, however it seems like there are
some potentially
unrelated CI failures going on right now which is making verification
a bit harder.

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Johannes Schindelin]

Hi Michael,

I stumbled about this patch when it broke CI in Git for Windows, where we
do _not_ use `NO_PYTHON`, even though Python is unavailable in the
build/test CI jobs. The existing tests handle this situation gracefully,
this here patch does not:

On Sun, 7 Jun 2026, Michael Montalbo via GitGitGadget wrote:

> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> new file mode 100755
> index 0000000000..f159cd86d8
> --- /dev/null
> +++ b/t/t4080-diff-process.sh
> @@ -0,0 +1,538 @@
> +#!/bin/sh
> +
> +test_description='diff process via long-running process'
> +
> +. ./test-lib.sh
> +
> +if test_have_prereq PYTHON
> +then
> +	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)

When neither `python3` nor `python` are available (which is the case in
the minimal Git for Windows SDK used in Git's CI runs), this fails under
`set -e`. Before even running the first test case. Resulting in an
unexpected TAP format error.

Now, we could "fix" this by imitating what `lib-p4` does (see
https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
which demonstrates that it is indeed a work-around on Windows):

-- snip --
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index fdf6da1c341e67..bd22c247ff3856 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -4,9 +4,10 @@ test_description='diff process via long-running process'
 
 . ./test-lib.sh
 
-if test_have_prereq PYTHON
+if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH"
 then
-	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+	skip_all='python interpreter not available'
+	test_done
 fi
 
 #
-- snap --

Of course, this uncovers _another_ problem with the Python script: It uses
Python3-only `f"..."` format strings, which cannot be handled by the
Python2 to which the `PYTHON_PATH` variable in `linux-TEST-vars` points.
So this requires _another follow-up (see also
https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):

-- snip --
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index bd22c247ff3856..ba14682a9086e4 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -39,7 +39,8 @@ setup_backend () {
 
 	def write_pkt(line):
 	    data = (line + "\n").encode()
-	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    hdr = "{:04x}".format(len(data) + 4).encode()
+	    sys.stdout.buffer.write(hdr + data)
 	    sys.stdout.buffer.flush()
 
 	def write_flush():
@@ -98,7 +99,8 @@ setup_backend () {
 	    new = read_content()
 	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
 	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
-	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
+	    log("command={} pathname={} old={} new={}".format(
+	        cmd, pathname, old_first, new_first))
 
 	    if mode == "error":
 	        write_flush()
@@ -130,7 +132,7 @@ setup_backend () {
 	        else:
 	            ol = old.count(b"\n")
 	            nl = new.count(b"\n")
-	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	            write_pkt("hunk 1 {} 1 {}".format(ol, nl))
 	        write_flush()
 	        write_pkt("status=success")
 	        write_flush()
-- snap --

And this is still not enough to make it work with Python2, see
https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:

-- snip --
[...]
+ git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diff boundary.c
  Traceback (most recent call last):
    File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
      assert read_pkt() == "git-diff-client"
    File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 4, in read_pkt
      hdr = sys.stdin.buffer.read(4)
  AttributeError: 'file' object has no attribute 'buffer'
-- snap --

I have experienced similar patterns in my career, where a single decision
required multiple follow-up fixes _just_ to avoid having to revert that
decision. This kind of doubling down has never ended well.

Therefore I would like to take a step back, and ask: Is it _really_ a good
idea to use Python here? Are we certain that we want to _require_ Python
to run this test and skip it if Python isn't available (as is the case in
the Windows-related parts of Git's very own CI) even if Python has nothing
at all to do with the feature that is being tested?

I don't want to be doomed to repeat history, and we can very well learn
e.g. from prior art in this very project, where the tests for the
clean/smudge filters (which _also_ want to speak pkt-line over stdio)
needlessly incurred Perl as a requirement to run the tests. It was
Matheus's heroic work in 52917a998ef3a (t0021: implementation the
rot13-filter.pl script in C, 2022-08-14) and 4d1d843be7a15 (tests: use the
new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided
that unnecessary prerequisite.

Likewise, there is `test-tool pkt-line` intended for driving the pkt-line
protocol via simple shell scripts.

So the conscious project direction has been: fold pkt-line test backends
into `test-tool` and drop the scripting-language prereq. Reintroducing the
same shape in Python would walk this back.

Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
2025-04-29) has been another clear sign that the Git project is actively
_removing_ scripting-language dependencies from the build and test
infrastructure, not adding new ones.

The clear prior art in Git's own tests for what t4080 wants to do, as of
today, is `t/helper/test-rot13-filter.c`, which could be imitated here
instead of (re-)introducing a dependency on a scripting language other
than Unix shell in Git's test suite.

The `PYTHON` prereq exists in exactly five files today, all `git p4`
related (where Python is an intrinsic prerequisite given that `git-p4.py`
_is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
`t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
and `t/t9836-git-p4-metadata-encoding-python3.sh`.

After 7cdbff14d482 (remove merge-recursive-old, 2006-11-20), this here
patch would be the first one, after almost 20 years, to re-introduce
Python as a dependency outside `git p4`.

And it would also be the first ever to embed a Python script as a heredoc:

> +fi
> +
> +#
> +# A single parametric diff process.
> +# Usage: diff-process-backend --mode=<mode> [--log=<path>]
> +#
> +# Modes:
> +#   whole-file  - report all lines as changed (default)
> +#   fixed-hunk  - always report hunk 5 2 5 2
> +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
> +#   bad-sync    - report hunk with mismatched unchanged totals
> +#   overlap     - report two overlapping hunks
> +#   no-hunks   - return no hunks (files considered equivalent)
> +#   error       - return status=error for every request
> +#   abort       - return status=abort for every request
> +#   crash       - read one request then exit without responding
> +#
> +setup_backend () {
> +	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
> +	import sys, os
> +
> +	def read_pkt():
> +	    hdr = sys.stdin.buffer.read(4)
> +	    if len(hdr) < 4: return None
> +	    length = int(hdr, 16)
> +	    if length == 0: return ""
> +	    data = sys.stdin.buffer.read(length - 4)
> +	    return data.decode().rstrip("\n")
> +
> +	def write_pkt(line):
> +	    data = (line + "\n").encode()
> +	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> +	    sys.stdout.buffer.flush()
> +
> +	def write_flush():
> +	    sys.stdout.buffer.write(b"0000")
> +	    sys.stdout.buffer.flush()
> +
> +	def read_content():
> +	    chunks = []
> +	    while True:
> +	        hdr = sys.stdin.buffer.read(4)
> +	        if len(hdr) < 4: break
> +	        length = int(hdr, 16)
> +	        if length == 0: break
> +	        chunks.append(sys.stdin.buffer.read(length - 4))
> +	    return b"".join(chunks)
> +
> +	mode = "whole-file"
> +	logfile = None
> +	for arg in sys.argv[1:]:
> +	    if arg.startswith("--mode="):
> +	        mode = arg[7:]
> +	    elif arg.startswith("--log="):
> +	        logfile = open(arg[6:], "a")
> +
> +	def log(msg):
> +	    if logfile:
> +	        logfile.write(msg + "\n")
> +	        logfile.flush()
> +
> +	# Handshake
> +	assert read_pkt() == "git-diff-client"
> +	assert read_pkt() == "version=1"
> +	read_pkt()
> +	write_pkt("git-diff-server")
> +	write_pkt("version=1")
> +	write_flush()
> +	while True:
> +	    p = read_pkt()
> +	    if p == "": break
> +	write_pkt("capability=hunks")
> +	write_flush()
> +
> +	log("ready")
> +
> +	while True:
> +	    cmd = None
> +	    pathname = None
> +	    while True:
> +	        p = read_pkt()
> +	        if p is None: sys.exit(0)
> +	        if p == "": break
> +	        if p.startswith("command="): cmd = p.split("=",1)[1]
> +	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
> +	    if cmd is None: sys.exit(0)
> +	    old = read_content()
> +	    new = read_content()
> +	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
> +	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> +	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> +
> +	    if mode == "error":
> +	        write_flush()
> +	        write_pkt("status=error")
> +	        write_flush()
> +	        continue
> +
> +	    if mode == "abort":
> +	        write_flush()
> +	        write_pkt("status=abort")
> +	        write_flush()
> +	        continue
> +
> +	    if mode == "crash":
> +	        sys.exit(1)
> +
> +	    if cmd == "hunks":
> +	        if mode == "fixed-hunk":
> +	            write_pkt("hunk 5 2 5 2")
> +	        elif mode == "bad-hunk":
> +	            write_pkt("hunk 999 1 999 1")
> +	        elif mode == "bad-sync":
> +	            write_pkt("hunk 1 2 1 1")
> +	        elif mode == "overlap":
> +	            write_pkt("hunk 1 5 1 5")
> +	            write_pkt("hunk 3 2 3 2")
> +	        elif mode == "no-hunks":
> +	            pass
> +	        else:
> +	            ol = old.count(b"\n")
> +	            nl = new.count(b"\n")
> +	            write_pkt(f"hunk 1 {ol} 1 {nl}")
> +	        write_flush()
> +	        write_pkt("status=success")
> +	        write_flush()
> +	    else:
> +	        write_flush()
> +	        write_pkt("status=error")
> +	        write_flush()
> +	PYEOF

The existing pattern is to provide larger scripts as fixtures in
associated `t/tNNNN/` directories, not as heredoc, see e.g.
`t/t1509/prepare-chroot.sh`. Writing scripts, especially lengthy ones, in
heredoc strings makes it virtually impossible to use static code analysis
or syntax highlighting to fend off banal errors.

Given the complexity of what t4080 tries to test (error, abort, crash,
bad-sync, no-hunks, multiple files in one session, capability
negotiation), it would unfortunately be infeasible to use `test-tool
pkt-line` from a shell script implementing that `diff.*.process` protocol.

So I've spiked a demo how the `test-tool diff-process-backend` could look
like (letting Opus do the menial typing, so that I can enjoy at least part
of a sunny Sunday outside), which also passes the CI build and test:
https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6

That commit is of course not intended to be used as-is; Feel free to pick
code parts of it and integrate them into your topic branch. Or write your
own test-tool helper from scratch if that's more your jam.

Ciao,
Johannes

> +	write_script diff-process-backend <<-SHEOF
> +	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
> +	SHEOF
> +}
> +
> +BACKEND="./diff-process-backend"
> +
> +test_expect_success PYTHON 'setup' '
> +	setup_backend &&
> +	echo "*.c diff=cdiff" >.gitattributes &&
> +	git add .gitattributes &&
> +
> +	# boundary.c: 10 lines, changes at 5-6 and 9-10.
> +	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
> +	cat >boundary.c <<-\EOF &&
> +	line1
> +	line2
> +	line3
> +	line4
> +	OLD5
> +	OLD6
> +	line7
> +	line8
> +	OLD9
> +	OLD10
> +	EOF
> +	git add boundary.c &&
> +
> +	# worddiff.c: single-line function, value changes 1 -> 999.
> +	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
> +	cat >worddiff.c <<-\EOF &&
> +	int value(void) { return 1; }
> +	EOF
> +	git add worddiff.c &&
> +
> +	# newfile.c: single-line function, value changes 42 -> 99.
> +	# Used by: new file, --exit-code, multiple drivers.
> +	cat >newfile.c <<-\EOF &&
> +	int new_func(void) { return 42; }
> +	EOF
> +	git add newfile.c &&
> +
> +	# logtest.c: single-line function for log/format-patch tests.
> +	# Needs two commits so log -1 has a diff.
> +	cat >logtest.c <<-\EOF &&
> +	int logfunc(void) { return 1; }
> +	EOF
> +	git add logtest.c &&
> +
> +	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
> +	cat >one.c <<-\EOF &&
> +	int first(void) { return 1; }
> +	EOF
> +	cat >two.c <<-\EOF &&
> +	int second(void) { return 2; }
> +	EOF
> +	git add one.c two.c &&
> +
> +	git commit -m "initial" &&
> +
> +	# Second commit for logtest.c (so log -1 has something to show).
> +	cat >logtest.c <<-\EOF &&
> +	int logfunc(void) { return 2; }
> +	EOF
> +	git add logtest.c &&
> +	git commit -m "change logtest.c" &&
> +
> +	# Working tree modifications (not committed).
> +	cat >boundary.c <<-\EOF &&
> +	line1
> +	line2
> +	line3
> +	line4
> +	NEW5
> +	NEW6
> +	line7
> +	line8
> +	NEW9
> +	NEW10
> +	EOF
> +
> +	cat >worddiff.c <<-\EOF &&
> +	int value(void) { return 999; }
> +	EOF
> +
> +	cat >newfile.c <<-\EOF &&
> +	int new_func(void) { return 99; }
> +	EOF
> +
> +	cat >one.c <<-\EOF &&
> +	int first(void) { return 10; }
> +	EOF
> +
> +	cat >two.c <<-\EOF
> +	int second(void) { return 20; }
> +	EOF
> +'
> +
> +#
> +# Core behavior: the tool controls which lines are marked as changed.
> +#
> +
> +test_expect_success PYTHON 'diff process hunk boundaries affect output' '
> +	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
> +	# only reports lines 5-6 as changed.  Lines 9-10 should not
> +	# appear as changed in the output.
> +	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
> +		diff boundary.c >actual &&
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^-OLD6" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^+NEW6" actual &&
> +	test_grep ! "^-OLD9" actual &&
> +	test_grep ! "^-OLD10" actual &&
> +	test_grep ! "^+NEW9" actual &&
> +	test_grep ! "^+NEW10" actual
> +'
> +
> +test_expect_success PYTHON 'diff process works with new file' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- newfile.c >actual 2>stderr &&
> +	test_grep "return 99" actual &&
> +	test_grep "pathname=newfile.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process works with added file (empty old side)' '
> +	cat >added.c <<-\EOF &&
> +	int added(void) { return 1; }
> +	EOF
> +	git add added.c &&
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --cached -- added.c >actual 2>stderr &&
> +	test_grep "added" actual &&
> +	test_grep "pathname=added.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process skipped for binary files' '
> +	printf "\\0binary" >binary.c &&
> +	git add binary.c &&
> +	git commit -m "add binary" &&
> +	printf "\\0changed" >binary.c &&
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- binary.c >actual &&
> +	test_grep "Binary files" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
> +	echo "not tracked by cdiff" >unmatched.txt &&
> +	git add unmatched.txt &&
> +	git commit -m "add unmatched.txt" &&
> +
> +	echo "modified" >unmatched.txt &&
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- unmatched.txt >actual &&
> +	test_grep "modified" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +test_expect_success PYTHON 'multiple drivers use separate processes' '
> +	echo "*.h diff=hdiff" >>.gitattributes &&
> +	git add .gitattributes &&
> +
> +	cat >multi.h <<-\EOF &&
> +	int header(void) { return 1; }
> +	EOF
> +	git add multi.h &&
> +	git commit -m "add multi.h" &&
> +
> +	cat >multi.h <<-\EOF &&
> +	int header(void) { return 2; }
> +	EOF
> +
> +	rm -f backend-c.log backend-h.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
> +	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
> +		diff -- newfile.c multi.h >actual 2>stderr &&
> +	test_grep "pathname=newfile.c" backend-c.log &&
> +	test_grep "pathname=multi.h" backend-h.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process works alongside textconv' '
> +	write_script uppercase-filter <<-\EOF &&
> +	tr "a-z" "A-Z" <"$1"
> +	EOF
> +
> +	cat >textconv.c <<-\EOF &&
> +	hello world
> +	EOF
> +	git add textconv.c &&
> +	git commit -m "add textconv.c" &&
> +
> +	cat >textconv.c <<-\EOF &&
> +	goodbye world
> +	EOF
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.textconv="./uppercase-filter" \
> +	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- textconv.c >actual 2>stderr &&
> +	# The diff process receives textconv-transformed (uppercase) content.
> +	test_grep "pathname=textconv.c" backend.log &&
> +	test_grep "old=HELLO WORLD" backend.log &&
> +	test_grep "new=GOODBYE WORLD" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +#
> +# Downstream features: word diff, log, equivalent files, exit code.
> +#
> +
> +test_expect_success PYTHON 'diff process with --word-diff' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --word-diff worddiff.c >actual 2>stderr &&
> +	test_grep "\[-1;-\]" actual &&
> +	test_grep "{+999;+}" actual &&
> +	test_grep "pathname=worddiff.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process works with git log -p' '
> +	# With no-hunks mode, the tool says the files are equivalent,
> +	# so log -p should show the commit but no diff content.
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
> +		log -1 -p -- logtest.c >actual 2>stderr &&
> +	test_grep "change logtest.c" actual &&
> +	test_grep ! "return 2" actual &&
> +	test_grep "command=hunks pathname=logtest.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
> +	cat >nohunks.c <<-\EOF &&
> +	int zero(void) { return 0; }
> +	EOF
> +	git add nohunks.c &&
> +	git commit -m "add nohunks.c" &&
> +
> +	cat >nohunks.c <<-\EOF &&
> +	int zero(void) { return 999; }
> +	EOF
> +
> +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> +		diff nohunks.c >actual &&
> +	test_must_be_empty actual
> +'
> +
> +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
> +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> +		diff --exit-code nohunks.c
> +'
> +
> +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
> +	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
> +		diff --exit-code newfile.c
> +'
> +
> +#
> +# Bypass mechanisms: flags and commands that skip the diff process.
> +#
> +
> +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --diff-algorithm=patience worddiff.c >actual &&
> +	test_grep "return 999" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +test_expect_success PYTHON 'diff process not used by --stat' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --stat worddiff.c >actual &&
> +	test_grep "worddiff.c" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +#
> +# Error handling and fallback.
> +#
> +
> +test_expect_success PYTHON 'diff process fallback on tool error status' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> +		diff boundary.c >actual 2>stderr &&
> +	# Fallback produces the full builtin diff (both change regions).
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^-OLD9" actual &&
> +	test_grep "^+NEW9" actual &&
> +	# Tool was contacted (it replied with error, not crash).
> +	test_grep "command=hunks pathname=boundary.c" backend.log &&
> +	test_grep "diff process.*failed" stderr
> +'
> +
> +test_expect_success PYTHON 'diff process error keeps tool available for next file' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> +		diff -- one.c two.c >actual 2>stderr &&
> +	# Unlike abort, error keeps the tool available: both files
> +	# are sent to the tool (and both fall back).
> +	test_grep "pathname=one.c" backend.log &&
> +	test_grep "pathname=two.c" backend.log &&
> +	test_grep "return 10" actual &&
> +	test_grep "return 20" actual
> +'
> +
> +test_expect_success PYTHON 'diff process abort disables for session' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
> +		diff -- one.c two.c >actual &&
> +	# Both files should still produce diff output via fallback.
> +	test_grep "return 10" actual &&
> +	test_grep "return 20" actual &&
> +	# The tool aborts on the first file and git clears its
> +	# capability.  The second file never contacts the tool.
> +	test_grep "pathname=one.c" backend.log &&
> +	test_grep ! "pathname=two.c" backend.log
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on tool crash' '
> +	git -c diff.cdiff.process="$BACKEND --mode=crash" \
> +		diff boundary.c >actual 2>stderr &&
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^-OLD9" actual &&
> +	test_grep "^+NEW9" actual &&
> +	# Crash is a communication failure, so a warning is emitted.
> +	test_grep "diff process.*failed" stderr
> +'
> +
> +test_expect_success PYTHON 'diff process startup failure only warns once' '
> +	git -c diff.cdiff.process="/nonexistent/tool" \
> +		diff -- one.c two.c >actual 2>stderr &&
> +	# Both files produce diff output via fallback.
> +	test_grep "return 10" actual &&
> +	test_grep "return 20" actual &&
> +	# Sentinel prevents repeated warnings: only one, not one per file.
> +	test_grep "diff process.*failed" stderr >warnings &&
> +	test_line_count = 1 warnings
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on bad hunks' '
> +	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
> +		diff boundary.c >actual 2>stderr &&
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^-OLD9" actual &&
> +	test_grep "^+NEW9" actual &&
> +	# Invalid hunks are caught by xdiff validation, not the
> +	# protocol layer, so no warning is emitted.
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
> +	cat >synctest.c <<-\EOF &&
> +	line1
> +	line2
> +	line3
> +	EOF
> +	git add synctest.c &&
> +	git commit -m "add synctest.c" &&
> +
> +	cat >synctest.c <<-\EOF &&
> +	line1
> +	changed
> +	line3
> +	EOF
> +
> +	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
> +	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
> +	# The synchronization invariant fails and git falls back.
> +	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
> +		diff synctest.c >actual 2>stderr &&
> +	test_grep "changed" actual
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
> +	# boundary.c has 10 lines, so both hunks are in bounds
> +	# but they overlap at lines 3-5, triggering the ordering check.
> +	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
> +		diff boundary.c >actual 2>stderr &&
> +	test_grep "NEW5" actual
> +'
> +
> +test_done
> diff --git a/userdiff.h b/userdiff.h
> index 51c26e0d41..a98eabe377 100644
> --- a/userdiff.h
> +++ b/userdiff.h
> @@ -3,6 +3,7 @@
>  
>  #include "notes-cache.h"
>  
> +struct diff_subprocess;
>  struct index_state;
>  struct repository;
>  
> @@ -33,6 +34,8 @@ struct userdiff_driver {
>  	int textconv_want_cache;
>  	const char *process;
>  	char *process_owned;
> +	struct diff_subprocess *diff_subprocess;
> +	unsigned diff_process_failed : 1;
>  };
>  enum userdiff_driver_type {
>  	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
> -- 
> gitgitgadget
> 
> 
> 

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Sun, Jun 7, 2026 at 7:36 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Hi Michael,
>
> I stumbled about this patch when it broke CI in Git for Windows, where we
> do _not_ use `NO_PYTHON`, even though Python is unavailable in the
> build/test CI jobs. The existing tests handle this situation gracefully,
> this here patch does not:
>
> On Sun, 7 Jun 2026, Michael Montalbo via GitGitGadget wrote:
>
> > diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> > new file mode 100755
> > index 0000000000..f159cd86d8
> > --- /dev/null
> > +++ b/t/t4080-diff-process.sh
> > @@ -0,0 +1,538 @@
> > +#!/bin/sh
> > +
> > +test_description='diff process via long-running process'
> > +
> > +. ./test-lib.sh
> > +
> > +if test_have_prereq PYTHON
> > +then
> > +     PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
>
> When neither `python3` nor `python` are available (which is the case in
> the minimal Git for Windows SDK used in Git's CI runs), this fails under
> `set -e`. Before even running the first test case. Resulting in an
> unexpected TAP format error.
>
> Now, we could "fix" this by imitating what `lib-p4` does (see
> https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
> which demonstrates that it is indeed a work-around on Windows):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index fdf6da1c341e67..bd22c247ff3856 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -4,9 +4,10 @@ test_description='diff process via long-running process'
>
>  . ./test-lib.sh
>
> -if test_have_prereq PYTHON
> +if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH"
>  then
> -       PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
> +       skip_all='python interpreter not available'
> +       test_done
>  fi
>
>  #
> -- snap --
>
> Of course, this uncovers _another_ problem with the Python script: It uses
> Python3-only `f"..."` format strings, which cannot be handled by the
> Python2 to which the `PYTHON_PATH` variable in `linux-TEST-vars` points.
> So this requires _another follow-up (see also
> https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index bd22c247ff3856..ba14682a9086e4 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -39,7 +39,8 @@ setup_backend () {
>
>         def write_pkt(line):
>             data = (line + "\n").encode()
> -           sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> +           hdr = "{:04x}".format(len(data) + 4).encode()
> +           sys.stdout.buffer.write(hdr + data)
>             sys.stdout.buffer.flush()
>
>         def write_flush():
> @@ -98,7 +99,8 @@ setup_backend () {
>             new = read_content()
>             old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
>             new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> -           log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> +           log("command={} pathname={} old={} new={}".format(
> +               cmd, pathname, old_first, new_first))
>
>             if mode == "error":
>                 write_flush()
> @@ -130,7 +132,7 @@ setup_backend () {
>                 else:
>                     ol = old.count(b"\n")
>                     nl = new.count(b"\n")
> -                   write_pkt(f"hunk 1 {ol} 1 {nl}")
> +                   write_pkt("hunk 1 {} 1 {}".format(ol, nl))
>                 write_flush()
>                 write_pkt("status=success")
>                 write_flush()
> -- snap --
>
> And this is still not enough to make it work with Python2, see
> https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:
>
> -- snip --
> [...]
> + git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diff boundary.c
>   Traceback (most recent call last):
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
>       assert read_pkt() == "git-diff-client"
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 4, in read_pkt
>       hdr = sys.stdin.buffer.read(4)
>   AttributeError: 'file' object has no attribute 'buffer'
> -- snap --
>
> I have experienced similar patterns in my career, where a single decision
> required multiple follow-up fixes _just_ to avoid having to revert that
> decision. This kind of doubling down has never ended well.
>
> Therefore I would like to take a step back, and ask: Is it _really_ a good
> idea to use Python here? Are we certain that we want to _require_ Python
> to run this test and skip it if Python isn't available (as is the case in
> the Windows-related parts of Git's very own CI) even if Python has nothing
> at all to do with the feature that is being tested?
>
> I don't want to be doomed to repeat history, and we can very well learn
> e.g. from prior art in this very project, where the tests for the
> clean/smudge filters (which _also_ want to speak pkt-line over stdio)
> needlessly incurred Perl as a requirement to run the tests. It was
> Matheus's heroic work in 52917a998ef3a (t0021: implementation the
> rot13-filter.pl script in C, 2022-08-14) and 4d1d843be7a15 (tests: use the
> new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided
> that unnecessary prerequisite.
>
> Likewise, there is `test-tool pkt-line` intended for driving the pkt-line
> protocol via simple shell scripts.
>
> So the conscious project direction has been: fold pkt-line test backends
> into `test-tool` and drop the scripting-language prereq. Reintroducing the
> same shape in Python would walk this back.
>
> Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
> 2025-04-29) has been another clear sign that the Git project is actively
> _removing_ scripting-language dependencies from the build and test
> infrastructure, not adding new ones.
>
> The clear prior art in Git's own tests for what t4080 wants to do, as of
> today, is `t/helper/test-rot13-filter.c`, which could be imitated here
> instead of (re-)introducing a dependency on a scripting language other
> than Unix shell in Git's test suite.
>
> The `PYTHON` prereq exists in exactly five files today, all `git p4`
> related (where Python is an intrinsic prerequisite given that `git-p4.py`
> _is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
> `t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
> and `t/t9836-git-p4-metadata-encoding-python3.sh`.
>
> After 7cdbff14d482 (remove merge-recursive-old, 2006-11-20), this here
> patch would be the first one, after almost 20 years, to re-introduce
> Python as a dependency outside `git p4`.
>
> And it would also be the first ever to embed a Python script as a heredoc:
>
> > +fi
> > +
> > +#
> > +# A single parametric diff process.
> > +# Usage: diff-process-backend --mode=<mode> [--log=<path>]
> > +#
> > +# Modes:
> > +#   whole-file  - report all lines as changed (default)
> > +#   fixed-hunk  - always report hunk 5 2 5 2
> > +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
> > +#   bad-sync    - report hunk with mismatched unchanged totals
> > +#   overlap     - report two overlapping hunks
> > +#   no-hunks   - return no hunks (files considered equivalent)
> > +#   error       - return status=error for every request
> > +#   abort       - return status=abort for every request
> > +#   crash       - read one request then exit without responding
> > +#
> > +setup_backend () {
> > +     cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
> > +     import sys, os
> > +
> > +     def read_pkt():
> > +         hdr = sys.stdin.buffer.read(4)
> > +         if len(hdr) < 4: return None
> > +         length = int(hdr, 16)
> > +         if length == 0: return ""
> > +         data = sys.stdin.buffer.read(length - 4)
> > +         return data.decode().rstrip("\n")
> > +
> > +     def write_pkt(line):
> > +         data = (line + "\n").encode()
> > +         sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> > +         sys.stdout.buffer.flush()
> > +
> > +     def write_flush():
> > +         sys.stdout.buffer.write(b"0000")
> > +         sys.stdout.buffer.flush()
> > +
> > +     def read_content():
> > +         chunks = []
> > +         while True:
> > +             hdr = sys.stdin.buffer.read(4)
> > +             if len(hdr) < 4: break
> > +             length = int(hdr, 16)
> > +             if length == 0: break
> > +             chunks.append(sys.stdin.buffer.read(length - 4))
> > +         return b"".join(chunks)
> > +
> > +     mode = "whole-file"
> > +     logfile = None
> > +     for arg in sys.argv[1:]:
> > +         if arg.startswith("--mode="):
> > +             mode = arg[7:]
> > +         elif arg.startswith("--log="):
> > +             logfile = open(arg[6:], "a")
> > +
> > +     def log(msg):
> > +         if logfile:
> > +             logfile.write(msg + "\n")
> > +             logfile.flush()
> > +
> > +     # Handshake
> > +     assert read_pkt() == "git-diff-client"
> > +     assert read_pkt() == "version=1"
> > +     read_pkt()
> > +     write_pkt("git-diff-server")
> > +     write_pkt("version=1")
> > +     write_flush()
> > +     while True:
> > +         p = read_pkt()
> > +         if p == "": break
> > +     write_pkt("capability=hunks")
> > +     write_flush()
> > +
> > +     log("ready")
> > +
> > +     while True:
> > +         cmd = None
> > +         pathname = None
> > +         while True:
> > +             p = read_pkt()
> > +             if p is None: sys.exit(0)
> > +             if p == "": break
> > +             if p.startswith("command="): cmd = p.split("=",1)[1]
> > +             if p.startswith("pathname="): pathname = p.split("=",1)[1]
> > +         if cmd is None: sys.exit(0)
> > +         old = read_content()
> > +         new = read_content()
> > +         old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
> > +         new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> > +         log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> > +
> > +         if mode == "error":
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "abort":
> > +             write_flush()
> > +             write_pkt("status=abort")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "crash":
> > +             sys.exit(1)
> > +
> > +         if cmd == "hunks":
> > +             if mode == "fixed-hunk":
> > +                 write_pkt("hunk 5 2 5 2")
> > +             elif mode == "bad-hunk":
> > +                 write_pkt("hunk 999 1 999 1")
> > +             elif mode == "bad-sync":
> > +                 write_pkt("hunk 1 2 1 1")
> > +             elif mode == "overlap":
> > +                 write_pkt("hunk 1 5 1 5")
> > +                 write_pkt("hunk 3 2 3 2")
> > +             elif mode == "no-hunks":
> > +                 pass
> > +             else:
> > +                 ol = old.count(b"\n")
> > +                 nl = new.count(b"\n")
> > +                 write_pkt(f"hunk 1 {ol} 1 {nl}")
> > +             write_flush()
> > +             write_pkt("status=success")
> > +             write_flush()
> > +         else:
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +     PYEOF
>
> The existing pattern is to provide larger scripts as fixtures in
> associated `t/tNNNN/` directories, not as heredoc, see e.g.
> `t/t1509/prepare-chroot.sh`. Writing scripts, especially lengthy ones, in
> heredoc strings makes it virtually impossible to use static code analysis
> or syntax highlighting to fend off banal errors.
>
> Given the complexity of what t4080 tries to test (error, abort, crash,
> bad-sync, no-hunks, multiple files in one session, capability
> negotiation), it would unfortunately be infeasible to use `test-tool
> pkt-line` from a shell script implementing that `diff.*.process` protocol.
>
> So I've spiked a demo how the `test-tool diff-process-backend` could look
> like (letting Opus do the menial typing, so that I can enjoy at least part
> of a sunny Sunday outside), which also passes the CI build and test:
> https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6
>
> That commit is of course not intended to be used as-is; Feel free to pick
> code parts of it and integrate them into your topic branch. Or write your
> own test-tool helper from scratch if that's more your jam.
>

Johannes, thank you for the great feedback. The historical context is
really helpful and
the concerns you raise make a lot of sense. I will take a look at your
spike and also work
on removing Python from the test.

> Ciao,
> Johannes
>
> > +     write_script diff-process-backend <<-SHEOF
> > +     exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
> > +     SHEOF
> > +}
> > +
> > +BACKEND="./diff-process-backend"
> > +
> > +test_expect_success PYTHON 'setup' '
> > +     setup_backend &&
> > +     echo "*.c diff=cdiff" >.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     # boundary.c: 10 lines, changes at 5-6 and 9-10.
> > +     # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     OLD5
> > +     OLD6
> > +     line7
> > +     line8
> > +     OLD9
> > +     OLD10
> > +     EOF
> > +     git add boundary.c &&
> > +
> > +     # worddiff.c: single-line function, value changes 1 -> 999.
> > +     # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 1; }
> > +     EOF
> > +     git add worddiff.c &&
> > +
> > +     # newfile.c: single-line function, value changes 42 -> 99.
> > +     # Used by: new file, --exit-code, multiple drivers.
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 42; }
> > +     EOF
> > +     git add newfile.c &&
> > +
> > +     # logtest.c: single-line function for log/format-patch tests.
> > +     # Needs two commits so log -1 has a diff.
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 1; }
> > +     EOF
> > +     git add logtest.c &&
> > +
> > +     # two.c/one.c: two-file pair for error/abort/startup-failure tests.
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 1; }
> > +     EOF
> > +     cat >two.c <<-\EOF &&
> > +     int second(void) { return 2; }
> > +     EOF
> > +     git add one.c two.c &&
> > +
> > +     git commit -m "initial" &&
> > +
> > +     # Second commit for logtest.c (so log -1 has something to show).
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 2; }
> > +     EOF
> > +     git add logtest.c &&
> > +     git commit -m "change logtest.c" &&
> > +
> > +     # Working tree modifications (not committed).
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     NEW5
> > +     NEW6
> > +     line7
> > +     line8
> > +     NEW9
> > +     NEW10
> > +     EOF
> > +
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 999; }
> > +     EOF
> > +
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 99; }
> > +     EOF
> > +
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 10; }
> > +     EOF
> > +
> > +     cat >two.c <<-\EOF
> > +     int second(void) { return 20; }
> > +     EOF
> > +'
> > +
> > +#
> > +# Core behavior: the tool controls which lines are marked as changed.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process hunk boundaries affect output' '
> > +     # The file has changes at lines 5-6 and 9-10, but fixed-hunk
> > +     # only reports lines 5-6 as changed.  Lines 9-10 should not
> > +     # appear as changed in the output.
> > +     git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
> > +             diff boundary.c >actual &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^-OLD6" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^+NEW6" actual &&
> > +     test_grep ! "^-OLD9" actual &&
> > +     test_grep ! "^-OLD10" actual &&
> > +     test_grep ! "^+NEW9" actual &&
> > +     test_grep ! "^+NEW10" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with new file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- newfile.c >actual 2>stderr &&
> > +     test_grep "return 99" actual &&
> > +     test_grep "pathname=newfile.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with added file (empty old side)' '
> > +     cat >added.c <<-\EOF &&
> > +     int added(void) { return 1; }
> > +     EOF
> > +     git add added.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --cached -- added.c >actual 2>stderr &&
> > +     test_grep "added" actual &&
> > +     test_grep "pathname=added.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process skipped for binary files' '
> > +     printf "\\0binary" >binary.c &&
> > +     git add binary.c &&
> > +     git commit -m "add binary" &&
> > +     printf "\\0changed" >binary.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- binary.c >actual &&
> > +     test_grep "Binary files" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
> > +     echo "not tracked by cdiff" >unmatched.txt &&
> > +     git add unmatched.txt &&
> > +     git commit -m "add unmatched.txt" &&
> > +
> > +     echo "modified" >unmatched.txt &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- unmatched.txt >actual &&
> > +     test_grep "modified" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'multiple drivers use separate processes' '
> > +     echo "*.h diff=hdiff" >>.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 1; }
> > +     EOF
> > +     git add multi.h &&
> > +     git commit -m "add multi.h" &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 2; }
> > +     EOF
> > +
> > +     rm -f backend-c.log backend-h.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
> > +         -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
> > +             diff -- newfile.c multi.h >actual 2>stderr &&
> > +     test_grep "pathname=newfile.c" backend-c.log &&
> > +     test_grep "pathname=multi.h" backend-h.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works alongside textconv' '
> > +     write_script uppercase-filter <<-\EOF &&
> > +     tr "a-z" "A-Z" <"$1"
> > +     EOF
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     hello world
> > +     EOF
> > +     git add textconv.c &&
> > +     git commit -m "add textconv.c" &&
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     goodbye world
> > +     EOF
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.textconv="./uppercase-filter" \
> > +         -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- textconv.c >actual 2>stderr &&
> > +     # The diff process receives textconv-transformed (uppercase) content.
> > +     test_grep "pathname=textconv.c" backend.log &&
> > +     test_grep "old=HELLO WORLD" backend.log &&
> > +     test_grep "new=GOODBYE WORLD" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +#
> > +# Downstream features: word diff, log, equivalent files, exit code.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process with --word-diff' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --word-diff worddiff.c >actual 2>stderr &&
> > +     test_grep "\[-1;-\]" actual &&
> > +     test_grep "{+999;+}" actual &&
> > +     test_grep "pathname=worddiff.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with git log -p' '
> > +     # With no-hunks mode, the tool says the files are equivalent,
> > +     # so log -p should show the commit but no diff content.
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
> > +             log -1 -p -- logtest.c >actual 2>stderr &&
> > +     test_grep "change logtest.c" actual &&
> > +     test_grep ! "return 2" actual &&
> > +     test_grep "command=hunks pathname=logtest.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 0; }
> > +     EOF
> > +     git add nohunks.c &&
> > +     git commit -m "add nohunks.c" &&
> > +
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 999; }
> > +     EOF
> > +
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff nohunks.c >actual &&
> > +     test_must_be_empty actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff --exit-code nohunks.c
> > +'
> > +
> > +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
> > +     test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
> > +             diff --exit-code newfile.c
> > +'
> > +
> > +#
> > +# Bypass mechanisms: flags and commands that skip the diff process.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --diff-algorithm=patience worddiff.c >actual &&
> > +     test_grep "return 999" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not used by --stat' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --stat worddiff.c >actual &&
> > +     test_grep "worddiff.c" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +#
> > +# Error handling and fallback.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool error status' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     # Fallback produces the full builtin diff (both change regions).
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Tool was contacted (it replied with error, not crash).
> > +     test_grep "command=hunks pathname=boundary.c" backend.log &&
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process error keeps tool available for next file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Unlike abort, error keeps the tool available: both files
> > +     # are sent to the tool (and both fall back).
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep "pathname=two.c" backend.log &&
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process abort disables for session' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
> > +             diff -- one.c two.c >actual &&
> > +     # Both files should still produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # The tool aborts on the first file and git clears its
> > +     # capability.  The second file never contacts the tool.
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep ! "pathname=two.c" backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool crash' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=crash" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Crash is a communication failure, so a warning is emitted.
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process startup failure only warns once' '
> > +     git -c diff.cdiff.process="/nonexistent/tool" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Both files produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # Sentinel prevents repeated warnings: only one, not one per file.
> > +     test_grep "diff process.*failed" stderr >warnings &&
> > +     test_line_count = 1 warnings
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on bad hunks' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Invalid hunks are caught by xdiff validation, not the
> > +     # protocol layer, so no warning is emitted.
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     EOF
> > +     git add synctest.c &&
> > +     git commit -m "add synctest.c" &&
> > +
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     changed
> > +     line3
> > +     EOF
> > +
> > +     # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
> > +     # line as changed, leaving 1 unchanged old vs 2 unchanged new.
> > +     # The synchronization invariant fails and git falls back.
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
> > +             diff synctest.c >actual 2>stderr &&
> > +     test_grep "changed" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
> > +     # boundary.c has 10 lines, so both hunks are in bounds
> > +     # but they overlap at lines 3-5, triggering the ordering check.
> > +     git -c diff.cdiff.process="$BACKEND --mode=overlap" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "NEW5" actual
> > +'
> > +
> > +test_done
> > diff --git a/userdiff.h b/userdiff.h
> > index 51c26e0d41..a98eabe377 100644
> > --- a/userdiff.h
> > +++ b/userdiff.h
> > @@ -3,6 +3,7 @@
> >
> >  #include "notes-cache.h"
> >
> > +struct diff_subprocess;
> >  struct index_state;
> >  struct repository;
> >
> > @@ -33,6 +34,8 @@ struct userdiff_driver {
> >       int textconv_want_cache;
> >       const char *process;
> >       char *process_owned;
> > +     struct diff_subprocess *diff_subprocess;
> > +     unsigned diff_process_failed : 1;
> >  };
> >  enum userdiff_driver_type {
> >       USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
> > --
> > gitgitgadget
> >
> >
> >

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Sun, Jun 7, 2026 at 7:36 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Hi Michael,
>
> I stumbled about this patch when it broke CI in Git for Windows, where we
> do _not_ use `NO_PYTHON`, even though Python is unavailable in the
> build/test CI jobs. The existing tests handle this situation gracefully,
> this here patch does not:
>
> On Sun, 7 Jun 2026, Michael Montalbo via GitGitGadget wrote:
>
> > diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> > new file mode 100755
> > index 0000000000..f159cd86d8
> > --- /dev/null
> > +++ b/t/t4080-diff-process.sh
> > @@ -0,0 +1,538 @@
> > +#!/bin/sh
> > +
> > +test_description='diff process via long-running process'
> > +
> > +. ./test-lib.sh
> > +
> > +if test_have_prereq PYTHON
> > +then
> > +     PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
>
> When neither `python3` nor `python` are available (which is the case in
> the minimal Git for Windows SDK used in Git's CI runs), this fails under
> `set -e`. Before even running the first test case. Resulting in an
> unexpected TAP format error.
>
> Now, we could "fix" this by imitating what `lib-p4` does (see
> https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
> which demonstrates that it is indeed a work-around on Windows):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index fdf6da1c341e67..bd22c247ff3856 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -4,9 +4,10 @@ test_description='diff process via long-running process'
>
>  . ./test-lib.sh
>
> -if test_have_prereq PYTHON
> +if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH"
>  then
> -       PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
> +       skip_all='python interpreter not available'
> +       test_done
>  fi
>
>  #
> -- snap --
>
> Of course, this uncovers _another_ problem with the Python script: It uses
> Python3-only `f"..."` format strings, which cannot be handled by the
> Python2 to which the `PYTHON_PATH` variable in `linux-TEST-vars` points.
> So this requires _another follow-up (see also
> https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index bd22c247ff3856..ba14682a9086e4 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -39,7 +39,8 @@ setup_backend () {
>
>         def write_pkt(line):
>             data = (line + "\n").encode()
> -           sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> +           hdr = "{:04x}".format(len(data) + 4).encode()
> +           sys.stdout.buffer.write(hdr + data)
>             sys.stdout.buffer.flush()
>
>         def write_flush():
> @@ -98,7 +99,8 @@ setup_backend () {
>             new = read_content()
>             old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
>             new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> -           log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> +           log("command={} pathname={} old={} new={}".format(
> +               cmd, pathname, old_first, new_first))
>
>             if mode == "error":
>                 write_flush()
> @@ -130,7 +132,7 @@ setup_backend () {
>                 else:
>                     ol = old.count(b"\n")
>                     nl = new.count(b"\n")
> -                   write_pkt(f"hunk 1 {ol} 1 {nl}")
> +                   write_pkt("hunk 1 {} 1 {}".format(ol, nl))
>                 write_flush()
>                 write_pkt("status=success")
>                 write_flush()
> -- snap --
>
> And this is still not enough to make it work with Python2, see
> https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:
>
> -- snip --
> [...]
> + git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diff boundary.c
>   Traceback (most recent call last):
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
>       assert read_pkt() == "git-diff-client"
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 4, in read_pkt
>       hdr = sys.stdin.buffer.read(4)
>   AttributeError: 'file' object has no attribute 'buffer'
> -- snap --
>
> I have experienced similar patterns in my career, where a single decision
> required multiple follow-up fixes _just_ to avoid having to revert that
> decision. This kind of doubling down has never ended well.
>
> Therefore I would like to take a step back, and ask: Is it _really_ a good
> idea to use Python here? Are we certain that we want to _require_ Python
> to run this test and skip it if Python isn't available (as is the case in
> the Windows-related parts of Git's very own CI) even if Python has nothing
> at all to do with the feature that is being tested?
>
> I don't want to be doomed to repeat history, and we can very well learn
> e.g. from prior art in this very project, where the tests for the
> clean/smudge filters (which _also_ want to speak pkt-line over stdio)
> needlessly incurred Perl as a requirement to run the tests. It was
> Matheus's heroic work in 52917a998ef3a (t0021: implementation the
> rot13-filter.pl script in C, 2022-08-14) and 4d1d843be7a15 (tests: use the
> new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided
> that unnecessary prerequisite.
>
> Likewise, there is `test-tool pkt-line` intended for driving the pkt-line
> protocol via simple shell scripts.
>
> So the conscious project direction has been: fold pkt-line test backends
> into `test-tool` and drop the scripting-language prereq. Reintroducing the
> same shape in Python would walk this back.
>
> Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
> 2025-04-29) has been another clear sign that the Git project is actively
> _removing_ scripting-language dependencies from the build and test
> infrastructure, not adding new ones.
>

Now I wonder if the extension / addition of more Perl test infra with my other
series:

https://lore.kernel.org/git/pull.2135.git.1780559158.gitgitgadget@gmail.com/T/

also goes against the project direction w.r.t. removing scripting languages.
Maybe I should also re-evaluate the usage of Perl there. I am leveraging
existing shell parsing logic written in Perl, but maybe the preference for
Perl-based lint rules is a mistake and should be avoided.

> The clear prior art in Git's own tests for what t4080 wants to do, as of
> today, is `t/helper/test-rot13-filter.c`, which could be imitated here
> instead of (re-)introducing a dependency on a scripting language other
> than Unix shell in Git's test suite.
>
> The `PYTHON` prereq exists in exactly five files today, all `git p4`
> related (where Python is an intrinsic prerequisite given that `git-p4.py`
> _is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
> `t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
> and `t/t9836-git-p4-metadata-encoding-python3.sh`.
>
> After 7cdbff14d482 (remove merge-recursive-old, 2006-11-20), this here
> patch would be the first one, after almost 20 years, to re-introduce
> Python as a dependency outside `git p4`.
>
> And it would also be the first ever to embed a Python script as a heredoc:
>
> > +fi
> > +
> > +#
> > +# A single parametric diff process.
> > +# Usage: diff-process-backend --mode=<mode> [--log=<path>]
> > +#
> > +# Modes:
> > +#   whole-file  - report all lines as changed (default)
> > +#   fixed-hunk  - always report hunk 5 2 5 2
> > +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
> > +#   bad-sync    - report hunk with mismatched unchanged totals
> > +#   overlap     - report two overlapping hunks
> > +#   no-hunks   - return no hunks (files considered equivalent)
> > +#   error       - return status=error for every request
> > +#   abort       - return status=abort for every request
> > +#   crash       - read one request then exit without responding
> > +#
> > +setup_backend () {
> > +     cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
> > +     import sys, os
> > +
> > +     def read_pkt():
> > +         hdr = sys.stdin.buffer.read(4)
> > +         if len(hdr) < 4: return None
> > +         length = int(hdr, 16)
> > +         if length == 0: return ""
> > +         data = sys.stdin.buffer.read(length - 4)
> > +         return data.decode().rstrip("\n")
> > +
> > +     def write_pkt(line):
> > +         data = (line + "\n").encode()
> > +         sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> > +         sys.stdout.buffer.flush()
> > +
> > +     def write_flush():
> > +         sys.stdout.buffer.write(b"0000")
> > +         sys.stdout.buffer.flush()
> > +
> > +     def read_content():
> > +         chunks = []
> > +         while True:
> > +             hdr = sys.stdin.buffer.read(4)
> > +             if len(hdr) < 4: break
> > +             length = int(hdr, 16)
> > +             if length == 0: break
> > +             chunks.append(sys.stdin.buffer.read(length - 4))
> > +         return b"".join(chunks)
> > +
> > +     mode = "whole-file"
> > +     logfile = None
> > +     for arg in sys.argv[1:]:
> > +         if arg.startswith("--mode="):
> > +             mode = arg[7:]
> > +         elif arg.startswith("--log="):
> > +             logfile = open(arg[6:], "a")
> > +
> > +     def log(msg):
> > +         if logfile:
> > +             logfile.write(msg + "\n")
> > +             logfile.flush()
> > +
> > +     # Handshake
> > +     assert read_pkt() == "git-diff-client"
> > +     assert read_pkt() == "version=1"
> > +     read_pkt()
> > +     write_pkt("git-diff-server")
> > +     write_pkt("version=1")
> > +     write_flush()
> > +     while True:
> > +         p = read_pkt()
> > +         if p == "": break
> > +     write_pkt("capability=hunks")
> > +     write_flush()
> > +
> > +     log("ready")
> > +
> > +     while True:
> > +         cmd = None
> > +         pathname = None
> > +         while True:
> > +             p = read_pkt()
> > +             if p is None: sys.exit(0)
> > +             if p == "": break
> > +             if p.startswith("command="): cmd = p.split("=",1)[1]
> > +             if p.startswith("pathname="): pathname = p.split("=",1)[1]
> > +         if cmd is None: sys.exit(0)
> > +         old = read_content()
> > +         new = read_content()
> > +         old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
> > +         new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> > +         log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> > +
> > +         if mode == "error":
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "abort":
> > +             write_flush()
> > +             write_pkt("status=abort")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "crash":
> > +             sys.exit(1)
> > +
> > +         if cmd == "hunks":
> > +             if mode == "fixed-hunk":
> > +                 write_pkt("hunk 5 2 5 2")
> > +             elif mode == "bad-hunk":
> > +                 write_pkt("hunk 999 1 999 1")
> > +             elif mode == "bad-sync":
> > +                 write_pkt("hunk 1 2 1 1")
> > +             elif mode == "overlap":
> > +                 write_pkt("hunk 1 5 1 5")
> > +                 write_pkt("hunk 3 2 3 2")
> > +             elif mode == "no-hunks":
> > +                 pass
> > +             else:
> > +                 ol = old.count(b"\n")
> > +                 nl = new.count(b"\n")
> > +                 write_pkt(f"hunk 1 {ol} 1 {nl}")
> > +             write_flush()
> > +             write_pkt("status=success")
> > +             write_flush()
> > +         else:
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +     PYEOF
>
> The existing pattern is to provide larger scripts as fixtures in
> associated `t/tNNNN/` directories, not as heredoc, see e.g.
> `t/t1509/prepare-chroot.sh`. Writing scripts, especially lengthy ones, in
> heredoc strings makes it virtually impossible to use static code analysis
> or syntax highlighting to fend off banal errors.
>
> Given the complexity of what t4080 tries to test (error, abort, crash,
> bad-sync, no-hunks, multiple files in one session, capability
> negotiation), it would unfortunately be infeasible to use `test-tool
> pkt-line` from a shell script implementing that `diff.*.process` protocol.
>
> So I've spiked a demo how the `test-tool diff-process-backend` could look
> like (letting Opus do the menial typing, so that I can enjoy at least part
> of a sunny Sunday outside), which also passes the CI build and test:
> https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6
>
> That commit is of course not intended to be used as-is; Feel free to pick
> code parts of it and integrate them into your topic branch. Or write your
> own test-tool helper from scratch if that's more your jam.
>
> Ciao,
> Johannes
>
> > +     write_script diff-process-backend <<-SHEOF
> > +     exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
> > +     SHEOF
> > +}
> > +
> > +BACKEND="./diff-process-backend"
> > +
> > +test_expect_success PYTHON 'setup' '
> > +     setup_backend &&
> > +     echo "*.c diff=cdiff" >.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     # boundary.c: 10 lines, changes at 5-6 and 9-10.
> > +     # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     OLD5
> > +     OLD6
> > +     line7
> > +     line8
> > +     OLD9
> > +     OLD10
> > +     EOF
> > +     git add boundary.c &&
> > +
> > +     # worddiff.c: single-line function, value changes 1 -> 999.
> > +     # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 1; }
> > +     EOF
> > +     git add worddiff.c &&
> > +
> > +     # newfile.c: single-line function, value changes 42 -> 99.
> > +     # Used by: new file, --exit-code, multiple drivers.
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 42; }
> > +     EOF
> > +     git add newfile.c &&
> > +
> > +     # logtest.c: single-line function for log/format-patch tests.
> > +     # Needs two commits so log -1 has a diff.
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 1; }
> > +     EOF
> > +     git add logtest.c &&
> > +
> > +     # two.c/one.c: two-file pair for error/abort/startup-failure tests.
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 1; }
> > +     EOF
> > +     cat >two.c <<-\EOF &&
> > +     int second(void) { return 2; }
> > +     EOF
> > +     git add one.c two.c &&
> > +
> > +     git commit -m "initial" &&
> > +
> > +     # Second commit for logtest.c (so log -1 has something to show).
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 2; }
> > +     EOF
> > +     git add logtest.c &&
> > +     git commit -m "change logtest.c" &&
> > +
> > +     # Working tree modifications (not committed).
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     NEW5
> > +     NEW6
> > +     line7
> > +     line8
> > +     NEW9
> > +     NEW10
> > +     EOF
> > +
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 999; }
> > +     EOF
> > +
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 99; }
> > +     EOF
> > +
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 10; }
> > +     EOF
> > +
> > +     cat >two.c <<-\EOF
> > +     int second(void) { return 20; }
> > +     EOF
> > +'
> > +
> > +#
> > +# Core behavior: the tool controls which lines are marked as changed.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process hunk boundaries affect output' '
> > +     # The file has changes at lines 5-6 and 9-10, but fixed-hunk
> > +     # only reports lines 5-6 as changed.  Lines 9-10 should not
> > +     # appear as changed in the output.
> > +     git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
> > +             diff boundary.c >actual &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^-OLD6" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^+NEW6" actual &&
> > +     test_grep ! "^-OLD9" actual &&
> > +     test_grep ! "^-OLD10" actual &&
> > +     test_grep ! "^+NEW9" actual &&
> > +     test_grep ! "^+NEW10" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with new file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- newfile.c >actual 2>stderr &&
> > +     test_grep "return 99" actual &&
> > +     test_grep "pathname=newfile.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with added file (empty old side)' '
> > +     cat >added.c <<-\EOF &&
> > +     int added(void) { return 1; }
> > +     EOF
> > +     git add added.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --cached -- added.c >actual 2>stderr &&
> > +     test_grep "added" actual &&
> > +     test_grep "pathname=added.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process skipped for binary files' '
> > +     printf "\\0binary" >binary.c &&
> > +     git add binary.c &&
> > +     git commit -m "add binary" &&
> > +     printf "\\0changed" >binary.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- binary.c >actual &&
> > +     test_grep "Binary files" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
> > +     echo "not tracked by cdiff" >unmatched.txt &&
> > +     git add unmatched.txt &&
> > +     git commit -m "add unmatched.txt" &&
> > +
> > +     echo "modified" >unmatched.txt &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- unmatched.txt >actual &&
> > +     test_grep "modified" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'multiple drivers use separate processes' '
> > +     echo "*.h diff=hdiff" >>.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 1; }
> > +     EOF
> > +     git add multi.h &&
> > +     git commit -m "add multi.h" &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 2; }
> > +     EOF
> > +
> > +     rm -f backend-c.log backend-h.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
> > +         -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
> > +             diff -- newfile.c multi.h >actual 2>stderr &&
> > +     test_grep "pathname=newfile.c" backend-c.log &&
> > +     test_grep "pathname=multi.h" backend-h.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works alongside textconv' '
> > +     write_script uppercase-filter <<-\EOF &&
> > +     tr "a-z" "A-Z" <"$1"
> > +     EOF
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     hello world
> > +     EOF
> > +     git add textconv.c &&
> > +     git commit -m "add textconv.c" &&
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     goodbye world
> > +     EOF
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.textconv="./uppercase-filter" \
> > +         -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- textconv.c >actual 2>stderr &&
> > +     # The diff process receives textconv-transformed (uppercase) content.
> > +     test_grep "pathname=textconv.c" backend.log &&
> > +     test_grep "old=HELLO WORLD" backend.log &&
> > +     test_grep "new=GOODBYE WORLD" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +#
> > +# Downstream features: word diff, log, equivalent files, exit code.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process with --word-diff' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --word-diff worddiff.c >actual 2>stderr &&
> > +     test_grep "\[-1;-\]" actual &&
> > +     test_grep "{+999;+}" actual &&
> > +     test_grep "pathname=worddiff.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with git log -p' '
> > +     # With no-hunks mode, the tool says the files are equivalent,
> > +     # so log -p should show the commit but no diff content.
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
> > +             log -1 -p -- logtest.c >actual 2>stderr &&
> > +     test_grep "change logtest.c" actual &&
> > +     test_grep ! "return 2" actual &&
> > +     test_grep "command=hunks pathname=logtest.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 0; }
> > +     EOF
> > +     git add nohunks.c &&
> > +     git commit -m "add nohunks.c" &&
> > +
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 999; }
> > +     EOF
> > +
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff nohunks.c >actual &&
> > +     test_must_be_empty actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff --exit-code nohunks.c
> > +'
> > +
> > +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
> > +     test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
> > +             diff --exit-code newfile.c
> > +'
> > +
> > +#
> > +# Bypass mechanisms: flags and commands that skip the diff process.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --diff-algorithm=patience worddiff.c >actual &&
> > +     test_grep "return 999" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not used by --stat' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --stat worddiff.c >actual &&
> > +     test_grep "worddiff.c" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +#
> > +# Error handling and fallback.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool error status' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     # Fallback produces the full builtin diff (both change regions).
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Tool was contacted (it replied with error, not crash).
> > +     test_grep "command=hunks pathname=boundary.c" backend.log &&
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process error keeps tool available for next file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Unlike abort, error keeps the tool available: both files
> > +     # are sent to the tool (and both fall back).
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep "pathname=two.c" backend.log &&
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process abort disables for session' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
> > +             diff -- one.c two.c >actual &&
> > +     # Both files should still produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # The tool aborts on the first file and git clears its
> > +     # capability.  The second file never contacts the tool.
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep ! "pathname=two.c" backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool crash' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=crash" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Crash is a communication failure, so a warning is emitted.
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process startup failure only warns once' '
> > +     git -c diff.cdiff.process="/nonexistent/tool" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Both files produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # Sentinel prevents repeated warnings: only one, not one per file.
> > +     test_grep "diff process.*failed" stderr >warnings &&
> > +     test_line_count = 1 warnings
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on bad hunks' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Invalid hunks are caught by xdiff validation, not the
> > +     # protocol layer, so no warning is emitted.
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     EOF
> > +     git add synctest.c &&
> > +     git commit -m "add synctest.c" &&
> > +
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     changed
> > +     line3
> > +     EOF
> > +
> > +     # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
> > +     # line as changed, leaving 1 unchanged old vs 2 unchanged new.
> > +     # The synchronization invariant fails and git falls back.
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
> > +             diff synctest.c >actual 2>stderr &&
> > +     test_grep "changed" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
> > +     # boundary.c has 10 lines, so both hunks are in bounds
> > +     # but they overlap at lines 3-5, triggering the ordering check.
> > +     git -c diff.cdiff.process="$BACKEND --mode=overlap" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "NEW5" actual
> > +'
> > +
> > +test_done
> > diff --git a/userdiff.h b/userdiff.h
> > index 51c26e0d41..a98eabe377 100644
> > --- a/userdiff.h
> > +++ b/userdiff.h
> > @@ -3,6 +3,7 @@
> >
> >  #include "notes-cache.h"
> >
> > +struct diff_subprocess;
> >  struct index_state;
> >  struct repository;
> >
> > @@ -33,6 +34,8 @@ struct userdiff_driver {
> >       int textconv_want_cache;
> >       const char *process;
> >       char *process_owned;
> > +     struct diff_subprocess *diff_subprocess;
> > +     unsigned diff_process_failed : 1;
> >  };
> >  enum userdiff_driver_type {
> >       USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
> > --
> > gitgitgadget
> >
> >
> >

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> So the conscious project direction has been: fold pkt-line test backends
> into `test-tool` and drop the scripting-language prereq. Reintroducing the
> same shape in Python would walk this back.
> ...
> The `PYTHON` prereq exists in exactly five files today, all `git p4`
> related (where Python is an intrinsic prerequisite given that `git-p4.py`
> _is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
> `t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
> and `t/t9836-git-p4-metadata-encoding-python3.sh`.
> ...
> That commit is of course not intended to be used as-is; Feel free to pick
> code parts of it and integrate them into your topic branch. Or write your
> own test-tool helper from scratch if that's more your jam.

Showing better direction to new folks with such a clear thinking is
very much appreciated.  Even though it is natural and perfectly OK
for tests that interacts with parts of Git that are written in these
languages (e.g., we are OK for gitweb tests to require Perl), we
should consciously keep ourselves clean and not adding unnecessary
dependencies.

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

Michael Montalbo <mmontalbo@gmail.com> writes:

> On Sun, Jun 7, 2026 at 7:36 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>>
>> Hi Michael,
>>
>> I stumbled about this patch when it broke CI in Git for Windows, where we
>> do _not_ use `NO_PYTHON`, even though Python is unavailable in the
>> build/test CI jobs. The existing tests handle this situation gracefully,
>> this here patch does not:
>> ...
>> Given the complexity of what t4080 tries to test (error, abort, crash,
>> bad-sync, no-hunks, multiple files in one session, capability
>> negotiation), it would unfortunately be infeasible to use `test-tool
>> pkt-line` from a shell script implementing that `diff.*.process` protocol.
>>
>> So I've spiked a demo how the `test-tool diff-process-backend` could look
>> like (letting Opus do the menial typing, so that I can enjoy at least part
>> of a sunny Sunday outside), which also passes the CI build and test:
>> https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6
>>
>> That commit is of course not intended to be used as-is; Feel free to pick
>> code parts of it and integrate them into your topic branch. Or write your
>> own test-tool helper from scratch if that's more your jam.
>>
>
> Johannes, thank you for the great feedback. The historical context is
> really helpful and
> the concerns you raise make a lot of sense. I will take a look at your
> spike and also work
> on removing Python from the test.

Another request.

Please do not force readers to scroll through a ~800 line message
just to read only 5 lines of response from you.  Keep relevant parts
of the message you are responding to in your message to help readers
understand the context in which your response was made, but trim
everything else that is not relevant from your quote.

Thanks.

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

Michael Montalbo <mmontalbo@gmail.com> writes:

>> So the conscious project direction has been: fold pkt-line test backends
>> into `test-tool` and drop the scripting-language prereq. Reintroducing the
>> same shape in Python would walk this back.
>>
>> Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
>> 2025-04-29) has been another clear sign that the Git project is actively
>> _removing_ scripting-language dependencies from the build and test
>> infrastructure, not adding new ones.
>
> Now I wonder if the extension / addition of more Perl test infra with my other
> series:
>
> https://lore.kernel.org/git/pull.2135.git.1780559158.gitgitgadget@gmail.com/T/
>
> also goes against the project direction w.r.t. removing scripting languages.
> Maybe I should also re-evaluate the usage of Perl there. I am leveraging
> existing shell parsing logic written in Perl, but maybe the preference for
> Perl-based lint rules is a mistake and should be avoided.

That sounds prudent (even though it is outside the scope of _this_
topic, of course).

Thanks.

[PATCH v4 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's output entirely, losing downstream
features like word diff, function context, color, and blame.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes.

 * Patch 1: xdiff plumbing for externally supplied hunks.
 * Patch 2: diff.<driver>.process config key.
 * Patch 3: refactor subprocess API to separate process lifecycle from
   hashmap management, since the diff process stores its subprocess on the
   userdiff driver rather than in a hashmap.
 * Patch 4: the main feature.
 * Patch 5: bypass knobs (--no-ext-diff, format-patch).
 * Patch 6: blame integration so the tool can declare commits as having no
   changes.

Changes since v3:

 * Replaced Python test backend with C test-tool helper (thanks to Johannes
   Schindelin).
 * Added test coverage cases for deleted file, malformed hunk line, and
   missing capability.
 * Fixed potential overflow in synchronization invariant check by counting
   from changed[] arrays instead of accumulating.
 * Accept start=0 with count=0 in the hunk protocol, matching what git diff
   itself emits for empty file sides.
 * Warn on external hunk validation failure with specific reasons (range
   exceeded, overlap, sync mismatch) to help tool authors debug their
   implementations.
 * Test backend follows the same convention (start=0 when count=0 for empty
   file sides).

Michael Montalbo (6):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 143 ++++++
 Makefile                                 |   2 +
 blame.c                                  |  40 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 297 ++++++++++++
 diff-process.h                           |  39 ++
 diff.c                                   |  29 +-
 diff.h                                   |   5 +
 meson.build                              |   1 +
 sub-process.c                            |  28 +-
 sub-process.h                            |   9 +-
 t/helper/meson.build                     |   1 +
 t/helper/test-diff-process-backend.c     | 299 ++++++++++++
 t/helper/test-tool.c                     |   1 +
 t/helper/test-tool.h                     |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 553 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  14 +
 xdiff/xdiffi.c                           | 123 ++++-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 27 files changed, 1614 insertions(+), 21 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh


base-commit: ea97ad8d017de0c9037451a78008a0fd60abea0c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v3:

 1:  13eb201d63 ! 1:  03f261dfe2 xdiff: support external hunks via xpparam_t
     @@ xdiff/xdiff.h: typedef struct s_mmbuffer {
       
      +/*
      + * Hunk descriptor for externally computed diffs.
     -+ * Line numbers are 1-based, matching unified diff convention.
     ++ * Line numbers are 1-based; a start of 0 is accepted when
     ++ * count is 0 (empty file side, matching git diff output).
      + */
      +struct xdl_hunk {
      +	long old_start, old_count;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +{
      +	size_t i;
      +	long j, prev_old_end = 0, prev_new_end = 0;
     -+	long total_old = 0, total_new = 0;
     ++	long changed_old = 0, changed_new = 0;
      +
      +	/*
      +	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +	for (i = 0; i < nr_hunks; i++) {
      +		struct xdl_hunk *h = &hunks[i];
      +
     -+		if (h->old_count < 0 || h->new_count < 0)
     ++		if (h->old_count < 0 || h->new_count < 0) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"negative count (old=%ld, new=%ld)",
     ++				(uintmax_t)(i + 1),
     ++				h->old_count, h->new_count);
      +			return -1;
     -+		if (h->old_start < 1 || h->new_start < 1)
     ++		}
     ++		if (h->old_start < 1 || h->new_start < 1) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"start must be >= 1 (old=%ld, new=%ld)",
     ++				(uintmax_t)(i + 1),
     ++				h->old_start, h->new_start);
      +			return -1;
     ++		}
      +
      +		/*
      +		 * Range must fit: start + count - 1 <= nrec,
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +		 * start > nrec + 1 and allows start == nrec + 1
      +		 * (the position after the last line).
      +		 */
     -+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
     ++		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"old range %ld+%ld exceeds %lu lines",
     ++				(uintmax_t)(i + 1),
     ++				h->old_start, h->old_count,
     ++				(unsigned long)xe->xdf1.nrec);
      +			return -1;
     -+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
     ++		}
     ++		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"new range %ld+%ld exceeds %lu lines",
     ++				(uintmax_t)(i + 1),
     ++				h->new_start, h->new_count,
     ++				(unsigned long)xe->xdf2.nrec);
      +			return -1;
     ++		}
      +
      +		/* Ordering: no overlap with previous hunk (adjacent is OK) */
      +		if (h->old_start < prev_old_end ||
     -+		    h->new_start < prev_new_end)
     ++		    h->new_start < prev_new_end) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"overlaps with previous hunk",
     ++				(uintmax_t)(i + 1));
      +			return -1;
     ++		}
      +
      +		for (j = 0; j < h->old_count; j++)
      +			xe->xdf1.changed[h->old_start - 1 + j] = true;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +
      +		prev_old_end = h->old_start + h->old_count;
      +		prev_new_end = h->new_start + h->new_count;
     -+		total_old += h->old_count;
     -+		total_new += h->new_count;
      +	}
      +
      +	/*
      +	 * Synchronization invariant: unchanged line counts must match.
      +	 * Otherwise xdl_build_script() would walk off one array.
     ++	 *
     ++	 * Count changed lines from the arrays rather than accumulating
     ++	 * during the loop to avoid any overflow in the summation.
      +	 */
     -+	if ((long)xe->xdf1.nrec - total_old !=
     -+	    (long)xe->xdf2.nrec - total_new)
     ++	for (j = 0; j < (long)xe->xdf1.nrec; j++)
     ++		if (xe->xdf1.changed[j])
     ++			changed_old++;
     ++	for (j = 0; j < (long)xe->xdf2.nrec; j++)
     ++		if (xe->xdf2.changed[j])
     ++			changed_new++;
     ++	if ((long)xe->xdf1.nrec - changed_old !=
     ++	    (long)xe->xdf2.nrec - changed_new) {
     ++		warning("diff process: unchanged line count mismatch "
     ++			"(old: %ld unchanged, new: %ld unchanged)",
     ++			(long)xe->xdf1.nrec - changed_old,
     ++			(long)xe->xdf2.nrec - changed_new);
      +		return -1;
     ++	}
      +
      +	return 0;
      +}
 2:  58f4763c63 = 2:  30617ee17b userdiff: add diff.<driver>.process config
 3:  d6c833dd42 ! 3:  459e485e6d sub-process: separate process lifecycle from hashmap management
     @@ Commit message
          and subprocess_stop() become thin wrappers that add hashmap
          operations on top.
      
     -    No functional change for existing callers.
     -
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
       ## sub-process.c ##
     @@ sub-process.c: void subprocess_stop(struct hashmap *hashmap, struct subprocess_e
       	kill(entry->process.pid, SIGTERM);
       	finish_command(&entry->process);
      +}
     -+
     + 
      +void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
      +{
      +	if (!entry)
      +		return;
     - 
     ++
      +	subprocess_stop_command(entry);
       	hashmap_remove(hashmap, &entry->ent, NULL);
       }
     @@ sub-process.c: int subprocess_start(struct hashmap *hashmap, struct subprocess_e
      +	int err;
      +
      +	err = subprocess_start_command(entry, cmd, startfn);
     -+	if (err) {
     ++	if (err)
      +		return err;
     -+	}
      +
      +	hashmap_entry_init(&entry->ent, strhash(cmd));
       	hashmap_add(hashmap, &entry->ent);
 4:  d044fa0ee5 ! 4:  10b3980f59 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
          textconv-transformed content.  The tool controls which lines
          are marked as changed while the display shows the file content.
          Patch output features (word diff, function context, color) work
     -    normally; summary formats like --stat use their own diff path
     -    and are not affected.
     +    normally; --stat uses its own diff codepath and never consults
     +    the diff process.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
          requests send command=hunks, pathname, and both file contents as
     @@ Commit message
          "hunks populated" from "files equivalent" from "not applicable"
          from "tool failure."
      
     +    Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
       ## Documentation/config/diff.adoc ##
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +at line 3 in the old file were replaced by 4 lines starting at
      +line 3 in the new file.  An `<old_count>` of 0 means no lines were
      +removed (pure insertion); a `<new_count>` of 0 means no lines were
     -+added (pure deletion).
     ++added (pure deletion).  A start value of 0 is accepted when
     ++the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
     ++added file), matching what `git diff` itself emits for empty
     ++file sides.
      +
      +Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
      +file `"foo\nbar"` both have 2 lines.
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +packet:          git< 0000
      +-----------------------
      +
     -+If the tool returns invalid hunks (out of bounds, overlapping), Git
     -+silently falls back to the builtin diff algorithm.
     ++If the tool returns invalid hunks (out of bounds, overlapping, or
     ++mismatched unchanged line counts), Git warns and falls back to the
     ++builtin diff algorithm.
      +
      +In case the tool cannot or does not want to process the content,
      +it is expected to respond with an "error" status.  Git warns and
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
       
      
       ## Makefile ##
     +@@ Makefile: TEST_BUILTINS_OBJS += test-csprng.o
     + TEST_BUILTINS_OBJS += test-date.o
     + TEST_BUILTINS_OBJS += test-delete-gpgsig.o
     + TEST_BUILTINS_OBJS += test-delta.o
     ++TEST_BUILTINS_OBJS += test-diff-process-backend.o
     + TEST_BUILTINS_OBJS += test-dir-iterator.o
     + TEST_BUILTINS_OBJS += test-drop-caches.o
     + TEST_BUILTINS_OBJS += test-dump-cache-tree.o
      @@ Makefile: LIB_OBJS += diff-delta.o
       LIB_OBJS += diff-merges.o
       LIB_OBJS += diff-lib.o
     @@ diff-process.c (new)
      +	if (errno || end == line || *end != '\0')
      +		return -1;
      +
     ++	/*
     ++	 * git diff emits start=0 when count=0 (empty file side).
     ++	 * Normalize to 1-based so downstream validation can assume start >= 1.
     ++	 */
     ++	if (!hunk->old_count && !hunk->old_start)
     ++		hunk->old_start = 1;
     ++	if (!hunk->new_count && !hunk->new_start)
     ++		hunk->new_start = 1;
     ++
      +	return 0;
      +}
      +
     @@ meson.build: libgit_sources = [
         'diffcore-break.c',
         'diffcore-delta.c',
      
     - ## t/.gitattributes ##
     -@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
     - /t8005/*.txt eol=lf
     - /t9*/*.dump eol=lf
     - /t0040*.sh whitespace=-indent-with-non-tab
     -+/t4080-diff-process.sh whitespace=-indent-with-non-tab
     + ## t/helper/meson.build ##
     +@@ t/helper/meson.build: test_tool_sources = [
     +   'test-date.c',
     +   'test-delete-gpgsig.c',
     +   'test-delta.c',
     ++  'test-diff-process-backend.c',
     +   'test-dir-iterator.c',
     +   'test-drop-caches.c',
     +   'test-dump-cache-tree.c',
     +
     + ## t/helper/test-diff-process-backend.c (new) ##
     +@@
     ++/*
     ++ * Test backend for the long-running diff process protocol
     ++ * (see diff-process.c and Documentation/gitattributes.adoc).
     ++ *
     ++ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
     ++ *
     ++ * Implements the server side of the pkt-line handshake and a per-file
     ++ * response loop.  The --mode= switch selects the response shape
     ++ * (success, error, abort, crash, malformed hunks).
     ++ *
     ++ * Per-file request from Git:
     ++ *
     ++ *   packet:          git> command=hunks
     ++ *   packet:          git> pathname=<path>
     ++ *   packet:          git> 0000
     ++ *   packet:          git> OLD_CONTENT
     ++ *   packet:          git> 0000
     ++ *   packet:          git> NEW_CONTENT
     ++ *   packet:          git> 0000
     ++ *
     ++ * Response varies by --mode (default: whole-file):
     ++ *
     ++ *   whole-file   packet: git< hunk 1 <old_lines> 1 <new_lines>
     ++ *   fixed-hunk   packet: git< hunk 5 2 5 2
     ++ *   no-hunks     (no hunk packets)
     ++ *   bad-hunk     packet: git< hunk 999 1 999 1
     ++ *   bad-parse    packet: git< garbage not a hunk
     ++ *   bad-sync     packet: git< hunk 1 2 1 1
     ++ *   overlap      packet: git< hunk 1 5 1 5
     ++ *                packet: git< hunk 3 2 3 2
     ++ *   no-cap       (omits capability=hunks during handshake)
     ++ *   error        (status=error instead of status=success)
     ++ *   abort        (status=abort instead of status=success)
     ++ *   crash        exit(1) before sending any response
     ++ *
     ++ * All non-error/abort modes end with:
     ++ *
     ++ *   packet:          git< 0000
     ++ *   packet:          git< status=success
     ++ *   packet:          git< 0000
     ++ *
     ++ * Each request is logged to --log as:
     ++ *
     ++ *   command=<cmd> pathname=<path> old=<first line> new=<first line>
     ++ */
     ++
     ++#include "test-tool.h"
     ++#include "pkt-line.h"
     ++#include "parse-options.h"
     ++#include "strbuf.h"
     ++
     ++static FILE *logfile;
     ++
     ++enum mode {
     ++	MODE_WHOLE_FILE,
     ++	MODE_FIXED_HUNK,
     ++	MODE_NO_HUNKS,
     ++	MODE_BAD_HUNK,
     ++	MODE_BAD_PARSE,
     ++	MODE_BAD_SYNC,
     ++	MODE_OVERLAP,
     ++	MODE_NO_CAP,
     ++	MODE_ERROR,
     ++	MODE_ABORT,
     ++	MODE_CRASH,
     ++};
     ++
     ++static enum mode parse_mode(const char *s)
     ++{
     ++	if (!strcmp(s, "whole-file"))
     ++		return MODE_WHOLE_FILE;
     ++	if (!strcmp(s, "fixed-hunk"))
     ++		return MODE_FIXED_HUNK;
     ++	if (!strcmp(s, "no-hunks"))
     ++		return MODE_NO_HUNKS;
     ++	if (!strcmp(s, "bad-hunk"))
     ++		return MODE_BAD_HUNK;
     ++	if (!strcmp(s, "bad-parse"))
     ++		return MODE_BAD_PARSE;
     ++	if (!strcmp(s, "bad-sync"))
     ++		return MODE_BAD_SYNC;
     ++	if (!strcmp(s, "overlap"))
     ++		return MODE_OVERLAP;
     ++	if (!strcmp(s, "no-cap"))
     ++		return MODE_NO_CAP;
     ++	if (!strcmp(s, "error"))
     ++		return MODE_ERROR;
     ++	if (!strcmp(s, "abort"))
     ++		return MODE_ABORT;
     ++	if (!strcmp(s, "crash"))
     ++		return MODE_CRASH;
     ++	die("unknown --mode=%s", s);
     ++}
     ++
     ++/*
     ++ * Read "key=value" packets up to a flush, capturing "command" and
     ++ * "pathname".  Returns 1 if a request was read, 0 on EOF.
     ++ *
     ++ * The first packet uses the gentle variant so that a clean shutdown
     ++ * by Git (EOF) does not produce a spurious "the remote end hung up
     ++ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
     ++ * variant: once inside a request, truncation is a protocol violation
     ++ * and dying loudly is the correct response.
     ++ */
     ++static int read_request_header(char **command, char **pathname)
     ++{
     ++	int first = 1;
     ++	char *line;
     ++
     ++	*command = *pathname = NULL;
     ++	for (;;) {
     ++		const char *value;
     ++
     ++		if (first) {
     ++			if (packet_read_line_gently(0, NULL, &line) < 0)
     ++				return 0;
     ++			first = 0;
     ++		} else {
     ++			line = packet_read_line(0, NULL);
     ++		}
     ++		if (!line)
     ++			break;
     ++		if (skip_prefix(line, "command=", &value))
     ++			*command = xstrdup(value);
     ++		else if (skip_prefix(line, "pathname=", &value))
     ++			*pathname = xstrdup(value);
     ++	}
     ++	return 1;
     ++}
     ++
     ++static size_t count_lines(const struct strbuf *buf)
     ++{
     ++	size_t lines = 0;
     ++
     ++	for (size_t i = 0; i < buf->len; i++)
     ++		if (buf->buf[i] == '\n')
     ++			lines++;
     ++
     ++	return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
     ++}
     ++
     ++static void send_status(const char *status)
     ++{
     ++	packet_flush(1);
     ++	packet_write_fmt(1, "%s\n", status);
     ++	packet_flush(1);
     ++}
     ++
     ++static void respond(enum mode mode,
     ++		    const struct strbuf *old_buf,
     ++		    const struct strbuf *new_buf)
     ++{
     ++	switch (mode) {
     ++	case MODE_ERROR:
     ++		send_status("status=error");
     ++		return;
     ++	case MODE_ABORT:
     ++		send_status("status=abort");
     ++		return;
     ++	case MODE_CRASH:
     ++		exit(1);
     ++	case MODE_FIXED_HUNK:
     ++		packet_write_fmt(1, "hunk 5 2 5 2\n");
     ++		break;
     ++	case MODE_BAD_HUNK:
     ++		packet_write_fmt(1, "hunk 999 1 999 1\n");
     ++		break;
     ++	case MODE_BAD_PARSE:
     ++		packet_write_fmt(1, "garbage not a hunk\n");
     ++		break;
     ++	case MODE_BAD_SYNC:
     ++		packet_write_fmt(1, "hunk 1 2 1 1\n");
     ++		break;
     ++	case MODE_OVERLAP:
     ++		packet_write_fmt(1, "hunk 1 5 1 5\n");
     ++		packet_write_fmt(1, "hunk 3 2 3 2\n");
     ++		break;
     ++	case MODE_NO_HUNKS:
     ++		break;
     ++	case MODE_NO_CAP:
     ++	case MODE_WHOLE_FILE: {
     ++		size_t old_lines = count_lines(old_buf);
     ++		size_t new_lines = count_lines(new_buf);
     ++		/*
     ++		 * Match git diff output: start=0 when count=0
     ++		 * (empty file side), 1 otherwise.
     ++		 */
     ++		packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
     ++				 " %"PRIuMAX" %"PRIuMAX"\n",
     ++				 (uintmax_t)(old_lines ? 1 : 0),
     ++				 (uintmax_t)old_lines,
     ++				 (uintmax_t)(new_lines ? 1 : 0),
     ++				 (uintmax_t)new_lines);
     ++		break;
     ++	}
     ++	}
     ++	send_status("status=success");
     ++}
     ++
     ++static void command_loop(enum mode mode)
     ++{
     ++	for (;;) {
     ++		char *command = NULL, *pathname = NULL;
     ++		struct strbuf obuf = STRBUF_INIT;
     ++		struct strbuf nbuf = STRBUF_INIT;
     ++
     ++		if (!read_request_header(&command, &pathname))
     ++			break; /* EOF: Git closed its end */
     ++
     ++		read_packetized_to_strbuf(0, &obuf, 0);
     ++		read_packetized_to_strbuf(0, &nbuf, 0);
     ++
     ++		if (logfile) {
     ++			fprintf(logfile,
     ++				"command=%s pathname=%s old=%.*s new=%.*s\n",
     ++				command ? command : "(none)",
     ++				pathname ? pathname : "(none)",
     ++				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
     ++				obuf.buf,
     ++				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
     ++				nbuf.buf);
     ++			fflush(logfile);
     ++		}
     ++
     ++		respond(mode, &obuf, &nbuf);
     ++
     ++		free(command);
     ++		free(pathname);
     ++		strbuf_release(&obuf);
     ++		strbuf_release(&nbuf);
     ++	}
     ++}
     ++
     ++static void handshake(enum mode mode)
     ++{
     ++	char *line;
     ++
     ++	line = packet_read_line(0, NULL);
     ++	if (!line || strcmp(line, "git-diff-client"))
     ++		die("bad welcome: '%s'", line ? line : "(eof)");
     ++	line = packet_read_line(0, NULL);
     ++	if (!line || strcmp(line, "version=1"))
     ++		die("bad version: '%s'", line ? line : "(eof)");
     ++	if (packet_read_line(0, NULL))
     ++		die("expected flush after version");
     ++
     ++	packet_write_fmt(1, "git-diff-server\n");
     ++	packet_write_fmt(1, "version=1\n");
     ++	packet_flush(1);
     ++
     ++	/* Drain capabilities advertised by Git */
     ++	while ((line = packet_read_line(0, NULL)))
     ++		; /* drain */
     ++
     ++	/* Respond with our capabilities (or none for no-cap mode) */
     ++	if (mode != MODE_NO_CAP)
     ++		packet_write_fmt(1, "capability=hunks\n");
     ++	packet_flush(1);
     ++}
     ++
     ++static const char *const usage_str[] = {
     ++	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
     ++	NULL
     ++};
     ++
     ++int cmd__diff_process_backend(int argc, const char **argv)
     ++{
     ++	const char *mode_str = NULL, *log_path = NULL;
     ++	enum mode mode = MODE_WHOLE_FILE;
     ++	struct option options[] = {
     ++		OPT_STRING(0, "mode", &mode_str, "mode",
     ++			   "response shape: whole-file (default), fixed-hunk,"
     ++			   " no-hunks, bad-hunk, bad-sync, overlap, error,"
     ++			   " abort, crash"),
     ++		OPT_STRING(0, "log", &log_path, "path",
     ++			   "append per-request summary to this file"),
     ++		OPT_END()
     ++	};
     ++
     ++	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
     ++	if (argc)
     ++		usage_with_options(usage_str, options);
     ++
     ++	if (mode_str)
     ++		mode = parse_mode(mode_str);
     ++
     ++	if (log_path) {
     ++		logfile = fopen(log_path, "a");
     ++		if (!logfile)
     ++			die_errno("failed to open log '%s'", log_path);
     ++	}
     ++
     ++	handshake(mode);
     ++	command_loop(mode);
     ++
     ++	if (logfile && fclose(logfile))
     ++		die_errno("error closing log");
     ++	return 0;
     ++}
     +
     + ## t/helper/test-tool.c ##
     +@@ t/helper/test-tool.c: static struct test_cmd cmds[] = {
     + 	{ "date", cmd__date },
     + 	{ "delete-gpgsig", cmd__delete_gpgsig },
     + 	{ "delta", cmd__delta },
     ++	{ "diff-process-backend", cmd__diff_process_backend },
     + 	{ "dir-iterator", cmd__dir_iterator },
     + 	{ "drop-caches", cmd__drop_caches },
     + 	{ "dump-cache-tree", cmd__dump_cache_tree },
     +
     + ## t/helper/test-tool.h ##
     +@@ t/helper/test-tool.h: int cmd__csprng(int argc, const char **argv);
     + int cmd__date(int argc, const char **argv);
     + int cmd__delta(int argc, const char **argv);
     + int cmd__delete_gpgsig(int argc, const char **argv);
     ++int cmd__diff_process_backend(int argc, const char **argv);
     + int cmd__dir_iterator(int argc, const char **argv);
     + int cmd__drop_caches(int argc, const char **argv);
     + int cmd__dump_cache_tree(int argc, const char **argv);
      
       ## t/meson.build ##
      @@ t/meson.build: integration_tests = [
     @@ t/t4080-diff-process.sh (new)
      +
      +. ./test-lib.sh
      +
     -+if test_have_prereq PYTHON
     -+then
     -+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
     -+fi
     ++# See t/helper/test-diff-process-backend.c for the backend implementation
     ++# and available --mode= options.
      +
     -+#
     -+# A single parametric diff process.
     -+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
     -+#
     -+# Modes:
     -+#   whole-file  - report all lines as changed (default)
     -+#   fixed-hunk  - always report hunk 5 2 5 2
     -+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
     -+#   bad-sync    - report hunk with mismatched unchanged totals
     -+#   overlap     - report two overlapping hunks
     -+#   no-hunks   - return no hunks (files considered equivalent)
     -+#   error       - return status=error for every request
     -+#   abort       - return status=abort for every request
     -+#   crash       - read one request then exit without responding
     -+#
     -+setup_backend () {
     -+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
     -+	import sys, os
     -+
     -+	def read_pkt():
     -+	    hdr = sys.stdin.buffer.read(4)
     -+	    if len(hdr) < 4: return None
     -+	    length = int(hdr, 16)
     -+	    if length == 0: return ""
     -+	    data = sys.stdin.buffer.read(length - 4)
     -+	    return data.decode().rstrip("\n")
     -+
     -+	def write_pkt(line):
     -+	    data = (line + "\n").encode()
     -+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
     -+	    sys.stdout.buffer.flush()
     -+
     -+	def write_flush():
     -+	    sys.stdout.buffer.write(b"0000")
     -+	    sys.stdout.buffer.flush()
     -+
     -+	def read_content():
     -+	    chunks = []
     -+	    while True:
     -+	        hdr = sys.stdin.buffer.read(4)
     -+	        if len(hdr) < 4: break
     -+	        length = int(hdr, 16)
     -+	        if length == 0: break
     -+	        chunks.append(sys.stdin.buffer.read(length - 4))
     -+	    return b"".join(chunks)
     -+
     -+	mode = "whole-file"
     -+	logfile = None
     -+	for arg in sys.argv[1:]:
     -+	    if arg.startswith("--mode="):
     -+	        mode = arg[7:]
     -+	    elif arg.startswith("--log="):
     -+	        logfile = open(arg[6:], "a")
     -+
     -+	def log(msg):
     -+	    if logfile:
     -+	        logfile.write(msg + "\n")
     -+	        logfile.flush()
     -+
     -+	# Handshake
     -+	assert read_pkt() == "git-diff-client"
     -+	assert read_pkt() == "version=1"
     -+	read_pkt()
     -+	write_pkt("git-diff-server")
     -+	write_pkt("version=1")
     -+	write_flush()
     -+	while True:
     -+	    p = read_pkt()
     -+	    if p == "": break
     -+	write_pkt("capability=hunks")
     -+	write_flush()
     -+
     -+	log("ready")
     -+
     -+	while True:
     -+	    cmd = None
     -+	    pathname = None
     -+	    while True:
     -+	        p = read_pkt()
     -+	        if p is None: sys.exit(0)
     -+	        if p == "": break
     -+	        if p.startswith("command="): cmd = p.split("=",1)[1]
     -+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
     -+	    if cmd is None: sys.exit(0)
     -+	    old = read_content()
     -+	    new = read_content()
     -+	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
     -+	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
     -+	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
     -+
     -+	    if mode == "error":
     -+	        write_flush()
     -+	        write_pkt("status=error")
     -+	        write_flush()
     -+	        continue
     -+
     -+	    if mode == "abort":
     -+	        write_flush()
     -+	        write_pkt("status=abort")
     -+	        write_flush()
     -+	        continue
     -+
     -+	    if mode == "crash":
     -+	        sys.exit(1)
     -+
     -+	    if cmd == "hunks":
     -+	        if mode == "fixed-hunk":
     -+	            write_pkt("hunk 5 2 5 2")
     -+	        elif mode == "bad-hunk":
     -+	            write_pkt("hunk 999 1 999 1")
     -+	        elif mode == "bad-sync":
     -+	            write_pkt("hunk 1 2 1 1")
     -+	        elif mode == "overlap":
     -+	            write_pkt("hunk 1 5 1 5")
     -+	            write_pkt("hunk 3 2 3 2")
     -+	        elif mode == "no-hunks":
     -+	            pass
     -+	        else:
     -+	            ol = old.count(b"\n")
     -+	            nl = new.count(b"\n")
     -+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
     -+	        write_flush()
     -+	        write_pkt("status=success")
     -+	        write_flush()
     -+	    else:
     -+	        write_flush()
     -+	        write_pkt("status=error")
     -+	        write_flush()
     -+	PYEOF
     -+	write_script diff-process-backend <<-SHEOF
     -+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
     -+	SHEOF
     -+}
     -+
     -+BACKEND="./diff-process-backend"
     ++BACKEND="test-tool diff-process-backend"
      +
     -+test_expect_success PYTHON 'setup' '
     -+	setup_backend &&
     ++test_expect_success 'setup' '
      +	echo "*.c diff=cdiff" >.gitattributes &&
      +	git add .gitattributes &&
      +
     @@ t/t4080-diff-process.sh (new)
      +	git add worddiff.c &&
      +
      +	# newfile.c: single-line function, value changes 42 -> 99.
     -+	# Used by: new file, --exit-code, multiple drivers.
     ++	# Used by: modified file, --exit-code, multiple drivers.
      +	cat >newfile.c <<-\EOF &&
      +	int new_func(void) { return 42; }
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +# Core behavior: the tool controls which lines are marked as changed.
      +#
      +
     -+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
     ++test_expect_success 'diff process hunk boundaries affect output' '
      +	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
      +	# only reports lines 5-6 as changed.  Lines 9-10 should not
      +	# appear as changed in the output.
     @@ t/t4080-diff-process.sh (new)
      +	test_grep ! "^+NEW10" actual
      +'
      +
     -+test_expect_success PYTHON 'diff process works with new file' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process works with modified file' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- newfile.c >actual 2>stderr &&
      +	test_grep "return 99" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process works with added file (empty old side)' '
     ++test_expect_success 'diff process works with added file (empty old side)' '
      +	cat >added.c <<-\EOF &&
      +	int added(void) { return 1; }
      +	EOF
      +	git add added.c &&
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --cached -- added.c >actual 2>stderr &&
      +	test_grep "added" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process skipped for binary files' '
     ++test_expect_success 'diff process works with deleted file (empty new side)' '
     ++	git add added.c &&
     ++	git commit -m "commit added.c" &&
     ++	git rm added.c &&
     ++
     ++	test_when_finished "rm -f backend.log" &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --cached -- added.c >actual 2>stderr &&
     ++	test_grep "deleted file" actual &&
     ++	test_grep "pathname=added.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process skipped for binary files' '
      +	printf "\\0binary" >binary.c &&
      +	git add binary.c &&
      +	git commit -m "add binary" &&
      +	printf "\\0changed" >binary.c &&
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- binary.c >actual &&
      +	test_grep "Binary files" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
     ++test_expect_success 'diff process not consulted for unmatched driver' '
      +	echo "not tracked by cdiff" >unmatched.txt &&
      +	git add unmatched.txt &&
      +	git commit -m "add unmatched.txt" &&
      +
      +	echo "modified" >unmatched.txt &&
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- unmatched.txt >actual &&
      +	test_grep "modified" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'multiple drivers use separate processes' '
     ++test_expect_success 'multiple drivers use separate processes' '
      +	echo "*.h diff=hdiff" >>.gitattributes &&
      +	git add .gitattributes &&
      +
     @@ t/t4080-diff-process.sh (new)
      +	int header(void) { return 2; }
      +	EOF
      +
     -+	rm -f backend-c.log backend-h.log &&
     ++	test_when_finished "rm -f backend-c.log backend-h.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
      +	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
      +		diff -- newfile.c multi.h >actual 2>stderr &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process works alongside textconv' '
     ++test_expect_success 'diff process works alongside textconv' '
      +	write_script uppercase-filter <<-\EOF &&
      +	tr "a-z" "A-Z" <"$1"
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +	goodbye world
      +	EOF
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.textconv="./uppercase-filter" \
      +	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- textconv.c >actual 2>stderr &&
     @@ t/t4080-diff-process.sh (new)
      +# Downstream features: word diff, log, equivalent files, exit code.
      +#
      +
     -+test_expect_success PYTHON 'diff process with --word-diff' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process with --word-diff' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --word-diff worddiff.c >actual 2>stderr &&
      +	test_grep "\[-1;-\]" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process works with git log -p' '
     ++test_expect_success 'diff process works with git log -p' '
      +	# With no-hunks mode, the tool says the files are equivalent,
      +	# so log -p should show the commit but no diff content.
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
      +		log -1 -p -- logtest.c >actual 2>stderr &&
      +	test_grep "change logtest.c" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
     ++test_expect_success 'diff process no hunks suppresses diff output' '
      +	cat >nohunks.c <<-\EOF &&
      +	int zero(void) { return 0; }
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty actual
      +'
      +
     -+test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
     ++test_expect_success 'diff process no hunks with --exit-code returns success' '
      +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
      +		diff --exit-code nohunks.c
      +'
      +
     -+test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
     ++test_expect_success 'diff process with --exit-code and hunks returns failure' '
      +	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
      +		diff --exit-code newfile.c
      +'
     @@ t/t4080-diff-process.sh (new)
      +# Bypass mechanisms: flags and commands that skip the diff process.
      +#
      +
     -+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process bypassed by --diff-algorithm' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --diff-algorithm=patience worddiff.c >actual &&
      +	test_grep "return 999" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process not used by --stat' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process not used by --stat' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --stat worddiff.c >actual &&
      +	test_grep "worddiff.c" actual &&
     @@ t/t4080-diff-process.sh (new)
      +# Error handling and fallback.
      +#
      +
     -+test_expect_success PYTHON 'diff process fallback on tool error status' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process fallback on tool error status' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
      +		diff boundary.c >actual 2>stderr &&
      +	# Fallback produces the full builtin diff (both change regions).
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process error keeps tool available for next file' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process error keeps tool available for next file' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
      +		diff -- one.c two.c >actual 2>stderr &&
      +	# Unlike abort, error keeps the tool available: both files
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "pathname=one.c" backend.log &&
      +	test_grep "pathname=two.c" backend.log &&
      +	test_grep "return 10" actual &&
     -+	test_grep "return 20" actual
     ++	test_grep "return 20" actual &&
     ++	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process abort disables for session' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process abort disables for session' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
     -+		diff -- one.c two.c >actual &&
     ++		diff -- one.c two.c >actual 2>stderr &&
      +	# Both files should still produce diff output via fallback.
      +	test_grep "return 10" actual &&
      +	test_grep "return 20" actual &&
      +	# The tool aborts on the first file and git clears its
      +	# capability.  The second file never contacts the tool.
      +	test_grep "pathname=one.c" backend.log &&
     -+	test_grep ! "pathname=two.c" backend.log
     ++	test_grep ! "pathname=two.c" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on tool crash' '
     ++test_expect_success 'diff process fallback on tool crash' '
      +	git -c diff.cdiff.process="$BACKEND --mode=crash" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "^-OLD5" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process startup failure only warns once' '
     ++test_expect_success 'diff process startup failure only warns once' '
      +	git -c diff.cdiff.process="/nonexistent/tool" \
      +		diff -- one.c two.c >actual 2>stderr &&
      +	# Both files produce diff output via fallback.
     @@ t/t4080-diff-process.sh (new)
      +	test_line_count = 1 warnings
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on bad hunks' '
     ++
     ++test_expect_success 'diff process fallback on bad hunks' '
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "^-OLD5" actual &&
      +	test_grep "^+NEW5" actual &&
      +	test_grep "^-OLD9" actual &&
      +	test_grep "^+NEW9" actual &&
     -+	# Invalid hunks are caught by xdiff validation, not the
     -+	# protocol layer, so no warning is emitted.
     -+	test_must_be_empty stderr
     ++	test_grep "exceeds.*lines" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
     ++test_expect_success 'diff process fallback on mismatched unchanged totals' '
      +	cat >synctest.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh (new)
      +	# The synchronization invariant fails and git falls back.
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
      +		diff synctest.c >actual 2>stderr &&
     -+	test_grep "changed" actual
     ++	test_grep "changed" actual &&
     ++	test_grep "unchanged line count mismatch" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     ++test_expect_success 'diff process fallback on overlapping hunks' '
      +	# boundary.c has 10 lines, so both hunks are in bounds
      +	# but they overlap at lines 3-5, triggering the ordering check.
      +	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
      +		diff boundary.c >actual 2>stderr &&
     -+	test_grep "NEW5" actual
     ++	test_grep "NEW5" actual &&
     ++	test_grep "overlaps with previous" stderr
     ++'
     ++
     ++test_expect_success 'diff process fallback on malformed hunk line' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual
     ++'
     ++
     ++test_expect_success 'diff process skipped when tool omits capability' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_must_be_empty stderr
      +'
      +
      +test_done
 5:  f4fd9aa682 ! 5:  6ec6716ea4 diff: bypass diff process with --no-ext-diff and in format-patch
     @@ Commit message
          external tool.
      
          Document that --diff-algorithm also bypasses the diff process,
     -    since it sets ignore_driver_algorithm which diff_process_fill_hunks
     -    already checks.
     +    since it forces the builtin algorithm.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ diff.h: struct diff_flags {
       	/**
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
     +@@ t/t4080-diff-process.sh: test_expect_success 'diff process bypassed by --diff-algorithm' '
       	test_path_is_missing backend.log
       '
       
     -+test_expect_success PYTHON 'diff process bypassed by --no-ext-diff' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process bypassed by --no-ext-diff' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --no-ext-diff worddiff.c >actual &&
      +	test_grep "return 999" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process not used by format-patch' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process not used by format-patch' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		format-patch -1 --stdout -- logtest.c >actual &&
      +	test_grep "return 2" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     - test_expect_success PYTHON 'diff process not used by --stat' '
     - 	rm -f backend.log &&
     + test_expect_success 'diff process not used by --stat' '
     + 	test_when_finished "rm -f backend.log" &&
       	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
 6:  370e766978 ! 6:  3dadafa1bc blame: consult diff process for no-hunk detection
     @@ Commit message
          The consultation happens at the pass_blame_to_parent() callsite
          using diff_process_fill_hunks(), matching how builtin_diff() in
          diff.c uses the same function.  A new diff_hunks_xpp() variant
     -    accepts a pre-populated xpparam_t for this callsite, while the
     -    existing diff_hunks() retains its original signature and behavior.
     -    The copy-detection callsite is unaffected since it does not use
     -    the diff process.
     +    accepts a pre-populated xpparam_t so callers can pass external
     +    hunks, while the existing diff_hunks() retains its original
     +    signature and behavior.  The copy-detection callsite is
     +    unaffected since it does not use the diff process.
      
          The subprocess is long-running (one startup cost amortized
          across the blame traversal), but each commit in the file's
     @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       		    parent, target, 0);
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     - 	test_grep "NEW5" actual
     +@@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omits capability' '
     + 	test_must_be_empty stderr
       '
       
      +#
      +# Blame integration.
      +#
      +
     -+test_expect_success PYTHON 'blame uses tool-provided hunks' '
     ++test_expect_success 'blame uses tool-provided hunks' '
      +	cat >blame-hunk.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	test_grep "$CHANGE" line6
      +'
      +
     -+test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
     ++test_expect_success 'blame skips commits with no hunks from diff process' '
      +	cat >blame.c <<-\EOF &&
     -+	int main(void)
     -+	{
     -+	    return 0;
     ++	int main(void) {
     ++	return 0;
      +	}
      +	EOF
      +	git add blame.c &&
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	cat >blame.c <<-\EOF &&
      +	int main(void)
      +	{
     -+	        return 0;
     ++	return 0;
      +	}
      +	EOF
      +	git add blame.c &&
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	test_grep "$ORIG_COMMIT" with
      +'
      +
     -+test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
     -+	rm -f backend.log &&
     ++test_expect_success 'blame --no-ext-diff bypasses diff process' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
      +		blame --no-ext-diff blame.c >actual &&
      +	# Without the process, blame attributes the reformat commit normally.
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
     ++test_expect_success 'blame --no-ext-diff uses builtin hunks' '
      +	# fixed-hunk mode would narrow blame to lines 5-6, but
      +	# --no-ext-diff should bypass it and use the builtin diff.
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
      +		blame --no-ext-diff blame-hunk.c >actual &&
      +	# Builtin diff attributes lines 9-10 to the change commit.

-- 
gitgitgadget

[PATCH v4 1/6] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm; this re-runs xdl_prepare_env() since
the first call may have dirtied the changed[] arrays.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |   7 ++-
 xdiff/xdiff.h     |  14 ++++++
 xdiff/xdiffi.c    | 123 +++++++++++++++++++++++++++++++++++++++++++++-
 xdiff/xprepare.c  |  10 ++++
 xdiff/xprepare.h  |   1 +
 5 files changed, 152 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index 5ee2b96d0a..76a24fc589 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..dd4915fe16 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,16 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based; a start of 0 is accepted when
+ * count is 0 (empty file side, matching git diff output).
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +98,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
+	struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index c5a892f91e..bf820b52e3 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,135 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long changed_old = 0, changed_new = 0;
+
+	/*
+	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
+	 * Clear them so only the external hunks are marked.
+	 */
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0) {
+			warning("diff process hunk %"PRIuMAX": "
+				"negative count (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_count, h->new_count);
+			return -1;
+		}
+		if (h->old_start < 1 || h->new_start < 1) {
+			warning("diff process hunk %"PRIuMAX": "
+				"start must be >= 1 (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_start, h->new_start);
+			return -1;
+		}
+
+		/*
+		 * Range must fit: start + count - 1 <= nrec,
+		 * rewritten to avoid overflow.  Same for both sides.
+		 *
+		 * When count is 0 (pure insert/delete) the check
+		 * reduces to 0 > nrec - start + 1, which rejects
+		 * start > nrec + 1 and allows start == nrec + 1
+		 * (the position after the last line).
+		 */
+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
+			warning("diff process hunk %"PRIuMAX": "
+				"old range %ld+%ld exceeds %lu lines",
+				(uintmax_t)(i + 1),
+				h->old_start, h->old_count,
+				(unsigned long)xe->xdf1.nrec);
+			return -1;
+		}
+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
+			warning("diff process hunk %"PRIuMAX": "
+				"new range %ld+%ld exceeds %lu lines",
+				(uintmax_t)(i + 1),
+				h->new_start, h->new_count,
+				(unsigned long)xe->xdf2.nrec);
+			return -1;
+		}
+
+		/* Ordering: no overlap with previous hunk (adjacent is OK) */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end) {
+			warning("diff process hunk %"PRIuMAX": "
+				"overlaps with previous hunk",
+				(uintmax_t)(i + 1));
+			return -1;
+		}
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 *
+	 * Count changed lines from the arrays rather than accumulating
+	 * during the loop to avoid any overflow in the summation.
+	 */
+	for (j = 0; j < (long)xe->xdf1.nrec; j++)
+		if (xe->xdf1.changed[j])
+			changed_old++;
+	for (j = 0; j < (long)xe->xdf2.nrec; j++)
+		if (xe->xdf2.changed[j])
+			changed_new++;
+	if ((long)xe->xdf1.nrec - changed_old !=
+	    (long)xe->xdf2.nrec - changed_new) {
+		warning("diff process: unchanged line count mismatch "
+			"(old: %ld unchanged, new: %ld unchanged)",
+			(long)xe->xdf1.nrec - changed_old,
+			(long)xe->xdf2.nrec - changed_new);
+		return -1;
+	}
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index 11bada2608..f4ab935332 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v4 2/6] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index b5412e6bc3..7547874aa2 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v4 3/6] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and
managing a hashmap that indexes running processes by command
string.  The hashmap suits callers like convert.c where many
files may share one filter process looked up by name, but
callers that manage process lifetime through their own data
structures do not need it.

Extract subprocess_start_command() and subprocess_stop_command()
so callers can reuse the child process setup and handshake
machinery without maintaining a hashmap.  subprocess_start()
and subprocess_stop() become thin wrappers that add hashmap
operations on top.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 28 +++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 83bf0a0e82..5468939338 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
 
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
+
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err)
+		return err;
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
gitgitgadget

[PATCH v4 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference line numbers in the content.
When textconv is also configured, the tool receives the
textconv-transformed content.  The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
normally; --stat uses its own diff codepath and never consults
the diff process.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet (success, error, or abort).  On error, Git warns and falls
back to the builtin diff algorithm for that file.  On abort, Git
silently falls back for the current file and stops sending further
requests to the tool for the remainder of the session.

When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
This also means --exit-code reports no changes for that file.

The subprocess is stored on the userdiff_driver struct and
launched on first use.  If the process fails to start, the
handshake fails, or a communication error occurs mid-stream,
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.

diff_process_fill_hunks() is the sole public entry point.  It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc       |   5 +
 Documentation/gitattributes.adoc     | 143 +++++++++
 Makefile                             |   2 +
 diff-process.c                       | 297 ++++++++++++++++++
 diff-process.h                       |  39 +++
 diff.c                               |  13 +
 diff.h                               |   3 +
 meson.build                          |   1 +
 t/helper/meson.build                 |   1 +
 t/helper/test-diff-process-backend.c | 299 ++++++++++++++++++
 t/helper/test-tool.c                 |   1 +
 t/helper/test-tool.h                 |   1 +
 t/meson.build                        |   1 +
 t/t4080-diff-process.sh              | 432 +++++++++++++++++++++++++++
 userdiff.h                           |   3 +
 15 files changed, 1241 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..ac0635bb3b 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index bd76167a45..49ed11d069 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,149 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If `diff.<name>.process` is defined, Git sends the old and new file
+content to an external tool and receives back a list of changed
+regions (pairs of line ranges in the old and new file).  Git uses
+these instead of its builtin diff algorithm, but still controls
+all output formatting, so features like word diff, function context,
+color, and blame work normally.  This is achieved by using the
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
+the diff process feeds results back into the standard pipeline.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+Then, define a "diff.<name>.process" configuration to specify
+the diff process command.
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+When Git encounters the first file that needs to be diffed, it starts
+the process and performs the handshake.  In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
+described below).
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
+data, each terminated with a flush packet.  The pathname is relative
+to the repository root.  When `diff.<name>.textconv` is also set,
+the tool receives the textconv-transformed content rather than the
+raw blob.  Git does not send binary files to the diff process.
+
+-----------------------
+packet:          git> command=hunks
+packet:          git> pathname=path/file.c
+packet:          git> 0000
+packet:          git> OLD_CONTENT
+packet:          git> 0000
+packet:          git> NEW_CONTENT
+packet:          git> 0000
+-----------------------
+
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
+
+  `hunk <old_start> <old_count> <new_start> <new_count>`
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
+replacement range in the new file.  Start values are 1-based and
+counts are non-negative.  Ranges must not extend beyond the end of
+the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
+at line 3 in the old file were replaced by 4 lines starting at
+line 3 in the new file.  An `<old_count>` of 0 means no lines were
+removed (pure insertion); a `<new_count>` of 0 means no lines were
+added (pure deletion).  A start value of 0 is accepted when
+the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
+added file), matching what `git diff` itself emits for empty
+file sides.
+
+Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
+Hunks must be listed in order and must not overlap.  Any line
+not covered by a hunk is treated as unchanged, so the total
+number of unchanged lines must be the same on both sides.
+For example, if the old file has 10 lines and the hunks cover
+4 of them (`old_count` values summing to 4), then 6 old lines
+are unchanged.  The new file must also have exactly 6 lines
+not covered by hunks, so the `new_count` values must sum to
+`new_file_lines - 6`.
+
+-----------------------
+packet:          git< hunk 1 3 1 5
+packet:          git< hunk 10 2 12 2
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
+as changed and feeds them into the standard diff pipeline.  Patch
+output features (word diff, function context, color) work normally.
+Note that `--stat` and other summary formats use their own diff path
+and are not affected by the diff process.
+
+If no hunk lines precede the flush, followed by "success", Git
+treats the files as having no changes: `git diff` produces no output
+and `git blame` skips the commit, attributing lines to earlier commits.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool returns invalid hunks (out of bounds, overlapping, or
+mismatched unchanged line counts), Git warns and falls back to the
+builtin diff algorithm.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status.  Git warns and
+falls back to the builtin diff algorithm for this file.  The tool
+remains available for subsequent files.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=error
+packet:          git< 0000
+-----------------------
+
+In case the tool cannot or does not want to process the content as
+well as any future content for the lifetime of the Git process, it
+is expected to respond with an "abort" status.  Git silently falls
+back to the builtin diff algorithm for this file and does not send
+further requests to the tool.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=abort
+packet:          git< 0000
+-----------------------
+
+If the tool dies during the communication or does not adhere to the
+protocol then Git will stop the process and fall back to the builtin
+diff algorithm.  Git warns once and does not restart the process for
+subsequent files.
+
+Tools should ignore unknown keys in the per-file request to remain
+forward-compatible.  Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index 0976a69b4c..5de4718b24 100644
--- a/Makefile
+++ b/Makefile
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o
 TEST_BUILTINS_OBJS += test-date.o
 TEST_BUILTINS_OBJS += test-delete-gpgsig.o
 TEST_BUILTINS_OBJS += test-delta.o
+TEST_BUILTINS_OBJS += test-diff-process-backend.o
 TEST_BUILTINS_OBJS += test-dir-iterator.o
 TEST_BUILTINS_OBJS += test-drop-caches.o
 TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..fec63cf233
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,297 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  The tool controls which lines are marked as changed
+ * while the display shows the file content (after any textconv
+ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * When the tool returns no hunks with status=success, it considers
+ * the files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *get_or_launch_process(
+		struct userdiff_driver *drv)
+{
+	struct diff_subprocess *entry;
+
+	if (drv->diff_subprocess)
+		return drv->diff_subprocess;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start_command(&entry->subprocess, drv->process,
+				     start_diff_process_fn)) {
+		free(entry);
+		drv->diff_process_failed = 1;
+		return NULL;
+	}
+
+	drv->diff_subprocess = entry;
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret = 0;
+
+	if (size < 0)
+		return -1;
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/*
+	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
+	 * All numbers must be non-negative decimal with no leading
+	 * whitespace or sign characters.
+	 */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_count = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_count = strtol(line, &end, 10);
+	if (errno || end == line || *end != '\0')
+		return -1;
+
+	/*
+	 * git diff emits start=0 when count=0 (empty file side).
+	 * Normalize to 1-based so downstream validation can assume start >= 1.
+	 */
+	if (!hunk->old_count && !hunk->old_start)
+		hunk->old_start = 1;
+	if (!hunk->new_count && !hunk->new_start)
+		hunk->new_start = 1;
+
+	return 0;
+}
+
+static enum diff_process_result get_hunks(
+		struct userdiff_driver *drv,
+		const char *path,
+		const char *old_buf, long old_size,
+		const char *new_buf, long new_size,
+		struct xdl_hunk **hunks_out,
+		size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	backend = get_or_launch_process(drv);
+	if (!backend)
+		return DIFF_PROCESS_ERROR;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto comm_error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto comm_error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto comm_error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto comm_error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto comm_error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		*hunks_out = hunks;
+		*nr_hunks_out = nr_hunks;
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_OK;
+	}
+
+	if (!strcmp(status.buf, "abort")) {
+		/*
+		 * The tool voluntarily withdrew: stop sending requests
+		 * but do not warn (this is not a failure).
+		 */
+		backend->supported_capabilities &= ~CAP_HUNKS;
+		free(hunks);
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_SKIP;
+	}
+
+	/* status=error or unknown status */
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+
+comm_error:
+	/*
+	 * Communication failure (broken pipe, malformed response).
+	 * Tear down the process and mark as failed so we do not
+	 * retry on every subsequent file.
+	 */
+	drv->diff_process_failed = 1;
+	drv->diff_subprocess = NULL;
+	subprocess_stop_command(&backend->subprocess);
+	free(backend);
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+	struct xdl_hunk *ext_hunks = NULL;
+	size_t nr = 0;
+	enum diff_process_result res;
+
+	if (!diffopt || !path)
+		return DIFF_PROCESS_SKIP;
+	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+		return DIFF_PROCESS_SKIP;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return DIFF_PROCESS_SKIP;
+	if (drv->diff_process_failed)
+		return DIFF_PROCESS_SKIP;
+
+	res = get_hunks(drv, path,
+			file_a->ptr, file_a->size,
+			file_b->ptr, file_b->size,
+			&ext_hunks, &nr);
+	if (res == DIFF_PROCESS_OK) {
+		if (!nr) {
+			free(ext_hunks);
+			return DIFF_PROCESS_EQUIVALENT;
+		}
+		xpp->external_hunks = ext_hunks;
+		xpp->external_hunks_nr = nr;
+		return DIFF_PROCESS_OK;
+	}
+	if (res == DIFF_PROCESS_ERROR) {
+		warning(_("diff process '%s' failed for '%s',"
+			  " falling back to builtin diff"),
+			drv->process, path);
+		return DIFF_PROCESS_ERROR;
+	}
+	return DIFF_PROCESS_SKIP;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..d34b42f811
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,39 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+#include "xdiff/xdiff.h"
+
+struct diff_options;
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
+	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
+	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
+/*
+ * Consult the diff process configured for 'path' and populate
+ * xpp->external_hunks with the returned hunks.
+ *
+ * Handles driver lookup, flag checks (--no-ext-diff,
+ * --diff-algorithm), subprocess management, and error reporting.
+ *
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
+ * (files are considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
+ */
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 5a584fa1d5..3d97a188b9 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -4054,6 +4055,17 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (diff_process_fill_hunks(o, name_a,
+					    &mf1, &mf2, &xpp)
+		    == DIFF_PROCESS_EQUIVALENT) {
+			if (textconv_one)
+				free(mf1.ptr);
+			if (textconv_two)
+				free(mf2.ptr);
+			goto free_ab_and_return;
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4134,6 +4146,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(xpp.external_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/diff.h b/diff.h
index bb5cddaf34..7dc157968d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/** Disables diff.<driver>.process. */
+	unsigned no_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 3247697f74..aa532f5200 100644
--- a/meson.build
+++ b/meson.build
@@ -328,6 +328,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 3235f10ab8..6abcda4afb 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -12,6 +12,7 @@ test_tool_sources = [
   'test-date.c',
   'test-delete-gpgsig.c',
   'test-delta.c',
+  'test-diff-process-backend.c',
   'test-dir-iterator.c',
   'test-drop-caches.c',
   'test-dump-cache-tree.c',
diff --git a/t/helper/test-diff-process-backend.c b/t/helper/test-diff-process-backend.c
new file mode 100644
index 0000000000..ad392694e6
--- /dev/null
+++ b/t/helper/test-diff-process-backend.c
@@ -0,0 +1,299 @@
+/*
+ * Test backend for the long-running diff process protocol
+ * (see diff-process.c and Documentation/gitattributes.adoc).
+ *
+ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
+ *
+ * Implements the server side of the pkt-line handshake and a per-file
+ * response loop.  The --mode= switch selects the response shape
+ * (success, error, abort, crash, malformed hunks).
+ *
+ * Per-file request from Git:
+ *
+ *   packet:          git> command=hunks
+ *   packet:          git> pathname=<path>
+ *   packet:          git> 0000
+ *   packet:          git> OLD_CONTENT
+ *   packet:          git> 0000
+ *   packet:          git> NEW_CONTENT
+ *   packet:          git> 0000
+ *
+ * Response varies by --mode (default: whole-file):
+ *
+ *   whole-file   packet: git< hunk 1 <old_lines> 1 <new_lines>
+ *   fixed-hunk   packet: git< hunk 5 2 5 2
+ *   no-hunks     (no hunk packets)
+ *   bad-hunk     packet: git< hunk 999 1 999 1
+ *   bad-parse    packet: git< garbage not a hunk
+ *   bad-sync     packet: git< hunk 1 2 1 1
+ *   overlap      packet: git< hunk 1 5 1 5
+ *                packet: git< hunk 3 2 3 2
+ *   no-cap       (omits capability=hunks during handshake)
+ *   error        (status=error instead of status=success)
+ *   abort        (status=abort instead of status=success)
+ *   crash        exit(1) before sending any response
+ *
+ * All non-error/abort modes end with:
+ *
+ *   packet:          git< 0000
+ *   packet:          git< status=success
+ *   packet:          git< 0000
+ *
+ * Each request is logged to --log as:
+ *
+ *   command=<cmd> pathname=<path> old=<first line> new=<first line>
+ */
+
+#include "test-tool.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "strbuf.h"
+
+static FILE *logfile;
+
+enum mode {
+	MODE_WHOLE_FILE,
+	MODE_FIXED_HUNK,
+	MODE_NO_HUNKS,
+	MODE_BAD_HUNK,
+	MODE_BAD_PARSE,
+	MODE_BAD_SYNC,
+	MODE_OVERLAP,
+	MODE_NO_CAP,
+	MODE_ERROR,
+	MODE_ABORT,
+	MODE_CRASH,
+};
+
+static enum mode parse_mode(const char *s)
+{
+	if (!strcmp(s, "whole-file"))
+		return MODE_WHOLE_FILE;
+	if (!strcmp(s, "fixed-hunk"))
+		return MODE_FIXED_HUNK;
+	if (!strcmp(s, "no-hunks"))
+		return MODE_NO_HUNKS;
+	if (!strcmp(s, "bad-hunk"))
+		return MODE_BAD_HUNK;
+	if (!strcmp(s, "bad-parse"))
+		return MODE_BAD_PARSE;
+	if (!strcmp(s, "bad-sync"))
+		return MODE_BAD_SYNC;
+	if (!strcmp(s, "overlap"))
+		return MODE_OVERLAP;
+	if (!strcmp(s, "no-cap"))
+		return MODE_NO_CAP;
+	if (!strcmp(s, "error"))
+		return MODE_ERROR;
+	if (!strcmp(s, "abort"))
+		return MODE_ABORT;
+	if (!strcmp(s, "crash"))
+		return MODE_CRASH;
+	die("unknown --mode=%s", s);
+}
+
+/*
+ * Read "key=value" packets up to a flush, capturing "command" and
+ * "pathname".  Returns 1 if a request was read, 0 on EOF.
+ *
+ * The first packet uses the gentle variant so that a clean shutdown
+ * by Git (EOF) does not produce a spurious "the remote end hung up
+ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
+ * variant: once inside a request, truncation is a protocol violation
+ * and dying loudly is the correct response.
+ */
+static int read_request_header(char **command, char **pathname)
+{
+	int first = 1;
+	char *line;
+
+	*command = *pathname = NULL;
+	for (;;) {
+		const char *value;
+
+		if (first) {
+			if (packet_read_line_gently(0, NULL, &line) < 0)
+				return 0;
+			first = 0;
+		} else {
+			line = packet_read_line(0, NULL);
+		}
+		if (!line)
+			break;
+		if (skip_prefix(line, "command=", &value))
+			*command = xstrdup(value);
+		else if (skip_prefix(line, "pathname=", &value))
+			*pathname = xstrdup(value);
+	}
+	return 1;
+}
+
+static size_t count_lines(const struct strbuf *buf)
+{
+	size_t lines = 0;
+
+	for (size_t i = 0; i < buf->len; i++)
+		if (buf->buf[i] == '\n')
+			lines++;
+
+	return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
+}
+
+static void send_status(const char *status)
+{
+	packet_flush(1);
+	packet_write_fmt(1, "%s\n", status);
+	packet_flush(1);
+}
+
+static void respond(enum mode mode,
+		    const struct strbuf *old_buf,
+		    const struct strbuf *new_buf)
+{
+	switch (mode) {
+	case MODE_ERROR:
+		send_status("status=error");
+		return;
+	case MODE_ABORT:
+		send_status("status=abort");
+		return;
+	case MODE_CRASH:
+		exit(1);
+	case MODE_FIXED_HUNK:
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		break;
+	case MODE_BAD_HUNK:
+		packet_write_fmt(1, "hunk 999 1 999 1\n");
+		break;
+	case MODE_BAD_PARSE:
+		packet_write_fmt(1, "garbage not a hunk\n");
+		break;
+	case MODE_BAD_SYNC:
+		packet_write_fmt(1, "hunk 1 2 1 1\n");
+		break;
+	case MODE_OVERLAP:
+		packet_write_fmt(1, "hunk 1 5 1 5\n");
+		packet_write_fmt(1, "hunk 3 2 3 2\n");
+		break;
+	case MODE_NO_HUNKS:
+		break;
+	case MODE_NO_CAP:
+	case MODE_WHOLE_FILE: {
+		size_t old_lines = count_lines(old_buf);
+		size_t new_lines = count_lines(new_buf);
+		/*
+		 * Match git diff output: start=0 when count=0
+		 * (empty file side), 1 otherwise.
+		 */
+		packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
+				 " %"PRIuMAX" %"PRIuMAX"\n",
+				 (uintmax_t)(old_lines ? 1 : 0),
+				 (uintmax_t)old_lines,
+				 (uintmax_t)(new_lines ? 1 : 0),
+				 (uintmax_t)new_lines);
+		break;
+	}
+	}
+	send_status("status=success");
+}
+
+static void command_loop(enum mode mode)
+{
+	for (;;) {
+		char *command = NULL, *pathname = NULL;
+		struct strbuf obuf = STRBUF_INIT;
+		struct strbuf nbuf = STRBUF_INIT;
+
+		if (!read_request_header(&command, &pathname))
+			break; /* EOF: Git closed its end */
+
+		read_packetized_to_strbuf(0, &obuf, 0);
+		read_packetized_to_strbuf(0, &nbuf, 0);
+
+		if (logfile) {
+			fprintf(logfile,
+				"command=%s pathname=%s old=%.*s new=%.*s\n",
+				command ? command : "(none)",
+				pathname ? pathname : "(none)",
+				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
+				obuf.buf,
+				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
+				nbuf.buf);
+			fflush(logfile);
+		}
+
+		respond(mode, &obuf, &nbuf);
+
+		free(command);
+		free(pathname);
+		strbuf_release(&obuf);
+		strbuf_release(&nbuf);
+	}
+}
+
+static void handshake(enum mode mode)
+{
+	char *line;
+
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "git-diff-client"))
+		die("bad welcome: '%s'", line ? line : "(eof)");
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "version=1"))
+		die("bad version: '%s'", line ? line : "(eof)");
+	if (packet_read_line(0, NULL))
+		die("expected flush after version");
+
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Drain capabilities advertised by Git */
+	while ((line = packet_read_line(0, NULL)))
+		; /* drain */
+
+	/* Respond with our capabilities (or none for no-cap mode) */
+	if (mode != MODE_NO_CAP)
+		packet_write_fmt(1, "capability=hunks\n");
+	packet_flush(1);
+}
+
+static const char *const usage_str[] = {
+	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
+	NULL
+};
+
+int cmd__diff_process_backend(int argc, const char **argv)
+{
+	const char *mode_str = NULL, *log_path = NULL;
+	enum mode mode = MODE_WHOLE_FILE;
+	struct option options[] = {
+		OPT_STRING(0, "mode", &mode_str, "mode",
+			   "response shape: whole-file (default), fixed-hunk,"
+			   " no-hunks, bad-hunk, bad-sync, overlap, error,"
+			   " abort, crash"),
+		OPT_STRING(0, "log", &log_path, "path",
+			   "append per-request summary to this file"),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
+	if (argc)
+		usage_with_options(usage_str, options);
+
+	if (mode_str)
+		mode = parse_mode(mode_str);
+
+	if (log_path) {
+		logfile = fopen(log_path, "a");
+		if (!logfile)
+			die_errno("failed to open log '%s'", log_path);
+	}
+
+	handshake(mode);
+	command_loop(mode);
+
+	if (logfile && fclose(logfile))
+		die_errno("error closing log");
+	return 0;
+}
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index b71a22b43b..3c3f95269c 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = {
 	{ "date", cmd__date },
 	{ "delete-gpgsig", cmd__delete_gpgsig },
 	{ "delta", cmd__delta },
+	{ "diff-process-backend", cmd__diff_process_backend },
 	{ "dir-iterator", cmd__dir_iterator },
 	{ "drop-caches", cmd__drop_caches },
 	{ "dump-cache-tree", cmd__dump_cache_tree },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f2885b33d5..a5bb755516 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);
 int cmd__date(int argc, const char **argv);
 int cmd__delta(int argc, const char **argv);
 int cmd__delete_gpgsig(int argc, const char **argv);
+int cmd__diff_process_backend(int argc, const char **argv);
 int cmd__dir_iterator(int argc, const char **argv);
 int cmd__drop_caches(int argc, const char **argv);
 int cmd__dump_cache_tree(int argc, const char **argv);
diff --git a/t/meson.build b/t/meson.build
index c5832fee05..027855ced7 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -512,6 +512,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..9bb579b564
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,432 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+# See t/helper/test-diff-process-backend.c for the backend implementation
+# and available --mode= options.
+
+BACKEND="test-tool diff-process-backend"
+
+test_expect_success 'setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# boundary.c: 10 lines, changes at 5-6 and 9-10.
+	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+
+	# worddiff.c: single-line function, value changes 1 -> 999.
+	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+
+	# newfile.c: single-line function, value changes 42 -> 99.
+	# Used by: modified file, --exit-code, multiple drivers.
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 42; }
+	EOF
+	git add newfile.c &&
+
+	# logtest.c: single-line function for log/format-patch tests.
+	# Needs two commits so log -1 has a diff.
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+
+	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
+	cat >one.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >two.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add one.c two.c &&
+
+	git commit -m "initial" &&
+
+	# Second commit for logtest.c (so log -1 has something to show).
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	# Working tree modifications (not committed).
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 99; }
+	EOF
+
+	cat >one.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+
+	cat >two.c <<-\EOF
+	int second(void) { return 20; }
+	EOF
+'
+
+#
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
+test_expect_success 'diff process hunk boundaries affect output' '
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	test_grep ! "^-OLD9" actual &&
+	test_grep ! "^-OLD10" actual &&
+	test_grep ! "^+NEW9" actual &&
+	test_grep ! "^+NEW10" actual
+'
+
+test_expect_success 'diff process works with modified file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- newfile.c >actual 2>stderr &&
+	test_grep "return 99" actual &&
+	test_grep "pathname=newfile.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with added file (empty old side)' '
+	cat >added.c <<-\EOF &&
+	int added(void) { return 1; }
+	EOF
+	git add added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "added" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with deleted file (empty new side)' '
+	git add added.c &&
+	git commit -m "commit added.c" &&
+	git rm added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "deleted file" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process skipped for binary files' '
+	printf "\\0binary" >binary.c &&
+	git add binary.c &&
+	git commit -m "add binary" &&
+	printf "\\0changed" >binary.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- binary.c >actual &&
+	test_grep "Binary files" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not consulted for unmatched driver' '
+	echo "not tracked by cdiff" >unmatched.txt &&
+	git add unmatched.txt &&
+	git commit -m "add unmatched.txt" &&
+
+	echo "modified" >unmatched.txt &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- unmatched.txt >actual &&
+	test_grep "modified" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'multiple drivers use separate processes' '
+	echo "*.h diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 1; }
+	EOF
+	git add multi.h &&
+	git commit -m "add multi.h" &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 2; }
+	EOF
+
+	test_when_finished "rm -f backend-c.log backend-h.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+		diff -- newfile.c multi.h >actual 2>stderr &&
+	test_grep "pathname=newfile.c" backend-c.log &&
+	test_grep "pathname=multi.h" backend-h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works alongside textconv' '
+	write_script uppercase-filter <<-\EOF &&
+	tr "a-z" "A-Z" <"$1"
+	EOF
+
+	cat >textconv.c <<-\EOF &&
+	hello world
+	EOF
+	git add textconv.c &&
+	git commit -m "add textconv.c" &&
+
+	cat >textconv.c <<-\EOF &&
+	goodbye world
+	EOF
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- textconv.c >actual 2>stderr &&
+	# The diff process receives textconv-transformed (uppercase) content.
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=HELLO WORLD" backend.log &&
+	test_grep "new=GOODBYE WORLD" backend.log &&
+	test_must_be_empty stderr
+'
+
+#
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
+test_expect_success 'diff process with --word-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --word-diff worddiff.c >actual 2>stderr &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual &&
+	test_grep "pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with git log -p' '
+	# With no-hunks mode, the tool says the files are equivalent,
+	# so log -p should show the commit but no diff content.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -1 -p -- logtest.c >actual 2>stderr &&
+	test_grep "change logtest.c" actual &&
+	test_grep ! "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process no hunks suppresses diff output' '
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add nohunks.c &&
+	git commit -m "add nohunks.c" &&
+
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff nohunks.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success 'diff process no hunks with --exit-code returns success' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code nohunks.c
+'
+
+test_expect_success 'diff process with --exit-code and hunks returns failure' '
+	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+		diff --exit-code newfile.c
+'
+
+#
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
+test_expect_success 'diff process bypassed by --diff-algorithm' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not used by --stat' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --stat worddiff.c >actual &&
+	test_grep "worddiff.c" actual &&
+	test_path_is_missing backend.log
+'
+
+#
+# Error handling and fallback.
+#
+
+test_expect_success 'diff process fallback on tool error status' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process error keeps tool available for next file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Unlike abort, error keeps the tool available: both files
+	# are sent to the tool (and both fall back).
+	test_grep "pathname=one.c" backend.log &&
+	test_grep "pathname=two.c" backend.log &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process abort disables for session' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool.
+	test_grep "pathname=one.c" backend.log &&
+	test_grep ! "pathname=two.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Crash is a communication failure, so a warning is emitted.
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process startup failure only warns once' '
+	git -c diff.cdiff.process="/nonexistent/tool" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# Sentinel prevents repeated warnings: only one, not one per file.
+	test_grep "diff process.*failed" stderr >warnings &&
+	test_line_count = 1 warnings
+'
+
+
+test_expect_success 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "exceeds.*lines" stderr
+'
+
+test_expect_success 'diff process fallback on mismatched unchanged totals' '
+	cat >synctest.c <<-\EOF &&
+	line1
+	line2
+	line3
+	EOF
+	git add synctest.c &&
+	git commit -m "add synctest.c" &&
+
+	cat >synctest.c <<-\EOF &&
+	line1
+	changed
+	line3
+	EOF
+
+	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
+	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
+	# The synchronization invariant fails and git falls back.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+		diff synctest.c >actual 2>stderr &&
+	test_grep "changed" actual &&
+	test_grep "unchanged line count mismatch" stderr
+'
+
+test_expect_success 'diff process fallback on overlapping hunks' '
+	# boundary.c has 10 lines, so both hunks are in bounds
+	# but they overlap at lines 3-5, triggering the ordering check.
+	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "NEW5" actual &&
+	test_grep "overlaps with previous" stderr
+'
+
+test_expect_success 'diff process fallback on malformed hunk line' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual
+'
+
+test_expect_success 'diff process skipped when tool omits capability' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_must_be_empty stderr
+'
+
+test_done
diff --git a/userdiff.h b/userdiff.h
index 51c26e0d41..a98eabe377 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -3,6 +3,7 @@
 
 #include "notes-cache.h"
 
+struct diff_subprocess;
 struct index_state;
 struct repository;
 
@@ -33,6 +34,8 @@ struct userdiff_driver {
 	int textconv_want_cache;
 	const char *process;
 	char *process_owned;
+	struct diff_subprocess *diff_subprocess;
+	unsigned diff_process_failed : 1;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v4 5/6] diff: bypass diff process with --no-ext-diff and in format-patch

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Make --no-ext-diff disable diff.<driver>.process in addition to
diff.<driver>.command.  Although the two mechanisms work differently
(command replaces Git's output, process feeds hunks back into the
pipeline), both invoke external tools and --no-ext-diff means
"no external tools."

Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that
sets both allow_external and no_diff_process, so a single option
controls both.  Passing --ext-diff explicitly clears
no_diff_process, so a later --ext-diff overrides an earlier
--no-ext-diff.

Disable the diff process unconditionally in format-patch so that
generated patches are always based on the builtin diff algorithm
and can be applied reliably by recipients who do not have the
external tool.

Document that --diff-algorithm also bypasses the diff process,
since it forces the builtin algorithm.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/diff-algorithm-option.adoc |  3 +++
 Documentation/diff-options.adoc          |  4 +++-
 builtin/log.c                            |  7 +++++++
 diff.c                                   | 16 ++++++++++++++--
 diff.h                                   |  4 +++-
 t/t4080-diff-process.sh                  | 16 ++++++++++++++++
 6 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..4d7e2ec35f 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index c8242e2462..a884445211 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -833,7 +833,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::
diff --git a/builtin/log.c b/builtin/log.c
index e464b30af4..363052f468 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * Disable diff.<driver>.process so that patches generated by
+	 * format-patch are always based on the builtin diff algorithm
+	 * and can be applied reliably.
+	 */
+	rev.diffopt.flags.no_diff_process = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff.c b/diff.c
index 3d97a188b9..4d9cb9b26b 100644
--- a/diff.c
+++ b/diff.c
@@ -5936,6 +5936,17 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.no_diff_process = unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6266,8 +6277,9 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index 7dc157968d..bc7da6986a 100644
--- a/diff.h
+++ b/diff.h
@@ -173,7 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
-	/** Disables diff.<driver>.process. */
+	/**
+	 * Disables diff.<driver>.process.  Set by --no-ext-diff.
+	 */
 	unsigned no_diff_process;
 
 	/**
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 9bb579b564..df4d08e31f 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -295,6 +295,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' '
 	test_path_is_missing backend.log
 '
 
+test_expect_success 'diff process bypassed by --no-ext-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --no-ext-diff worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not used by format-patch' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		format-patch -1 --stdout -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_path_is_missing backend.log
+'
+
 test_expect_success 'diff process not used by --stat' '
 	test_when_finished "rm -f backend.log" &&
 	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
-- 
gitgitgadget

[PATCH v4 6/6] blame: consult diff process for no-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The consultation happens at the pass_blame_to_parent() callsite
using diff_process_fill_hunks(), matching how builtin_diff() in
diff.c uses the same function.  A new diff_hunks_xpp() variant
accepts a pre-populated xpparam_t so callers can pass external
hunks, while the existing diff_hunks() retains its original
signature and behavior.  The copy-detection callsite is
unaffected since it does not use the diff process.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                 |  40 +++++++++++----
 t/t4080-diff-process.sh | 105 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 136 insertions(+), 9 deletions(-)

diff --git a/blame.c b/blame.c
index 977cbb7097..354e6c15f4 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "xdiff-interface.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -314,17 +316,25 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 
-static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
+			  xdl_emit_hunk_consume_func_t hunk_func,
+			  void *cb_data, xpparam_t *xpp)
 {
-	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
 
-	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
+}
+
+static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+{
+	xpparam_t xpp = {0};
+
+	xpp.flags = xdl_opts;
+	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1943,6 +1953,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *parent, int ignore_diffs)
 {
 	mmfile_t file_p, file_o;
+	xpparam_t xpp = {0};
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
 
@@ -1961,10 +1972,21 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
-		die("unable to generate diff (%s -> %s)",
-		    oid_to_hex(&parent->commit->object.oid),
-		    oid_to_hex(&target->commit->object.oid));
+	xpp.flags = sb->xdl_opts;
+	/*
+	 * If the diff process considers the files equivalent,
+	 * skip the diff so blame looks past this commit.
+	 */
+	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
+				    &file_p, &file_o, &xpp)
+	    != DIFF_PROCESS_EQUIVALENT) {
+		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
+				   &d, &xpp))
+			die("unable to generate diff (%s -> %s)",
+			    oid_to_hex(&parent->commit->object.oid),
+			    oid_to_hex(&target->commit->object.oid));
+	}
+	free(xpp.external_hunks);
 	/* The rest are the same as the parent */
 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
 		    parent, target, 0);
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index df4d08e31f..9fc3c01eec 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -445,4 +445,109 @@ test_expect_success 'diff process skipped when tool omits capability' '
 	test_must_be_empty stderr
 '
 
+#
+# Blame integration.
+#
+
+test_expect_success 'blame uses tool-provided hunks' '
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "add blame-hunk.c" &&
+	ORIG=$(git rev-parse --short HEAD) &&
+
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "change blame-hunk.c" &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+
+	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
+	# so blame should attribute lines 9-10 to the original commit
+	# even though the builtin diff would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		blame blame-hunk.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	sed -n "6p" actual >line6 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "$CHANGE" line6
+'
+
+test_expect_success 'blame skips commits with no hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void) {
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without no-hunks mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With no-hunks mode, the process considers the files equivalent
+	# and blame skips the reformat commit, attributing to the original.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		blame blame.c >with &&
+	test_grep ! "$BLAME_COMMIT" with &&
+	test_grep "$ORIG_COMMIT" with
+'
+
+test_expect_success 'blame --no-ext-diff bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		blame --no-ext-diff blame.c >actual &&
+	# Without the process, blame attributes the reformat commit normally.
+	test_grep "$BLAME_COMMIT" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame --no-ext-diff uses builtin hunks' '
+	# fixed-hunk mode would narrow blame to lines 5-6, but
+	# --no-ext-diff should bypass it and use the builtin diff.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		blame --no-ext-diff blame-hunk.c >actual &&
+	# Builtin diff attributes lines 9-10 to the change commit.
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
 test_done
-- 
gitgitgadget

Re: [PREVIEW v4 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo]

On Sun, Jun 14, 2026 at 11:55 AM Michael Montalbo via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> This series adds diff.<driver>.process, a long-running subprocess protocol
> that lets an external tool control which lines Git considers changed while
> Git handles all output formatting...

Now I'm realizing there are some diff flags that are not properly supported
yet with an external diff provider. Flags like -L and the stat-related
flags should probably behave like blame and consult the external diff
process. In general, there should be a more explicit and comprehensive
explanation for which flags interact with external diff processes and
which do not, included principled reasons for why that is the case.

[PATCH v5 0/9] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's diff output with the program's own
output, so display features like word diff, function context, and color
cannot operate on it; and because the program is consulted only for that
patch output, blame, --stat, and git log -L fall back to Git's builtin line
matching and cannot benefit from the tool at all.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes, which propagates through
patch output, the --stat summary, blame, and git log -L. The request also
carries the two blobs' object names (old-oid/new-oid) so a tool can cache
its analysis keyed on the pair.

 * Patch 1: document how an external diff driver (diff.<driver>.command)
   relates to the rest of Git's diff features, so the contrast with the new
   process driver is clear.
 * Patch 2: xdiff plumbing for externally supplied hunks.
 * Patch 3: diff.<driver>.process config key.
 * Patch 4: refactor subprocess API to separate process lifecycle from
   hashmap management, since the diff process stores its subprocess on the
   userdiff driver rather than in a hashmap.
 * Patch 5: the main feature, including the old-oid/new-oid request metadata
   for blob-pair caching.
 * Patch 6: bypass knobs (--no-ext-diff, format-patch).
 * Patch 7: blame integration so the tool can declare commits as having no
   changes; introduces the shared xdi_diff_process() consult-then-diff
   helper that blame and git log -L both use.
 * Patch 8: --stat/--numstat/--shortstat consult the tool, so the summary
   agrees with the patch output.
 * Patch 9: git log -L range tracking consults the tool, so a reformat-only
   commit is dropped from the log rather than shown with an empty diff.

A "Which features consult the diff process" section in gitattributes(5) lays
out, per feature, why each does or does not consult the process (patch
output, blame, summary formats, and the -L line-range view do; pickaxe -G,
patch-id, merge, range-diff, --check, and --raw do not, with reasons).
Combined diffs (--cc) remain on the builtin algorithm and are noted as
future work.

Changes since v4:

 * New preparatory doc patch (patch 1): document how an external diff driver
   (diff.<driver>.command) relates to the rest of Git's diff features, so
   the contrast with the process driver added later in the series is
   explicit.

 * New: --stat/--numstat/--shortstat now consult the process (patch 8). A
   file the tool reports as equivalent contributes no stat line, matching
   the empty patch git diff produces for it. Summary formats do not apply
   textconv, so the tool is fed raw content there, as the builtin --stat
   already counts raw lines.

 * New: git log -L range tracking now consults the process (patch 9).
   Previously the tracking pass used the builtin diff while the displayed
   diff used the tool, so a reformat-only commit could be selected by
   tracking and then rendered with an empty diff. Tracking now agrees with
   the display and the commit is dropped.

 * New: the request carries the old and new blob object names
   (old-oid/new-oid), so a tool can cache its line matching keyed on the
   blob pair. A side's oid is sent only when the tool receives that raw
   blob; it is omitted under textconv (which rewrites the bytes) and for a
   working-tree side with no stored object. This is where the process
   differs from diff.<name>.command, which never composes with textconv and
   so always feeds the raw blob its oid names.

 * Refactor: blame and git log -L now consult the process through a single
   xdi_diff_process() helper instead of open-coding the consult-then-diff
   dance; builtin_diff() keeps its own path so it can short-circuit
   equivalent files before its funcname/word-diff setup. The whitespace
   bypass keys off the effective diff parameters (xpp), which removes
   blame's separate -w guard, and blame's diff_hunks() sheds an intermediate
   variant it no longer needs.

 * Correctness: external-hunk validation now checks per-gap alignment, not
   just the total unchanged-line count. xdl_build_script() walks the two
   files in lockstep over unchanged lines, so a hunk set whose totals
   balance but whose per-gap line counts do not (e.g. hunk 1 1 3 1)
   previously passed validation and produced a corrupt diff, --stat, and
   blame attribution. Such responses are now rejected and Git falls back to
   the builtin diff.

 * Robustness: hunk accumulation is bounded by the two files' combined line
   count, so a misbehaving tool that floods hunk lines cannot grow memory
   without bound before validation.

 * Forward-compat: the hunk-line parser now ignores unknown trailing fields
   after the four counts, so a future protocol version can append a field
   without older clients rejecting it.

 * Feature interactions: the whitespace-ignoring options (-w,
   --ignore-space-change, --ignore-blank-lines, ...), -I, and --anchored
   bypass the process (the tool is never told about them and could not honor
   them), and git blame -w does the same. A change that only adds or removes
   the trailing newline cannot be expressed as line hunks, so it also falls
   back to the builtin diff (preserving the "\ No newline at end of file"
   marker).

 * Documentation: added the per-feature "Which features consult the diff
   process" section; documented that the process trusts the tool's notion of
   "unchanged" (it is not byte-validated, so like git diff -w such a patch
   may not apply against the old blob), that --exit-code/--quiet report
   success for tool- equivalent files, and that diff.<name>.command takes
   precedence when both it and .process are configured.

 * Tests: t4080 grew coverage for the per-gap check, the hunk- flood cap,
   the whitespace/-w bypasses, the trailing-newline and added/deleted-file
   fallbacks, multi-hunk output through patch and --stat, --stat
   --exit-code, a mode-only change, and a mixed equivalent/changed
   multi-file diffstat.

Michael Montalbo (9):
  gitattributes: document how external diff drivers relate to diff
    features
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection
  diff: consult diff process for --stat counts
  line-log: consult diff process for range tracking

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 274 +++++++
 Makefile                                 |   2 +
 blame.c                                  |  24 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 529 +++++++++++++
 diff-process.h                           |  75 ++
 diff.c                                   |  57 +-
 diff.h                                   |   6 +
 line-log.c                               |  33 +-
 meson.build                              |   1 +
 sub-process.c                            |  28 +-
 sub-process.h                            |   9 +-
 t/helper/meson.build                     |   1 +
 t/helper/test-diff-process-backend.c     | 381 +++++++++
 t/helper/test-tool.c                     |   1 +
 t/helper/test-tool.h                     |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 950 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  16 +
 xdiff/xdiffi.c                           |  84 +-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 28 files changed, 2504 insertions(+), 18 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh


base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v4:

  -:  ---------- >  1:  0fd994a3d3 gitattributes: document how external diff drivers relate to diff features
  1:  03f261dfe2 !  2:  2004502549 xdiff: support external hunks via xpparam_t
     @@ Commit message
          the diff algorithm, then continues through compaction and emission
          as usual.
      
     -    Validate supplied hunks before use: reject out-of-bounds line
     -    numbers, overlapping or out-of-order hunks, negative counts, and
     -    violations of the synchronization invariant (unchanged line counts
     -    must match between files).  On validation failure, fall back to
     -    the builtin diff algorithm; this re-runs xdl_prepare_env() since
     -    the first call may have dirtied the changed[] arrays.
     +    Validate supplied hunks before use.  Out-of-bounds line numbers,
     +    overlapping or out-of-order hunks, and misaligned unchanged runs are
     +    treated as a malformed tool response: xdl_populate_hunks_from_external()
     +    warns, returns -1, and xdl_diff() falls back to the builtin diff
     +    algorithm for that file.  The run of unchanged lines between two hunks
     +    (and before the first and after the last) must be the same length on
     +    both sides; xdl_build_script() walks the two files in lockstep over
     +    unchanged lines, so a balanced total is not enough.  Non-negative
     +    counts and 1-based starts are instead caller preconditions, checked
     +    with BUG(), since the caller normalizes hunks before this point.
     +
     +    On rejection xdl_diff() frees the environment it prepared and falls
     +    through to xdl_do_diff(), which prepares a fresh one for the builtin
     +    pass.
      
          Skip trim_common_tail() in xdi_diff() when external hunks are
          present, since external hunks reference line numbers in the
     @@ xdiff/xdiff.h: typedef struct s_mmbuffer {
       } mmbuffer_t;
       
      +/*
     -+ * Hunk descriptor for externally computed diffs.
     -+ * Line numbers are 1-based; a start of 0 is accepted when
     -+ * count is 0 (empty file side, matching git diff output).
     ++ * Hunk descriptor for externally computed diffs, in xdiff's own
     ++ * coordinates: line numbers are 1-based and a hunk's start is the
     ++ * first line it covers.  A caller translates any external "empty side"
     ++ * idiom (such as git diff's start-0/count-0) to a 1-based start before
     ++ * handing hunks over.
      + */
      +struct xdl_hunk {
      +	long old_start, old_count;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
       
      +/*
      + * Populate the changed[] arrays from externally supplied hunks,
     -+ * bypassing the diff algorithm.  Validates that hunks are in order,
     -+ * non-overlapping, and within bounds.
     ++ * bypassing the diff algorithm.  The caller normalizes and validates
     ++ * the hunks first (order, overlap, and lockstep alignment), so this
     ++ * only marks lines changed after asserting the memory-safety
     ++ * preconditions it depends on: non-negative counts and 1-based starts
     ++ * (checked with BUG()), and an in-bounds range (a silent -1 so the
     ++ * caller can fall back to the builtin diff rather than index changed[]
     ++ * out of range).  Keeping this diagnostic-free leaves user-facing
     ++ * messages to the git layer.
      + *
     -+ * Returns 0 on success, -1 on validation failure.
     ++ * Returns 0 on success, -1 if a hunk is out of range.
      + */
      +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
      +					    struct xdl_hunk *hunks,
      +					    size_t nr_hunks)
      +{
      +	size_t i;
     -+	long j, prev_old_end = 0, prev_new_end = 0;
     -+	long changed_old = 0, changed_new = 0;
     ++	long j;
      +
      +	/*
      +	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +	for (i = 0; i < nr_hunks; i++) {
      +		struct xdl_hunk *h = &hunks[i];
      +
     -+		if (h->old_count < 0 || h->new_count < 0) {
     -+			warning("diff process hunk %"PRIuMAX": "
     ++		/*
     ++		 * Non-negative counts and 1-based starts are caller
     ++		 * preconditions (it normalizes hunks into xdiff coordinates
     ++		 * before this point), so a violation is a bug, not a bad
     ++		 * tool response.
     ++		 */
     ++		if (h->old_count < 0 || h->new_count < 0)
     ++			BUG("external hunk %"PRIuMAX": "
      +				"negative count (old=%ld, new=%ld)",
      +				(uintmax_t)(i + 1),
      +				h->old_count, h->new_count);
     -+			return -1;
     -+		}
     -+		if (h->old_start < 1 || h->new_start < 1) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"start must be >= 1 (old=%ld, new=%ld)",
     ++		if (h->old_start < 1 || h->new_start < 1)
     ++			BUG("external hunk %"PRIuMAX": "
     ++				"start not 1-based (old=%ld, new=%ld)",
      +				(uintmax_t)(i + 1),
      +				h->old_start, h->new_start);
     -+			return -1;
     -+		}
      +
      +		/*
     -+		 * Range must fit: start + count - 1 <= nrec,
     -+		 * rewritten to avoid overflow.  Same for both sides.
     -+		 *
     -+		 * When count is 0 (pure insert/delete) the check
     -+		 * reduces to 0 > nrec - start + 1, which rejects
     -+		 * start > nrec + 1 and allows start == nrec + 1
     -+		 * (the position after the last line).
     ++		 * The caller validates ordering, overlap and lockstep
     ++		 * alignment (and diagnoses a bad response).  This is only a
     ++		 * silent in-bounds guard so the marking loop cannot index
     ++		 * changed[] out of range: start + count - 1 <= nrec,
     ++		 * rewritten to avoid overflow.  A count of 0 (pure
     ++		 * insert/delete) allows start == nrec + 1, the position
     ++		 * after the last line.  On a miss, return -1 and let the
     ++		 * caller fall back to the builtin diff.
      +		 */
     -+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"old range %ld+%ld exceeds %lu lines",
     -+				(uintmax_t)(i + 1),
     -+				h->old_start, h->old_count,
     -+				(unsigned long)xe->xdf1.nrec);
     -+			return -1;
     -+		}
     -+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"new range %ld+%ld exceeds %lu lines",
     -+				(uintmax_t)(i + 1),
     -+				h->new_start, h->new_count,
     -+				(unsigned long)xe->xdf2.nrec);
     ++		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1 ||
     ++		    h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
      +			return -1;
     -+		}
     -+
     -+		/* Ordering: no overlap with previous hunk (adjacent is OK) */
     -+		if (h->old_start < prev_old_end ||
     -+		    h->new_start < prev_new_end) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"overlaps with previous hunk",
     -+				(uintmax_t)(i + 1));
     -+			return -1;
     -+		}
      +
      +		for (j = 0; j < h->old_count; j++)
      +			xe->xdf1.changed[h->old_start - 1 + j] = true;
      +		for (j = 0; j < h->new_count; j++)
      +			xe->xdf2.changed[h->new_start - 1 + j] = true;
     -+
     -+		prev_old_end = h->old_start + h->old_count;
     -+		prev_new_end = h->new_start + h->new_count;
     -+	}
     -+
     -+	/*
     -+	 * Synchronization invariant: unchanged line counts must match.
     -+	 * Otherwise xdl_build_script() would walk off one array.
     -+	 *
     -+	 * Count changed lines from the arrays rather than accumulating
     -+	 * during the loop to avoid any overflow in the summation.
     -+	 */
     -+	for (j = 0; j < (long)xe->xdf1.nrec; j++)
     -+		if (xe->xdf1.changed[j])
     -+			changed_old++;
     -+	for (j = 0; j < (long)xe->xdf2.nrec; j++)
     -+		if (xe->xdf2.changed[j])
     -+			changed_new++;
     -+	if ((long)xe->xdf1.nrec - changed_old !=
     -+	    (long)xe->xdf2.nrec - changed_new) {
     -+		warning("diff process: unchanged line count mismatch "
     -+			"(old: %ld unchanged, new: %ld unchanged)",
     -+			(long)xe->xdf1.nrec - changed_old,
     -+			(long)xe->xdf2.nrec - changed_new);
     -+		return -1;
      +	}
      +
      +	return 0;
  2:  30617ee17b =  3:  926cf01af6 userdiff: add diff.<driver>.process config
  3:  459e485e6d =  4:  363d459ff6 sub-process: separate process lifecycle from hashmap management
  4:  10b3980f59 !  5:  d003bc1f15 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
          textconv-transformed content.  The tool controls which lines
          are marked as changed while the display shows the file content.
          Patch output features (word diff, function context, color) work
     -    normally; --stat uses its own diff codepath and never consults
     -    the diff process.
     +    normally.  A new "Which features consult the diff process"
     +    documentation section lays out which features use the tool's hunks,
     +    which compute independently, and why; the summary formats such as
     +    --stat still use the builtin diff for now.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
     -    requests send command=hunks, pathname, and both file contents as
     -    packetized data.  The tool responds with hunk lines and a status
     -    packet (success, error, or abort).  On error, Git warns and falls
     -    back to the builtin diff algorithm for that file.  On abort, Git
     -    silently falls back for the current file and stops sending further
     -    requests to the tool for the remainder of the session.
     +    requests send command=hunks, pathname, the old and new blob object
     +    names as old-oid/new-oid, and both file contents as packetized data.
     +    The tool responds with hunk lines and a status packet (success,
     +    error, or abort).  On error, Git warns and falls back to the builtin
     +    diff algorithm for that file.  On abort, Git silently falls back for
     +    the current file and stops sending further requests to the tool for
     +    the remainder of the session.
     +
     +    old-oid/new-oid name the two blobs so a tool can cache its analysis
     +    keyed on the pair.  A side's oid is sent only when the content the
     +    tool receives is that raw blob: it is omitted under textconv, which
     +    rewrites the bytes, and for a working-tree side with no stored
     +    object, so an oid that is sent always names the bytes the tool
     +    receives.  This is where the process protocol diverges from
     +    diff.<driver>.command, which never composes with textconv (the
     +    command replaces the whole diff and always gets the raw blob).  Tools
     +    ignore unknown request keys, so old tools skip them.
      
          When the tool returns no hunks followed by status=success, Git
          treats the file as having no changes and produces no diff output.
     @@ Commit message
          the failure is cached on the driver to avoid retrying and
          re-warning on every subsequent file.
      
     +    Git falls back to the builtin diff (rather than consulting the
     +    tool) when an option the tool cannot honor is in effect: the
     +    whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and
     +    --anchored.  The bypass keys off the effective diff parameters (xpp)
     +    rather than diffopt, so a later caller whose flags live elsewhere is
     +    covered uniformly.  A change that only adds or removes the trailing
     +    newline is likewise not expressible as hunks, so it too uses the
     +    builtin diff.  The hunk parser ignores unknown trailing fields on a
     +    hunk line for response forward-compatibility.
     +
     +    Hunk accumulation is bounded by the combined byte count of the two
     +    files, so a misbehaving tool that floods hunk lines cannot grow
     +    memory without bound before validation runs.
     +
          diff_process_fill_hunks() is the sole public entry point.  It
          handles driver lookup, flag checks, subprocess management, and
          error reporting, returning an enum that lets callers distinguish
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +long-running process protocol (described in
      +Documentation/technical/long-running-process-protocol.adoc).
      +Unlike `diff.<name>.command`, which replaces Git's output entirely,
     -+the diff process feeds results back into the standard pipeline.
     ++the diff process feeds results back into the standard pipeline.  If
     ++both are configured for a path, `diff.<name>.command` takes precedence
     ++for the patch output it replaces; the summary formats, `git blame`,
     ++and `git log -L` never run the command and still consult the process.
      +
      +First, in `.gitattributes`, assign the `diff` attribute for paths.
      +
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +the process and performs the handshake.  In the handshake, the welcome
      +message sent by Git is "git-diff-client", only version 1 is supported,
      +and the supported capability is "hunks" (the changed regions
     -+described below).
     ++described below).  The tool replies with "git-diff-server", the
     ++version it supports, and the capabilities it supports.
      +
      +For each file, Git sends a list of "key=value" pairs terminated with
      +a flush packet, followed by the old and new file content as packetized
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +-----------------------
      +packet:          git> command=hunks
      +packet:          git> pathname=path/file.c
     ++packet:          git> old-oid=<hex>
     ++packet:          git> new-oid=<hex>
      +packet:          git> 0000
      +packet:          git> OLD_CONTENT
      +packet:          git> 0000
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +packet:          git> 0000
      +-----------------------
      +
     ++The optional `old-oid` and `new-oid` keys give the object names of the
     ++old and new blobs, so a tool can cache its analysis keyed on the pair.
     ++A side's key is sent only when the content for that side is the raw
     ++blob it names: it is omitted when the content is textconv-transformed,
     ++and for a working-tree side that has no stored object.  A tool that
     ++does not recognize these keys ignores them.
     ++
      +The tool is expected to respond with zero or more hunk lines,
      +a flush packet, and a status packet terminated with a flush packet.
      +Each hunk line has the form:
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +
      +where `<old_start>` and `<old_count>` identify a range of lines in
      +the old file, and `<new_start>` and `<new_count>` identify the
     -+replacement range in the new file.  Start values are 1-based and
     -+counts are non-negative.  Ranges must not extend beyond the end of
     -+the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
     -+at line 3 in the old file were replaced by 4 lines starting at
     -+line 3 in the new file.  An `<old_count>` of 0 means no lines were
     -+removed (pure insertion); a `<new_count>` of 0 means no lines were
     -+added (pure deletion).  A start value of 0 is accepted when
     -+the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
     -+added file), matching what `git diff` itself emits for empty
     -+file sides.
     ++replacement range in the new file.  The four fields are separated by
     ++single spaces.  Start values are 1-based and counts are non-negative.
     ++For example, `hunk 3 2 3 4` means that 2 lines starting at line 3 in
     ++the old file were replaced by 4 lines starting at line 3 in the new
     ++file.  An `<old_count>` of 0 means no lines were removed (pure
     ++insertion); a `<new_count>` of 0 means no lines were added (pure
     ++deletion).  For a side with a count of 0 (a pure insertion or
     ++deletion) the start is the 1-based line the change sits before,
     ++ranging from 1 to one past the last line (the line count plus 1, to
     ++place the change at the end of the file); like every start it must
     ++keep the unchanged runs aligned on both sides (see below), so for a
     ++given change it takes one specific value, not an arbitrary one.  A
     ++start of 0 is also accepted and treated as 1, matching the
     ++empty-file-side form `git diff` emits (e.g. `hunk 0 0 1 5` for a newly
     ++added file).  A nonzero range must not extend beyond the end of the
     ++file.  Git ignores any extra
     ++whitespace-separated tokens after `<new_count>`, so a future protocol
     ++version can append fields to a hunk line (for example a "moved"
     ++marker) without older tools rejecting it.
      +
      +Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
      +file `"foo\nbar"` both have 2 lines.
      +
     -+Hunks must be listed in order and must not overlap.  Any line
     -+not covered by a hunk is treated as unchanged, so the total
     -+number of unchanged lines must be the same on both sides.
     -+For example, if the old file has 10 lines and the hunks cover
     -+4 of them (`old_count` values summing to 4), then 6 old lines
     -+are unchanged.  The new file must also have exactly 6 lines
     -+not covered by hunks, so the `new_count` values must sum to
     -+`new_file_lines - 6`.
     ++Hunks must be listed in order and must not overlap.  Any line not
     ++covered by a hunk is treated as unchanged and is paired, in order,
     ++with the unchanged lines on the other side.  Each run of unchanged
     ++lines between two hunks (and the run before the first hunk and
     ++after the last) must therefore be the same length on both sides,
     ++not merely equal in total.  For the hunks `1 3 1 5` and `10 2 12 2`
     ++below, lines 4-9 of the old file and lines 6-11 of the new file are
     ++both the six unchanged lines between the two hunks.  A response that
     ++balances only the total unchanged count but misaligns one of these
     ++runs is rejected, and Git falls back to the builtin diff.
     ++
     ++Git does not check that the lines a hunk leaves unchanged are
     ++byte-for-byte identical between the two sides; it pairs them by
     ++position and shows the new side as context.  A tool may therefore
     ++report lines that differ textually (a pure reformatting, say) as
     ++unchanged, and the diff reflects that judgment.  This is
     ++the point of a semantic backend, but it means a misbehaving tool can
     ++produce a diff whose context does not match the old blob; as with
     ++`git diff -w`, such a patch may not apply against the old content.
      +
      +-----------------------
      +packet:          git< hunk 1 3 1 5
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +-----------------------
      +
      +If the tool responds with hunks and "success", Git marks those lines
     -+as changed and feeds them into the standard diff pipeline.  Patch
     -+output features (word diff, function context, color) work normally.
     -+Note that `--stat` and other summary formats use their own diff path
     -+and are not affected by the diff process.
     ++as changed and feeds them into the standard diff pipeline.  Git may
     ++still slide or regroup those changes against matching context for
     ++display, exactly as it compacts its own diffs, so the tool controls
     ++which lines are reported as changed, not the precise hunk boundaries.
     ++Patch output features (word diff, function context, color) work
     ++normally.  Summary formats such as `--stat` still compute their counts
     ++with the builtin diff for now; see "Which features consult the diff
     ++process" below for the full picture and the reasoning behind it.
      +
      +If no hunk lines precede the flush, followed by "success", Git
     -+treats the files as having no changes: `git diff` produces no output
     -+and `git blame` skips the commit, attributing lines to earlier commits.
     ++treats the files as having no changes: `git diff` produces no output,
     ++`git diff --exit-code` and `--quiet` report success even though the
     ++stored blobs differ, and `git blame` skips the commit, attributing
     ++lines to earlier commits.
     ++The one exception is a change that only adds or removes the file's
     ++trailing newline: it cannot be expressed as line hunks, so when the
     ++line content otherwise matches Git keeps the builtin diff for that
     ++file (preserving the `\ No newline at end of file` marker) instead of
     ++treating the two sides as equal.
      +
      +-----------------------
      +packet:          git< 0000
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +packet:          git< 0000
      +-----------------------
      +
     -+If the tool returns invalid hunks (out of bounds, overlapping, or
     -+mismatched unchanged line counts), Git warns and falls back to the
     -+builtin diff algorithm.
     ++If the tool returns well-formed but invalid hunks (out of bounds,
     ++overlapping, or with misaligned unchanged runs), Git warns and falls
     ++back to the builtin diff for that file; the tool stays available for
     ++subsequent files.  A malformed hunk line, by contrast (bad syntax, a
     ++nonzero count paired with a start of 0, or more hunks than the file
     ++has lines), is a protocol violation: Git stops the process and does
     ++not send it further requests, as described below.
      +
      +In case the tool cannot or does not want to process the content,
      +it is expected to respond with an "error" status.  Git warns and
     -+falls back to the builtin diff algorithm for this file.  The tool
     ++falls back to the builtin diff algorithm for this file, treating any
     ++status other than "success" or "abort" the same way.  The tool
      +remains available for subsequent files.
      +
      +-----------------------
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +forward-compatible.  Future versions of Git may send additional
      +`command=` values; tools that receive an unrecognized command should
      +respond with `status=error` rather than terminating.
     ++
     ++Which features consult the diff process
     ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ++
     ++The diff process answers a single question: given two blobs, which
     ++line ranges differ?  Whether a particular feature consults it follows
     ++from whether that is the question the feature is really asking.
     ++
     ++Features that ask "which lines changed" use the tool's hunks in place
     ++of the builtin algorithm:
     ++
     ++- `git diff` patch output, together with everything layered on it:
     ++  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
     ++  headers, and the `-L` line-range display.  These operate on the
     ++  lines the patch step already emitted, so they reflect the tool's
     ++  hunks without any further negotiation.
     ++- `git blame`: a commit whose change the tool reports as equivalent is
     ++  skipped, and its lines are attributed to an earlier commit.
     ++
     ++Features that ask a different question do not consult the process, by
     ++design:
     ++
     ++- The pickaxe `-G<regex>` searches the textual diff for a pattern; it
     ++  asks "does this string appear in the diff," not "did these lines
     ++  change."  (`-S` runs at an earlier stage and is likewise unaffected.)
     ++- `git patch-id` must produce a stable hash for `git rebase` and
     ++  cherry-pick detection; deriving it from a configured tool would make
     ++  equal patches hash differently from machine to machine.
     ++- The merge machinery (`git merge-tree`, `rerere`) computes merge
     ++  content and conflict signatures rather than display output, so the
     ++  tool's hunks must not alter its results.
     ++- `git range-diff` diffs patch text, not source blobs, so source-file
     ++  hunks do not apply to it.
     ++- `--check` reports whitespace errors in added lines using the builtin
     ++  diff's notion of which lines are added, not the tool's.  It can
     ++  therefore flag (and exit non-zero on) a line the tool treats as
     ++  unchanged and that `git diff` shows as context.  Whitespace breakage
     ++  is a property of the literal bytes, so `--check` keeps the builtin
     ++  partition deliberately; a future change could wire it to the tool if
     ++  matching `git diff` exactly became desirable.
     ++- `--raw`, `--name-only`, and `--name-status` compare object ids at
     ++  the tree level and never run a line-level diff at all.
     ++
     ++Some features ask "which lines changed" but still use the builtin
     ++algorithm for now, and may consult the process in a later change: the
     ++summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
     ++commit selection and parent range propagation (as distinct from its
     ++display, which is covered above); and combined diffs (`--cc` and merge
     ++diffs), whose protocol would have to be extended from a single old/new
     ++pair to one comparison per merge parent.
     ++
     ++`--diff-algorithm` bypasses the process entirely, for every feature
     ++listed above.  The whitespace-ignoring options (`-w`,
     ++`--ignore-space-change`, `--ignore-blank-lines`, and the like),
     ++`-I<regex>`, and `--anchored` also bypass it for the affected files:
     ++the tool is never told about these options, so it could not honor
     ++them, and Git falls back to the builtin diff, which does.
      +
       Defining a custom hunk-header
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     @@ diff-process.c (new)
      + *   tool< capability=hunks / flush
      + *
      + * Per-file:
     -+ *   git> command=hunks / pathname=<path> / flush
     ++ *   git> command=hunks / pathname=<path> / [old-oid=<hex>] / [new-oid=<hex>] / flush
      + *   git> <old content packetized> / flush
      + *   git> <new content packetized> / flush
      + *   tool< hunk <old_start> <old_count> <new_start> <new_count>
     @@ diff-process.c (new)
      +#include "diff-process.h"
      +#include "diff.h"
      +#include "gettext.h"
     ++#include "hex.h"
      +#include "repository.h"
      +#include "sigchain.h"
      +#include "userdiff.h"
     @@ diff-process.c (new)
      +	return packet_flush_gently(fd);
      +}
      +
     -+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
     ++/*
     ++ * A hunk in the diff process's presentation coordinates: the line
     ++ * numbering it reports over the protocol.  Kept distinct from struct
     ++ * xdl_hunk (xdiff's coordinates) so that only translated hunks ever
     ++ * reach the diff algorithm; diff_process_hunk_to_xdl() is the single
     ++ * crossing point.
     ++ */
     ++struct diff_process_hunk {
     ++	long old_start, old_count;
     ++	long new_start, new_count;
     ++};
     ++
     ++/*
     ++ * Parse one non-negative decimal field of a hunk line into *out and
     ++ * advance *line past it.  Fields must be plain decimal with no leading
     ++ * whitespace or sign (isdigit() takes an unsigned char to stay defined
     ++ * for high-bit bytes).  The first three fields are followed by a single
     ++ * space; the last (is_last) is followed by end-of-string or a space.
     ++ * Trailing space-separated tokens after the last field are allowed and
     ++ * ignored, so a future protocol version can append fields (e.g. a
     ++ * "moved" marker) without older tools rejecting the line -- mirroring
     ++ * the request-side rule that tools ignore unknown keys.
     ++ */
     ++static int parse_hunk_field(const char **line, long *out, int is_last)
      +{
     ++	const char *p = *line;
      +	char *end;
      +
     -+	/*
     -+	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
     -+	 * All numbers must be non-negative decimal with no leading
     -+	 * whitespace or sign characters.
     -+	 */
     -+	if (!skip_prefix(line, "hunk ", &line))
     -+		return -1;
     -+
     -+	if (!isdigit(*line))
     ++	if (!isdigit((unsigned char)*p))
      +		return -1;
      +	errno = 0;
     -+	hunk->old_start = strtol(line, &end, 10);
     -+	if (errno || end == line || *end++ != ' ')
     ++	*out = strtol(p, &end, 10);
     ++	if (errno || end == p)
      +		return -1;
     -+	line = end;
     ++	if (is_last) {
     ++		if (*end != '\0' && *end != ' ')
     ++			return -1;
     ++	} else {
     ++		if (*end != ' ')
     ++			return -1;
     ++		end++;
     ++	}
     ++	*line = end;
     ++	return 0;
     ++}
      +
     -+	if (!isdigit(*line))
     ++static int parse_hunk_line(const char *line,
     ++			   struct diff_process_hunk *presented)
     ++{
     ++	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
     ++	if (!skip_prefix(line, "hunk ", &line))
      +		return -1;
     -+	errno = 0;
     -+	hunk->old_count = strtol(line, &end, 10);
     -+	if (errno || end == line || *end++ != ' ')
     ++	if (parse_hunk_field(&line, &presented->old_start, 0) ||
     ++	    parse_hunk_field(&line, &presented->old_count, 0) ||
     ++	    parse_hunk_field(&line, &presented->new_start, 0) ||
     ++	    parse_hunk_field(&line, &presented->new_count, 1))
      +		return -1;
     -+	line = end;
     ++	return 0;
     ++}
      +
     -+	if (!isdigit(*line))
     -+		return -1;
     -+	errno = 0;
     -+	hunk->new_start = strtol(line, &end, 10);
     -+	if (errno || end == line || *end++ != ' ')
     -+		return -1;
     -+	line = end;
     ++/*
     ++ * Translate a hunk from the diff process's presentation coordinates
     ++ * into xdiff's.
     ++ *
     ++ * Protocol starts are already 1-based positions (the line a change
     ++ * sits before), the same numbering xdiff uses, so the only adjustment
     ++ * is for an empty file side: "git diff" addresses it with a start of 0
     ++ * and a count of 0 (e.g. "0 0 1 5" adds five lines to an empty old
     ++ * side), and since xdiff uses start-1 as an array index that 0 becomes
     ++ * 1 here.  This is NOT the full inverse of xdl_emit_hunk_hdr()
     ++ * (xdiff/xutils.c): that emitter shifts a count-0 range to start-1 for
     ++ * the displayed "@@" header, but the protocol keeps the unshifted
     ++ * 1-based position for a mid-file insert or delete.  This is the single
     ++ * point where presentation coordinates become xdiff coordinates, so
     ++ * xdl_populate_hunks_from_external() may assume 1-based starts.
     ++ *
     ++ * Returns -1 for a start of 0 paired with a nonzero count, which names
     ++ * no line in either coordinate system.  (parse_hunk_line() already
     ++ * guarantees non-negative starts and counts.)
     ++ */
     ++static int diff_process_hunk_to_xdl(const struct diff_process_hunk *presented,
     ++				    struct xdl_hunk *xdl)
     ++{
     ++	long old_start = presented->old_start;
     ++	long new_start = presented->new_start;
      +
     -+	if (!isdigit(*line))
     ++	if ((!old_start && presented->old_count) ||
     ++	    (!new_start && presented->new_count))
      +		return -1;
     -+	errno = 0;
     -+	hunk->new_count = strtol(line, &end, 10);
     -+	if (errno || end == line || *end != '\0')
     -+		return -1;
     -+
     -+	/*
     -+	 * git diff emits start=0 when count=0 (empty file side).
     -+	 * Normalize to 1-based so downstream validation can assume start >= 1.
     -+	 */
     -+	if (!hunk->old_count && !hunk->old_start)
     -+		hunk->old_start = 1;
     -+	if (!hunk->new_count && !hunk->new_start)
     -+		hunk->new_start = 1;
     -+
     ++	if (!old_start)
     ++		old_start = 1;
     ++	if (!new_start)
     ++		new_start = 1;
     ++
     ++	xdl->old_start = old_start;
     ++	xdl->old_count = presented->old_count;
     ++	xdl->new_start = new_start;
     ++	xdl->new_count = presented->new_count;
      +	return 0;
      +}
      +
     @@ diff-process.c (new)
      +		const char *path,
      +		const char *old_buf, long old_size,
      +		const char *new_buf, long new_size,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
      +		struct xdl_hunk **hunks_out,
      +		size_t *nr_hunks_out)
      +{
     @@ diff-process.c (new)
      +	int fd_in, fd_out;
      +	struct strbuf status = STRBUF_INIT;
      +	struct xdl_hunk *hunks = NULL;
     ++	struct diff_process_hunk presented;
      +	struct xdl_hunk hunk;
      +	size_t nr_hunks = 0, alloc_hunks = 0;
     ++	size_t max_hunks;
      +	int len;
      +	char *line;
      +
     @@ diff-process.c (new)
      +
      +	/* Send request */
      +	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
     -+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
     -+	    packet_flush_gently(fd_in))
     ++	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path))
     ++		goto comm_error;
     ++	/*
     ++	 * old-oid/new-oid let the tool key a cache on the blob pair.  A
     ++	 * side is sent only when its content is the raw blob (the caller
     ++	 * passes NULL otherwise, e.g. for textconv'd content), so an oid
     ++	 * that is present always names the bytes the tool receives.
     ++	 */
     ++	if (oid_a &&
     ++	    packet_write_fmt_gently(fd_in, "old-oid=%s\n", oid_to_hex(oid_a)))
     ++		goto comm_error;
     ++	if (oid_b &&
     ++	    packet_write_fmt_gently(fd_in, "new-oid=%s\n", oid_to_hex(oid_b)))
     ++		goto comm_error;
     ++	if (packet_flush_gently(fd_in))
      +		goto comm_error;
      +
      +	/* Send old file content */
     @@ diff-process.c (new)
      +	if (send_file_content(fd_in, new_buf, new_size))
      +		goto comm_error;
      +
     ++	/*
     ++	 * Hunks are non-overlapping and each useful hunk covers at least
     ++	 * one line, so a valid response cannot contain more hunks than the
     ++	 * two files have lines, which is bounded by their byte sizes.  Cap
     ++	 * the accumulation accordingly so a misbehaving tool that floods
     ++	 * hunk lines cannot drive unbounded memory growth before validation.
     ++	 */
     ++	max_hunks = (size_t)old_size + (size_t)new_size + 1;
     ++
      +	/* Read hunks until flush packet */
      +	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
      +	       line) {
     -+		if (parse_hunk_line(line, &hunk) < 0)
     ++		if (parse_hunk_line(line, &presented) < 0)
     ++			goto comm_error;
     ++		if (diff_process_hunk_to_xdl(&presented, &hunk) < 0)
     ++			goto comm_error;
     ++		if (nr_hunks >= max_hunks) {
     ++			warning(_("diff process '%s' sent too many hunks"
     ++				  " for '%s'"), drv->process, path);
      +			goto comm_error;
     ++		}
      +		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
      +		hunks[nr_hunks++] = hunk;
      +	}
     @@ diff-process.c (new)
      +	return DIFF_PROCESS_ERROR;
      +}
      +
     ++/*
     ++ * Whether exactly one of the two blobs ends in a newline.  A change
     ++ * that only adds or removes the trailing newline is not expressible as
     ++ * line hunks, so a tool comparing lines reports the files as equal.
     ++ */
     ++static int eof_newline_differs(const mmfile_t *a, const mmfile_t *b)
     ++{
     ++	int a_nl = a->size > 0 && a->ptr[a->size - 1] == '\n';
     ++	int b_nl = b->size > 0 && b->ptr[b->size - 1] == '\n';
     ++	return a_nl != b_nl;
     ++}
     ++
     ++/*
     ++ * Number of lines in a blob, matching xdiff's record count: one per
     ++ * newline, plus one more if the last line has no trailing newline.
     ++ */
     ++static long count_lines(const char *buf, long size)
     ++{
     ++	long lines = 0, i;
     ++
     ++	for (i = 0; i < size; i++)
     ++		if (buf[i] == '\n')
     ++			lines++;
     ++	if (size > 0 && buf[size - 1] != '\n')
     ++		lines++;
     ++	return lines;
     ++}
     ++
     ++/*
     ++ * Validate the tool's hunks (already in xdiff coordinates) against the
     ++ * two blobs before they bypass the diff algorithm.  Each hunk must fit
     ++ * within its file, the hunks must be ordered and non-overlapping, and
     ++ * the unchanged run before each hunk (and after the last) must be the
     ++ * same length on both sides -- xdl_build_script() walks the two files
     ++ * in lockstep over unchanged lines, so a mismatched gap desynchronizes
     ++ * it and yields a corrupt diff even when the totals balance.  This is
     ++ * the git layer's job so xdiff stays diagnostic-free; on a bad response
     ++ * we warn and the caller falls back to the builtin diff.  Returns 0 if
     ++ * valid, -1 (after warning) otherwise.
     ++ */
     ++static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,
     ++				   long old_lines, long new_lines,
     ++				   const char *process, const char *path)
     ++{
     ++	size_t i;
     ++	long prev_old_end = 0, prev_new_end = 0;
     ++
     ++	for (i = 0; i < nr; i++) {
     ++		const struct xdl_hunk *h = &hunks[i];
     ++
     ++		if (h->old_count > old_lines - h->old_start + 1 ||
     ++		    h->new_count > new_lines - h->new_start + 1) {
     ++			warning(_("diff process '%s' returned a hunk past the "
     ++				  "end of '%s'; using the builtin diff"),
     ++				process, path);
     ++			return -1;
     ++		}
     ++		if (h->old_start < prev_old_end || h->new_start < prev_new_end) {
     ++			warning(_("diff process '%s' returned overlapping hunks "
     ++				  "for '%s'; using the builtin diff"),
     ++				process, path);
     ++			return -1;
     ++		}
     ++		if (h->old_start - prev_old_end != h->new_start - prev_new_end) {
     ++			warning(_("diff process '%s' returned hunks that leave "
     ++				  "'%s' misaligned; using the builtin diff"),
     ++				process, path);
     ++			return -1;
     ++		}
     ++		prev_old_end = h->old_start + h->old_count;
     ++		prev_new_end = h->new_start + h->new_count;
     ++	}
     ++	if (old_lines - prev_old_end != new_lines - prev_new_end) {
     ++		warning(_("diff process '%s' returned hunks that leave '%s' "
     ++			  "misaligned; using the builtin diff"),
     ++			process, path);
     ++		return -1;
     ++	}
     ++	return 0;
     ++}
     ++
      +enum diff_process_result diff_process_fill_hunks(
      +		struct diff_options *diffopt,
      +		const char *path,
      +		const mmfile_t *file_a,
      +		const mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
      +		xpparam_t *xpp)
      +{
      +	struct userdiff_driver *drv;
     @@ diff-process.c (new)
      +		return DIFF_PROCESS_SKIP;
      +	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
      +		return DIFF_PROCESS_SKIP;
     ++	/*
     ++	 * Whitespace-ignoring, regex-ignore (-I) and anchored options
     ++	 * change which lines count as different, but the tool is never
     ++	 * told about them, so its hunks could not honor them.  Rather
     ++	 * than silently override the user's request, fall back to the
     ++	 * builtin diff, which does honor these flags.  Key this off xpp
     ++	 * (the parameters this diff actually runs with) rather than
     ++	 * diffopt, so a caller like blame that keeps its flags outside
     ++	 * diffopt is covered without a separate guard of its own.
     ++	 */
     ++	if ((xpp->flags & (XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES)) ||
     ++	    xpp->ignore_regex_nr || xpp->anchors_nr)
     ++		return DIFF_PROCESS_SKIP;
      +
      +	drv = userdiff_find_by_path(diffopt->repo->index, path);
      +	if (!drv || !drv->process)
     @@ diff-process.c (new)
      +	res = get_hunks(drv, path,
      +			file_a->ptr, file_a->size,
      +			file_b->ptr, file_b->size,
     ++			oid_a, oid_b,
      +			&ext_hunks, &nr);
      +	if (res == DIFF_PROCESS_OK) {
      +		if (!nr) {
      +			free(ext_hunks);
     ++			/*
     ++			 * Zero hunks means the tool considers the line
     ++			 * content identical, but it cannot express a
     ++			 * trailing-newline-only change.  When that is the
     ++			 * actual difference, fall back to the builtin diff
     ++			 * so the "\ No newline at end of file" marker is
     ++			 * preserved instead of reporting the files equal.
     ++			 */
     ++			if (eof_newline_differs(file_a, file_b))
     ++				return DIFF_PROCESS_SKIP;
      +			return DIFF_PROCESS_EQUIVALENT;
      +		}
     ++		if (validate_external_hunks(ext_hunks, nr,
     ++					    count_lines(file_a->ptr, file_a->size),
     ++					    count_lines(file_b->ptr, file_b->size),
     ++					    drv->process, path) < 0) {
     ++			free(ext_hunks);
     ++			return DIFF_PROCESS_SKIP;
     ++		}
      +		xpp->external_hunks = ext_hunks;
      +		xpp->external_hunks_nr = nr;
      +		return DIFF_PROCESS_OK;
     @@ diff-process.h (new)
      +#include "xdiff/xdiff.h"
      +
      +struct diff_options;
     ++struct object_id;
      +
      +enum diff_process_result {
     -+	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
     ++	DIFF_PROCESS_ERROR = -1, /* failed; caller falls back to builtin */
      +	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
     -+	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
     ++	DIFF_PROCESS_SKIP,       /* process did not apply: use builtin */
      +	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
      +};
      +
     @@ diff-process.h (new)
      + * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
      + * The caller owns xpp->external_hunks and must free() it.
      + *
     -+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
     -+ * (files are considered identical); caller should skip diff/blame.
     ++ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks and
     ++ * the blobs are not a trailing-newline-only change (files are
     ++ * considered identical); caller should skip diff/blame.
      + * Returns DIFF_PROCESS_SKIP when no process applies; caller
      + * should use the builtin diff algorithm.
      + * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
      + * caller should fall back to the builtin diff algorithm.
     ++ *
     ++ * oid_a/oid_b, when non-NULL, are sent to the tool as old-oid/new-oid
     ++ * so it can key a cache on the blob pair.  Pass NULL for a side whose
     ++ * content is not the raw blob (e.g. textconv'd) or whose object name is
     ++ * unknown, so any oid that is sent always names the bytes the tool
     ++ * receives.
      + */
      +enum diff_process_result diff_process_fill_hunks(
      +		struct diff_options *diffopt,
      +		const char *path,
      +		const mmfile_t *file_a,
      +		const mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
      +		xpparam_t *xpp);
      +
      +#endif /* DIFF_PROCESS_H */
     @@ diff.c: static void builtin_diff(const char *name_a,
       		xpp.anchors = o->anchors;
       		xpp.anchors_nr = o->anchors_nr;
      +
     -+		if (diff_process_fill_hunks(o, name_a,
     -+					    &mf1, &mf2, &xpp)
     ++		/*
     ++		 * Send the blob oids only for a side whose content is the
     ++		 * raw blob: textconv rewrites the bytes, and a working-tree
     ++		 * side has no stored oid, so pass NULL there rather than an
     ++		 * oid that would not name what the tool receives.
     ++		 */
     ++		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
     ++					    (textconv_one || !one->oid_valid) ? NULL : &one->oid,
     ++					    (textconv_two || !two->oid_valid) ? NULL : &two->oid,
     ++					    &xpp)
      +		    == DIFF_PROCESS_EQUIVALENT) {
      +			if (textconv_one)
      +				free(mf1.ptr);
     @@ t/helper/test-diff-process-backend.c (new)
      + *
      + *   packet:          git> command=hunks
      + *   packet:          git> pathname=<path>
     ++ *   packet:          git> [old-oid=<hex>]   (omitted for textconv/worktree)
     ++ *   packet:          git> [new-oid=<hex>]
      + *   packet:          git> 0000
      + *   packet:          git> OLD_CONTENT
      + *   packet:          git> 0000
     @@ t/helper/test-diff-process-backend.c (new)
      + *
      + * Response varies by --mode (default: whole-file):
      + *
     -+ *   whole-file   packet: git< hunk 1 <old_lines> 1 <new_lines>
     ++ *   whole-file   packet: git< hunk <1|0> <old_lines> <1|0> <new_lines>
     ++ *                (start is 0 for an empty side, matching git diff)
      + *   fixed-hunk   packet: git< hunk 5 2 5 2
      + *   no-hunks     (no hunk packets)
      + *   bad-hunk     packet: git< hunk 999 1 999 1
      + *   bad-parse    packet: git< garbage not a hunk
      + *   bad-sync     packet: git< hunk 1 2 1 1
     ++ *   bad-gap      packet: git< hunk 1 1 3 1
     ++ *   bad-start    packet: git< hunk 0 1 1 1
     ++ *   multi-hunk   packet: git< hunk 5 2 5 2
     ++ *                packet: git< hunk 9 2 9 2
     ++ *   insert       packet: git< hunk 3 0 3 2   (mid-file count-0 insertion)
     ++ *   flood        packet: git< hunk 1 1 1 1   (x100000)
      + *   overlap      packet: git< hunk 1 5 1 5
      + *                packet: git< hunk 3 2 3 2
      + *   no-cap       (omits capability=hunks during handshake)
     @@ t/helper/test-diff-process-backend.c (new)
      + *   abort        (status=abort instead of status=success)
      + *   crash        exit(1) before sending any response
      + *
     -+ * All non-error/abort modes end with:
     ++ * All success modes (not error/abort/crash) end with:
      + *
      + *   packet:          git< 0000
      + *   packet:          git< status=success
     @@ t/helper/test-diff-process-backend.c (new)
      + *
      + * Each request is logged to --log as:
      + *
     -+ *   command=<cmd> pathname=<path> old=<first line> new=<first line>
     ++ *   command=<cmd> pathname=<path> old-oid=<hex> new-oid=<hex> old=<first line> new=<first line>
      + */
      +
      +#include "test-tool.h"
     @@ t/helper/test-diff-process-backend.c (new)
      +	MODE_BAD_HUNK,
      +	MODE_BAD_PARSE,
      +	MODE_BAD_SYNC,
     ++	MODE_BAD_GAP,
     ++	MODE_BAD_START,
     ++	MODE_MULTI_HUNK,
     ++	MODE_INSERT,
     ++	MODE_FLOOD,
      +	MODE_OVERLAP,
      +	MODE_NO_CAP,
      +	MODE_ERROR,
     @@ t/helper/test-diff-process-backend.c (new)
      +		return MODE_BAD_PARSE;
      +	if (!strcmp(s, "bad-sync"))
      +		return MODE_BAD_SYNC;
     ++	if (!strcmp(s, "bad-gap"))
     ++		return MODE_BAD_GAP;
     ++	if (!strcmp(s, "bad-start"))
     ++		return MODE_BAD_START;
     ++	if (!strcmp(s, "multi-hunk"))
     ++		return MODE_MULTI_HUNK;
     ++	if (!strcmp(s, "insert"))
     ++		return MODE_INSERT;
     ++	if (!strcmp(s, "flood"))
     ++		return MODE_FLOOD;
      +	if (!strcmp(s, "overlap"))
      +		return MODE_OVERLAP;
      +	if (!strcmp(s, "no-cap"))
     @@ t/helper/test-diff-process-backend.c (new)
      + * variant: once inside a request, truncation is a protocol violation
      + * and dying loudly is the correct response.
      + */
     -+static int read_request_header(char **command, char **pathname)
     ++static int read_request_header(char **command, char **pathname,
     ++			       char **old_oid, char **new_oid)
      +{
      +	int first = 1;
      +	char *line;
      +
     -+	*command = *pathname = NULL;
     ++	*command = *pathname = *old_oid = *new_oid = NULL;
      +	for (;;) {
      +		const char *value;
      +
     @@ t/helper/test-diff-process-backend.c (new)
      +			*command = xstrdup(value);
      +		else if (skip_prefix(line, "pathname=", &value))
      +			*pathname = xstrdup(value);
     ++		else if (skip_prefix(line, "old-oid=", &value))
     ++			*old_oid = xstrdup(value);
     ++		else if (skip_prefix(line, "new-oid=", &value))
     ++			*new_oid = xstrdup(value);
      +	}
      +	return 1;
      +}
     @@ t/helper/test-diff-process-backend.c (new)
      +	case MODE_BAD_SYNC:
      +		packet_write_fmt(1, "hunk 1 2 1 1\n");
      +		break;
     ++	case MODE_BAD_GAP:
     ++		/*
     ++		 * Globally balanced (1 changed line on each side, so the
     ++		 * total unchanged counts match) but the gap before the
     ++		 * change differs between sides: old line 1 vs new line 3.
     ++		 * Exercises the per-gap lockstep-alignment check.
     ++		 */
     ++		packet_write_fmt(1, "hunk 1 1 3 1\n");
     ++		break;
     ++	case MODE_BAD_START:
     ++		/*
     ++		 * A start of 0 is valid only for an empty (count 0) range;
     ++		 * pairing it with a nonzero count names no line in either
     ++		 * the protocol's or xdiff's coordinates, so the translation
     ++		 * rejects it and git falls back to the builtin diff.
     ++		 */
     ++		packet_write_fmt(1, "hunk 0 1 1 1\n");
     ++		break;
     ++	case MODE_MULTI_HUNK:
     ++		/*
     ++		 * Two valid, non-overlapping, gap-aligned hunks.  Exercises
     ++		 * the accepting branch of the per-gap lockstep check with a
     ++		 * non-zero previous-hunk end (the realistic two-region case).
     ++		 */
     ++		packet_write_fmt(1, "hunk 5 2 5 2\n");
     ++		packet_write_fmt(1, "hunk 9 2 9 2\n");
     ++		break;
     ++	case MODE_INSERT:
     ++		/*
     ++		 * A mid-file pure insertion (count 0 on the old side) in the
     ++		 * protocol's 1-based-position form: 2 lines inserted before
     ++		 * old line 3.  Exercises the count-0 path, which uses the
     ++		 * unshifted position (not git diff's "-3,0" display start).
     ++		 */
     ++		packet_write_fmt(1, "hunk 3 0 3 2\n");
     ++		break;
     ++	case MODE_FLOOD: {
     ++		/*
     ++		 * Emit far more hunks than any small file has lines, so Git
     ++		 * trips its accumulation cap and falls back before reading
     ++		 * them all.
     ++		 */
     ++		int i;
     ++		for (i = 0; i < 100000; i++)
     ++			packet_write_fmt(1, "hunk 1 1 1 1\n");
     ++		break;
     ++	}
      +	case MODE_OVERLAP:
      +		packet_write_fmt(1, "hunk 1 5 1 5\n");
      +		packet_write_fmt(1, "hunk 3 2 3 2\n");
     @@ t/helper/test-diff-process-backend.c (new)
      +{
      +	for (;;) {
      +		char *command = NULL, *pathname = NULL;
     ++		char *old_oid = NULL, *new_oid = NULL;
      +		struct strbuf obuf = STRBUF_INIT;
      +		struct strbuf nbuf = STRBUF_INIT;
      +
     -+		if (!read_request_header(&command, &pathname))
     ++		if (!read_request_header(&command, &pathname,
     ++					 &old_oid, &new_oid))
      +			break; /* EOF: Git closed its end */
      +
      +		read_packetized_to_strbuf(0, &obuf, 0);
     @@ t/helper/test-diff-process-backend.c (new)
      +
      +		if (logfile) {
      +			fprintf(logfile,
     -+				"command=%s pathname=%s old=%.*s new=%.*s\n",
     ++				"command=%s pathname=%s old-oid=%s new-oid=%s"
     ++				" old=%.*s new=%.*s\n",
      +				command ? command : "(none)",
      +				pathname ? pathname : "(none)",
     ++				old_oid ? old_oid : "(none)",
     ++				new_oid ? new_oid : "(none)",
      +				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
      +				obuf.buf,
      +				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
     @@ t/helper/test-diff-process-backend.c (new)
      +
      +		free(command);
      +		free(pathname);
     ++		free(old_oid);
     ++		free(new_oid);
      +		strbuf_release(&obuf);
      +		strbuf_release(&nbuf);
      +	}
     @@ t/helper/test-diff-process-backend.c (new)
      +	enum mode mode = MODE_WHOLE_FILE;
      +	struct option options[] = {
      +		OPT_STRING(0, "mode", &mode_str, "mode",
     -+			   "response shape: whole-file (default), fixed-hunk,"
     -+			   " no-hunks, bad-hunk, bad-sync, overlap, error,"
     -+			   " abort, crash"),
     ++			   "response shape (default whole-file);"
     ++			   " see the file header for the full list of modes"),
      +		OPT_STRING(0, "log", &log_path, "path",
      +			   "append per-request summary to this file"),
      +		OPT_END()
     @@ t/t4080-diff-process.sh (new)
      +
      +test_description='diff process via long-running process'
      +
     ++TEST_PASSES_SANITIZE_LEAK=true
      +. ./test-lib.sh
      +
      +# See t/helper/test-diff-process-backend.c for the backend implementation
     @@ t/t4080-diff-process.sh (new)
      +	EOF
      +	git add logtest.c &&
      +
     -+	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
     ++	# one.c/two.c: two-file pair for error/abort/startup-failure tests.
      +	cat >one.c <<-\EOF &&
      +	int first(void) { return 1; }
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +	test_grep ! "^+NEW10" actual
      +'
      +
     ++test_expect_success 'diff process accepts valid multi-hunk output' '
     ++	# multi-hunk reports both changed regions (5-6 and 9-10) as two
     ++	# gap-aligned hunks.  This exercises the accepting branch of the
     ++	# per-gap lockstep check (non-zero previous-hunk end) and must
     ++	# produce a correct two-region diff with the lines between the
     ++	# hunks kept as context.
     ++	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	test_grep "^ line7" actual &&
     ++	test_grep "^ line8" actual &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process accepts a mid-file count-0 insertion' '
     ++	# insert mode reports "hunk 3 0 3 2": a pure insertion (count 0 on
     ++	# the old side) in the protocol 1-based-position form.  Exercises
     ++	# the count-0 hunk path that the other valid-hunk modes (full
     ++	# replacements, equal-count modifies) never hit.  Empty stderr is
     ++	# the discriminator: a mishandled count-0 start would be rejected
     ++	# by the lockstep check and warn.
     ++	cat >insert.c <<-\EOF &&
     ++	a
     ++	b
     ++	c
     ++	d
     ++	e
     ++	EOF
     ++	git add insert.c &&
     ++	git commit -m "add insert.c" &&
     ++	cat >insert.c <<-\EOF &&
     ++	a
     ++	b
     ++	X
     ++	Y
     ++	c
     ++	d
     ++	e
     ++	EOF
     ++	git -c diff.cdiff.process="$BACKEND --mode=insert" \
     ++		diff insert.c >actual 2>stderr &&
     ++	test_grep "^+X" actual &&
     ++	test_grep "^+Y" actual &&
     ++	test_grep "^ c" actual &&
     ++	test_must_be_empty stderr
     ++'
     ++
      +test_expect_success 'diff process works with modified file' '
      +	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     @@ t/t4080-diff-process.sh (new)
      +		diff --exit-code nohunks.c
      +'
      +
     ++test_expect_success 'diff process equivalent commit: --exit-code and --quiet agree' '
     ++	# A committed blob pair (not a worktree file) whose oids differ but
     ++	# the tool reports equivalent.  --exit-code and --quiet must agree
     ++	# with the shown diff (empty) and report success, not fall back to
     ++	# the byte-level "oids differ" answer.
     ++	cat >ecq.c <<-\EOF &&
     ++	alpha
     ++	EOF
     ++	git add ecq.c &&
     ++	git commit -m "ecq v1" &&
     ++	cat >ecq.c <<-\EOF &&
     ++	beta
     ++	EOF
     ++	git add ecq.c &&
     ++	git commit -m "ecq v2" &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff --exit-code HEAD^ HEAD -- ecq.c &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff --quiet HEAD^ HEAD -- ecq.c
     ++'
     ++
     ++test_expect_success 'diff process falls back for trailing-newline-only change' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	printf "a\nb\nc\n" >eofnl.c &&
     ++	git add eofnl.c &&
     ++	git commit -m "add eofnl.c" &&
     ++	printf "a\nb\nc" >eofnl.c &&
     ++	# Same lines, only the final newline removed.  The tool reports
     ++	# no hunks (it sees identical lines), but that change is not
     ++	# expressible as hunks, so git falls back to the builtin diff
     ++	# rather than treating the files as equivalent.
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		diff eofnl.c >actual 2>stderr &&
     ++	test_grep "No newline at end of file" actual &&
     ++	test_grep "pathname=eofnl.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process falls back for added file (empty old side)' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	printf "x\ny\nz\n" >addnl.c &&
     ++	git add addnl.c &&
     ++	# The empty old side has no trailing newline while the new side
     ++	# does, so the newline fallback shows the addition rather than
     ++	# letting no-hunks suppress the whole new file.
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		diff --cached addnl.c >actual 2>stderr &&
     ++	test_grep "^+x" actual &&
     ++	test_grep "pathname=addnl.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
      +test_expect_success 'diff process with --exit-code and hunks returns failure' '
      +	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
      +		diff --exit-code newfile.c
     @@ t/t4080-diff-process.sh (new)
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success 'diff process not used by --stat' '
     ++test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
      +	test_when_finished "rm -f backend.log" &&
     -+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     -+		diff --stat worddiff.c >actual &&
     -+	test_grep "worddiff.c" actual &&
     -+	test_path_is_missing backend.log
     ++	printf "a\nb\nc\n" >wsbypass.c &&
     ++	git add wsbypass.c &&
     ++	git commit -m "add wsbypass.c" &&
     ++	printf "a\n  b  \nc\n" >wsbypass.c &&
     ++	# The tool is never told about these options and could not honor
     ++	# them, so git bypasses the process for each (covering the whole
     ++	# XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES mask, not just -w).
     ++	for opt in -w -b --ignore-space-at-eol --ignore-blank-lines
     ++	do
     ++		rm -f backend.log &&
     ++		git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++			diff $opt wsbypass.c >actual 2>stderr &&
     ++		test_path_is_missing backend.log &&
     ++		test_must_be_empty stderr ||
     ++		return 1
     ++	done &&
     ++	# -w additionally suppresses the whitespace-only change via the
     ++	# builtin diff that now runs.
     ++	git -c diff.cdiff.process="$BACKEND" diff -w wsbypass.c >actual &&
     ++	test_must_be_empty actual
      +'
      +
      +#
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "^+NEW5" actual &&
      +	test_grep "^-OLD9" actual &&
      +	test_grep "^+NEW9" actual &&
     -+	test_grep "exceeds.*lines" stderr
     ++	test_grep "hunk past the end" stderr
      +'
      +
      +test_expect_success 'diff process fallback on mismatched unchanged totals' '
     @@ t/t4080-diff-process.sh (new)
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
      +		diff synctest.c >actual 2>stderr &&
      +	test_grep "changed" actual &&
     -+	test_grep "unchanged line count mismatch" stderr
     ++	test_grep "misaligned" stderr
     ++'
     ++
     ++test_expect_success 'diff process fallback on misaligned hunk gap' '
     ++	# bad-gap reports hunk 1 1 3 1 on boundary.c: one changed line
     ++	# on each side, so the total unchanged counts match, but the
     ++	# unchanged run before the change differs (old line 1 vs new
     ++	# line 3).  A global count check would accept this and emit a
     ++	# corrupt diff; the per-gap lockstep check rejects it and git
     ++	# falls back to the builtin algorithm.
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-gap" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	# The builtin fallback shows both changed regions as additions
     ++	# (a corrupt-accepted hunk would show NEW5 only as context).
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	test_grep "misaligned" stderr
      +'
      +
      +test_expect_success 'diff process fallback on overlapping hunks' '
      +	# boundary.c has 10 lines, so both hunks are in bounds
     -+	# but they overlap at lines 3-5, triggering the ordering check.
     ++	# but they overlap at lines 3-4, triggering the ordering check.
      +	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "NEW5" actual &&
     -+	test_grep "overlaps with previous" stderr
     ++	test_grep "overlapping hunks" stderr
      +'
      +
      +test_expect_success 'diff process fallback on malformed hunk line' '
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "^+NEW5" actual
      +'
      +
     -+test_expect_success 'diff process skipped when tool omits capability' '
     -+	git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
     ++test_expect_success 'diff process fallback on start 0 with nonzero count' '
     ++	# bad-start reports hunk 0 1 1 1.  A start of 0 is valid only for
     ++	# an empty (count 0) range, so the presentation-to-xdiff
     ++	# translation rejects it and git falls back to the builtin diff
     ++	# instead of handing xdiff an out-of-range start.
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-start" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "^-OLD5" actual &&
      +	test_grep "^+NEW5" actual &&
     ++	test_grep "diff process.*failed" stderr
     ++'
     ++
     ++test_expect_success 'diff process caps a flood of hunks and falls back' '
     ++	# flood emits far more hunks than the file has lines.  Git must
     ++	# stop accumulating and fall back to the builtin diff rather than
     ++	# grow memory without bound.
     ++	git -c diff.cdiff.process="$BACKEND --mode=flood" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "too many hunks" stderr
     ++'
     ++
     ++test_expect_success 'diff process skipped when tool omits capability' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-cap --log=backend.log" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	# Builtin diff runs: all changes appear, including lines 9-10
     ++	# that a tool-provided hunk would have narrowed away.
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	# The process launched (creating the log) but was
     ++	# never sent a per-file request, so no hunks command is logged.
     ++	test_path_is_file backend.log &&
     ++	test_grep ! "command=hunks" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process receives old-oid and new-oid for a blob pair' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	cat >oidpair.c <<-\EOF &&
     ++	int f(void) { return 1; }
     ++	EOF
     ++	git add oidpair.c &&
     ++	git commit -m "oidpair v1" &&
     ++	old=$(git rev-parse HEAD:oidpair.c) &&
     ++
     ++	cat >oidpair.c <<-\EOF &&
     ++	int f(void) { return 2; }
     ++	EOF
     ++	git add oidpair.c &&
     ++	git commit -m "oidpair v2" &&
     ++	new=$(git rev-parse HEAD:oidpair.c) &&
     ++
     ++	# Both sides are stored blobs, so their object names are sent.
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff HEAD^ HEAD -- oidpair.c >actual 2>stderr &&
     ++	test_grep "old-oid=$old new-oid=$new" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process omits old-oid and new-oid for textconv content' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	write_script oidcat <<-\EOF &&
     ++	cat "$1"
     ++	EOF
     ++	cat >oidtc.c <<-\EOF &&
     ++	alpha
     ++	EOF
     ++	git add oidtc.c &&
     ++	git commit -m "oidtc v1" &&
     ++	cat >oidtc.c <<-\EOF &&
     ++	beta
     ++	EOF
     ++	git add oidtc.c &&
     ++	git commit -m "oidtc v2" &&
     ++
     ++	# textconv rewrites the bytes, so the raw-blob object name that
     ++	# would otherwise identify each side is omitted.
     ++	git -c diff.cdiff.textconv="./oidcat" \
     ++	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff HEAD^ HEAD -- oidtc.c >actual 2>stderr &&
     ++	test_grep "pathname=oidtc.c" backend.log &&
     ++	test_grep "old-oid=(none) new-oid=(none)" backend.log &&
      +	test_must_be_empty stderr
      +'
      +
  5:  6ec6716ea4 !  6:  b2e80f014e diff: bypass diff process with --no-ext-diff and in format-patch
     @@ Documentation/diff-options.adoc: endif::git-format-patch[]
       `--textconv`::
       `--no-textconv`::
      
     + ## Documentation/gitattributes.adoc ##
     +@@ Documentation/gitattributes.adoc: display, which is covered above); and combined diffs (`--cc` and merge
     + diffs), whose protocol would have to be extended from a single old/new
     + pair to one comparison per merge parent.
     + 
     +-`--diff-algorithm` bypasses the process entirely, for every feature
     +-listed above.  The whitespace-ignoring options (`-w`,
     +-`--ignore-space-change`, `--ignore-blank-lines`, and the like),
     ++`--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
     ++for every feature listed above.  The whitespace-ignoring options
     ++(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like),
     + `-I<regex>`, and `--anchored` also bypass it for the affected files:
     + the tool is never told about these options, so it could not honor
     + them, and Git falls back to the builtin diff, which does.
     +
       ## builtin/log.c ##
      @@ builtin/log.c: int cmd_format_patch(int argc,
       	if (argc > 1)
     @@ diff.h: struct diff_flags {
       
      -	/** Disables diff.<driver>.process. */
      +	/**
     -+	 * Disables diff.<driver>.process.  Set by --no-ext-diff.
     ++	 * Disables diff.<driver>.process.  Set by --no-ext-diff and by
     ++	 * format-patch.
      +	 */
       	unsigned no_diff_process;
       
     @@ t/t4080-diff-process.sh: test_expect_success 'diff process bypassed by --diff-al
      +	test_path_is_missing backend.log
      +'
      +
     - test_expect_success 'diff process not used by --stat' '
     + test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
       	test_when_finished "rm -f backend.log" &&
     - 	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     + 	printf "a\nb\nc\n" >wsbypass.c &&
  6:  3dadafa1bc !  7:  cf5bb8984a blame: consult diff process for no-hunk detection
     @@ Commit message
          commit as having no changes, causing blame to attribute lines
          to earlier commits.
      
     -    The consultation happens at the pass_blame_to_parent() callsite
     -    using diff_process_fill_hunks(), matching how builtin_diff() in
     -    diff.c uses the same function.  A new diff_hunks_xpp() variant
     -    accepts a pre-populated xpparam_t so callers can pass external
     -    hunks, while the existing diff_hunks() retains its original
     -    signature and behavior.  The copy-detection callsite is
     -    unaffected since it does not use the diff process.
     +    Introduce xdi_diff_process(), a process-aware xdi_diff() that
     +    consults the process, runs xdiff on the tool's hunks or on the
     +    builtin algorithm when it does not apply, frees the hunks, and
     +    reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller
     +    can drop or skip the change.  It is the shared consult-then-diff path
     +    for consumers that work on raw hunks: blame's pass_blame_to_parent()
     +    uses it here, and git log -L reuses it later.  builtin_diff() keeps
     +    consulting the process directly, because it tests for equivalence
     +    early, before its funcname-pattern and word-diff setup, so a
     +    reformat-only file short-circuits without that work.
      
     -    The subprocess is long-running (one startup cost amortized
     -    across the blame traversal), but each commit in the file's
     -    history incurs a round-trip to the tool.
     +    Blame's -w option is not communicated to the process and it could not
     +    honor it, so blame must fall back to the builtin diff there.  Because
     +    blame keeps its whitespace flags in sb->xdl_opts rather than diffopt,
     +    the process bypass keys off xpp (the flags the diff actually runs
     +    with), which covers blame without a guard of its own.
     +
     +    The subprocess is long-running (one startup cost amortized across the
     +    blame traversal), but each commit in the file's history incurs a
     +    round-trip to the tool.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ blame.c
       #include "alloc.h"
       #include "commit-slab.h"
       #include "bloom.h"
     -@@ blame.c: static struct commit *fake_working_tree_commit(struct repository *r,
     - 
     - 
     - 
     --static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     --		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     -+static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
     -+			  xdl_emit_hunk_consume_func_t hunk_func,
     -+			  void *cb_data, xpparam_t *xpp)
     - {
     --	xpparam_t xpp = {0};
     - 	xdemitconf_t xecfg = {0};
     - 	xdemitcb_t ecb = {NULL};
     - 
     --	xpp.flags = xdl_opts;
     - 	xecfg.hunk_func = hunk_func;
     - 	ecb.priv = cb_data;
     --	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
     -+	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
     -+}
     -+
     -+static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     -+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     -+{
     -+	xpparam_t xpp = {0};
     -+
     -+	xpp.flags = xdl_opts;
     -+	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
     - }
     - 
     - static const char *get_next_line(const char *start, const char *end)
      @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       				 struct blame_origin *parent, int ignore_diffs)
       {
       	mmfile_t file_p, file_o;
      +	xpparam_t xpp = {0};
     ++	xdemitconf_t xecfg = {0};
     ++	xdemitcb_t ecb = {NULL};
       	struct blame_chunk_cb_data d;
       	struct blame_entry *newdest = NULL;
       
     @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       	sb->num_get_patch++;
       
      -	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
     --		die("unable to generate diff (%s -> %s)",
     --		    oid_to_hex(&parent->commit->object.oid),
     --		    oid_to_hex(&target->commit->object.oid));
      +	xpp.flags = sb->xdl_opts;
     ++	xecfg.hunk_func = blame_chunk_cb;
     ++	ecb.priv = &d;
      +	/*
     -+	 * If the diff process considers the files equivalent,
     -+	 * skip the diff so blame looks past this commit.
     ++	 * Consult the diff process, then attribute the resulting chunks
     ++	 * via blame_chunk_cb.  It bypasses the process for the whitespace-
     ++	 * ignoring options it cannot honor (they live in xpp.flags, which
     ++	 * the consultation checks), and when the process reports the blobs
     ++	 * equivalent it runs no diff, so blame passes this commit and looks
     ++	 * past it.  Look up the driver by the parent (old) path, as
     ++	 * builtin_diff() does with name_a, so a renamed file resolves to the
     ++	 * same driver across diff, blame, and line-log.  Pass no
     ++	 * old-oid/new-oid: blame diffs each blob pair once, so the tool gains
     ++	 * nothing from a per-invocation cache key.
      +	 */
     -+	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
     -+				    &file_p, &file_o, &xpp)
     -+	    != DIFF_PROCESS_EQUIVALENT) {
     -+		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
     -+				   &d, &xpp))
     -+			die("unable to generate diff (%s -> %s)",
     -+			    oid_to_hex(&parent->commit->object.oid),
     -+			    oid_to_hex(&target->commit->object.oid));
     -+	}
     -+	free(xpp.external_hunks);
     - 	/* The rest are the same as the parent */
     - 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
     - 		    parent, target, 0);
     ++	if (xdi_diff_process(&sb->revs->diffopt, parent->path,
     ++			     &file_p, &file_o, NULL, NULL, &xpp, &xecfg, &ecb)
     ++	    == DIFF_PROCESS_ERROR)
     + 		die("unable to generate diff (%s -> %s)",
     + 		    oid_to_hex(&parent->commit->object.oid),
     + 		    oid_to_hex(&target->commit->object.oid));
     +
     + ## diff-process.c ##
     +@@
     + #include "sub-process.h"
     + #include "pkt-line.h"
     + #include "strbuf.h"
     ++#include "xdiff-interface.h"
     + #include "xdiff/xdiff.h"
     + 
     + #define CAP_HUNKS (1u << 0)
     +@@ diff-process.c: enum diff_process_result diff_process_fill_hunks(
     + 	}
     + 	return DIFF_PROCESS_SKIP;
     + }
     ++
     ++enum diff_process_result xdi_diff_process(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		mmfile_t *file_a,
     ++		mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
     ++		xpparam_t *xpp,
     ++		xdemitconf_t *xecfg,
     ++		xdemitcb_t *ecb)
     ++{
     ++	enum diff_process_result res;
     ++
     ++	/*
     ++	 * Consult the diff process, then run xdiff either constrained to
     ++	 * the tool's hunks or, when the process does not apply, computing
     ++	 * the diff itself as a fallback.  EQUIVALENT short-circuits: the
     ++	 * caller decides what "no change" means for it (drop the commit,
     ++	 * skip the file, ...), so xdiff is not run.
     ++	 *
     ++	 * A SKIP/ERROR from the process just selects the builtin path
     ++	 * (its warning, if any, was already emitted), so the result then
     ++	 * reflects whether xdiff itself succeeded, not the process.
     ++	 */
     ++	res = diff_process_fill_hunks(diffopt, path, file_a, file_b,
     ++				      oid_a, oid_b, xpp);
     ++	if (res == DIFF_PROCESS_EQUIVALENT)
     ++		return res;
     ++
     ++	res = xdi_diff(file_a, file_b, xpp, xecfg, ecb) < 0
     ++		? DIFF_PROCESS_ERROR : DIFF_PROCESS_OK;
     ++
     ++	FREE_AND_NULL(xpp->external_hunks);
     ++	xpp->external_hunks_nr = 0;
     ++	return res;
     ++}
     +
     + ## diff-process.h ##
     +@@ diff-process.h: enum diff_process_result diff_process_fill_hunks(
     + 		const struct object_id *oid_b,
     + 		xpparam_t *xpp);
     + 
     ++/*
     ++ * Process-aware xdi_diff(): consult the diff process for 'path', then
     ++ * run xdiff either constrained to the tool's hunks or computing the
     ++ * diff itself when the process does not apply or fails.  Frees any
     ++ * hunks it obtained before returning.
     ++ *
     ++ * Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool
     ++ * reports the blobs equal, so the caller can drop or skip the change;
     ++ * DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and
     ++ * DIFF_PROCESS_ERROR if xdiff itself errored.
     ++ *
     ++ * The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as
     ++ * for a direct xdi_diff() call.  oid_a/oid_b are forwarded to
     ++ * diff_process_fill_hunks() (see there).
     ++ */
     ++enum diff_process_result xdi_diff_process(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		mmfile_t *file_a,
     ++		mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
     ++		xpparam_t *xpp,
     ++		xdemitconf_t *xecfg,
     ++		xdemitcb_t *ecb);
     ++
     + #endif /* DIFF_PROCESS_H */
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omits capability' '
     +@@ t/t4080-diff-process.sh: test_expect_success 'diff process omits old-oid and new-oid for textconv content
       	test_must_be_empty stderr
       '
       
     @@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omi
      +	test_grep "$CHANGE" line9 &&
      +	test_path_is_missing backend.log
      +'
     ++
     ++test_expect_success 'blame -w bypasses diff process' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	printf "alpha\nbeta\ngamma\n" >blamew.c &&
     ++	git add blamew.c &&
     ++	git commit -m "add blamew.c" &&
     ++	orig=$(git rev-parse --short HEAD) &&
     ++	printf "alpha\n   beta   \ngamma\n" >blamew.c &&
     ++	git commit -am "reindent beta" &&
     ++	reindent=$(git rev-parse --short HEAD) &&
     ++	# blame -w must ignore the whitespace-only change and attribute
     ++	# beta to the original commit, not the reindent commit.  The tool
     ++	# is never told about -w, so blame must bypass it (not let tool
     ++	# hunks override -w).
     ++	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=backend.log" \
     ++		blame -w blamew.c >actual &&
     ++	sed -n "2p" actual >line2 &&
     ++	test_grep "$orig" line2 &&
     ++	test_grep ! "$reindent" line2 &&
     ++	test_path_is_missing backend.log
     ++'
      +
       test_done
  -:  ---------- >  8:  c1d02d0e15 diff: consult diff process for --stat counts
  -:  ---------- >  9:  c3c17ba8fc line-log: consult diff process for range tracking

-- 
gitgitgadget

[PATCH v5 1/9] gitattributes: document how external diff drivers relate to diff features

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

The "Defining an external diff driver" section explains how to
configure diff.<driver>.command but not how the driver relates to the
rest of Git's diff machinery.  In particular, the command only
replaces the textual patch: word diff, function context, color, and
the like cannot apply to its output, while the summary formats, blame,
and git log -L do not run it at all and keep using the builtin diff.

Spell this out so the scope of an external diff driver is clear.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index bd76167a45..2c4fbfd7f1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -784,6 +784,16 @@ with the above configuration, i.e. `j-c-diff`, with 7
 parameters, just like `GIT_EXTERNAL_DIFF` program is called.
 See linkgit:git[1] for details.
 
+An external diff driver replaces the patch Git would otherwise
+produce for the path: Git runs the command and shows its output in
+place of its own.  Output features that post-process Git's diff do
+not apply to it; word diff, function context (`-W`), `--color-moved`,
+and coloring all act on Git's builtin diff, not the driver's output.
+The driver is consulted only when Git generates a textual patch.  The
+summary formats (`--stat`, `--numstat`, `--shortstat`, and
+`--dirstat`), `git blame`, and `git log -L` do not run it and
+continue to use Git's builtin diff.
+
 If the program is able to ignore certain changes (similar to
 `git diff --ignore-space-change`), then also set the option
 `trustExitCode` to true.  It is then expected to return exit code 1 if
-- 
gitgitgadget

[PATCH v5 2/9] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use.  Out-of-bounds line numbers,
overlapping or out-of-order hunks, and misaligned unchanged runs are
treated as a malformed tool response: xdl_populate_hunks_from_external()
warns, returns -1, and xdl_diff() falls back to the builtin diff
algorithm for that file.  The run of unchanged lines between two hunks
(and before the first and after the last) must be the same length on
both sides; xdl_build_script() walks the two files in lockstep over
unchanged lines, so a balanced total is not enough.  Non-negative
counts and 1-based starts are instead caller preconditions, checked
with BUG(), since the caller normalizes hunks before this point.

On rejection xdl_diff() frees the environment it prepared and falls
through to xdl_do_diff(), which prepares a fresh one for the builtin
pass.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 16 +++++++++
 xdiff/xdiffi.c    | 84 +++++++++++++++++++++++++++++++++++++++++++++--
 xdiff/xprepare.c  | 10 ++++++
 xdiff/xprepare.h  |  1 +
 5 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index db6938689f..1fa16af668 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..4736bcdb07 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,18 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs, in xdiff's own
+ * coordinates: line numbers are 1-based and a hunk's start is the
+ * first line it covers.  A caller translates any external "empty side"
+ * idiom (such as git diff's start-0/count-0) to a 1-based start before
+ * handing hunks over.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +100,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
+	struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index c5a892f91e..73a456f5dd 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,96 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  The caller normalizes and validates
+ * the hunks first (order, overlap, and lockstep alignment), so this
+ * only marks lines changed after asserting the memory-safety
+ * preconditions it depends on: non-negative counts and 1-based starts
+ * (checked with BUG()), and an in-bounds range (a silent -1 so the
+ * caller can fall back to the builtin diff rather than index changed[]
+ * out of range).  Keeping this diagnostic-free leaves user-facing
+ * messages to the git layer.
+ *
+ * Returns 0 on success, -1 if a hunk is out of range.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j;
+
+	/*
+	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
+	 * Clear them so only the external hunks are marked.
+	 */
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		struct xdl_hunk *h = &hunks[i];
+
+		/*
+		 * Non-negative counts and 1-based starts are caller
+		 * preconditions (it normalizes hunks into xdiff coordinates
+		 * before this point), so a violation is a bug, not a bad
+		 * tool response.
+		 */
+		if (h->old_count < 0 || h->new_count < 0)
+			BUG("external hunk %"PRIuMAX": "
+				"negative count (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_count, h->new_count);
+		if (h->old_start < 1 || h->new_start < 1)
+			BUG("external hunk %"PRIuMAX": "
+				"start not 1-based (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_start, h->new_start);
+
+		/*
+		 * The caller validates ordering, overlap and lockstep
+		 * alignment (and diagnoses a bad response).  This is only a
+		 * silent in-bounds guard so the marking loop cannot index
+		 * changed[] out of range: start + count - 1 <= nrec,
+		 * rewritten to avoid overflow.  A count of 0 (pure
+		 * insert/delete) allows start == nrec + 1, the position
+		 * after the last line.  On a miss, return -1 and let the
+		 * caller fall back to the builtin diff.
+		 */
+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1 ||
+		    h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+	}
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index 11bada2608..f4ab935332 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v5 3/9] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index b5412e6bc3..7547874aa2 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v5 4/9] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and
managing a hashmap that indexes running processes by command
string.  The hashmap suits callers like convert.c where many
files may share one filter process looked up by name, but
callers that manage process lifetime through their own data
structures do not need it.

Extract subprocess_start_command() and subprocess_stop_command()
so callers can reuse the child process setup and handshake
machinery without maintaining a hashmap.  subprocess_start()
and subprocess_stop() become thin wrappers that add hashmap
operations on top.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 28 +++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 2d5c965169..3cef42b088 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
 
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
+
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err)
+		return err;
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
gitgitgadget

[PATCH v5 5/9] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference line numbers in the content.
When textconv is also configured, the tool receives the
textconv-transformed content.  The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
normally.  A new "Which features consult the diff process"
documentation section lays out which features use the tool's hunks,
which compute independently, and why; the summary formats such as
--stat still use the builtin diff for now.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, the old and new blob object
names as old-oid/new-oid, and both file contents as packetized data.
The tool responds with hunk lines and a status packet (success,
error, or abort).  On error, Git warns and falls back to the builtin
diff algorithm for that file.  On abort, Git silently falls back for
the current file and stops sending further requests to the tool for
the remainder of the session.

old-oid/new-oid name the two blobs so a tool can cache its analysis
keyed on the pair.  A side's oid is sent only when the content the
tool receives is that raw blob: it is omitted under textconv, which
rewrites the bytes, and for a working-tree side with no stored
object, so an oid that is sent always names the bytes the tool
receives.  This is where the process protocol diverges from
diff.<driver>.command, which never composes with textconv (the
command replaces the whole diff and always gets the raw blob).  Tools
ignore unknown request keys, so old tools skip them.

When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
This also means --exit-code reports no changes for that file.

The subprocess is stored on the userdiff_driver struct and
launched on first use.  If the process fails to start, the
handshake fails, or a communication error occurs mid-stream,
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.

Git falls back to the builtin diff (rather than consulting the
tool) when an option the tool cannot honor is in effect: the
whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and
--anchored.  The bypass keys off the effective diff parameters (xpp)
rather than diffopt, so a later caller whose flags live elsewhere is
covered uniformly.  A change that only adds or removes the trailing
newline is likewise not expressible as hunks, so it too uses the
builtin diff.  The hunk parser ignores unknown trailing fields on a
hunk line for response forward-compatibility.

Hunk accumulation is bounded by the combined byte count of the two
files, so a misbehaving tool that floods hunk lines cannot grow
memory without bound before validation runs.

diff_process_fill_hunks() is the sole public entry point.  It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc       |   5 +
 Documentation/gitattributes.adoc     | 249 +++++++++++
 Makefile                             |   2 +
 diff-process.c                       | 491 ++++++++++++++++++++
 diff-process.h                       |  49 ++
 diff.c                               |  21 +
 diff.h                               |   3 +
 meson.build                          |   1 +
 t/helper/meson.build                 |   1 +
 t/helper/test-diff-process-backend.c | 381 ++++++++++++++++
 t/helper/test-tool.c                 |   1 +
 t/helper/test-tool.h                 |   1 +
 t/meson.build                        |   1 +
 t/t4080-diff-process.sh              | 645 +++++++++++++++++++++++++++
 userdiff.h                           |   3 +
 15 files changed, 1854 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..ac0635bb3b 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 2c4fbfd7f1..f4ca4a8c7e 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -831,6 +831,255 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If `diff.<name>.process` is defined, Git sends the old and new file
+content to an external tool and receives back a list of changed
+regions (pairs of line ranges in the old and new file).  Git uses
+these instead of its builtin diff algorithm, but still controls
+all output formatting, so features like word diff, function context,
+color, and blame work normally.  This is achieved by using the
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
+the diff process feeds results back into the standard pipeline.  If
+both are configured for a path, `diff.<name>.command` takes precedence
+for the patch output it replaces; the summary formats, `git blame`,
+and `git log -L` never run the command and still consult the process.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+Then, define a "diff.<name>.process" configuration to specify
+the diff process command.
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+When Git encounters the first file that needs to be diffed, it starts
+the process and performs the handshake.  In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
+described below).  The tool replies with "git-diff-server", the
+version it supports, and the capabilities it supports.
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
+data, each terminated with a flush packet.  The pathname is relative
+to the repository root.  When `diff.<name>.textconv` is also set,
+the tool receives the textconv-transformed content rather than the
+raw blob.  Git does not send binary files to the diff process.
+
+-----------------------
+packet:          git> command=hunks
+packet:          git> pathname=path/file.c
+packet:          git> old-oid=<hex>
+packet:          git> new-oid=<hex>
+packet:          git> 0000
+packet:          git> OLD_CONTENT
+packet:          git> 0000
+packet:          git> NEW_CONTENT
+packet:          git> 0000
+-----------------------
+
+The optional `old-oid` and `new-oid` keys give the object names of the
+old and new blobs, so a tool can cache its analysis keyed on the pair.
+A side's key is sent only when the content for that side is the raw
+blob it names: it is omitted when the content is textconv-transformed,
+and for a working-tree side that has no stored object.  A tool that
+does not recognize these keys ignores them.
+
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
+
+  `hunk <old_start> <old_count> <new_start> <new_count>`
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
+replacement range in the new file.  The four fields are separated by
+single spaces.  Start values are 1-based and counts are non-negative.
+For example, `hunk 3 2 3 4` means that 2 lines starting at line 3 in
+the old file were replaced by 4 lines starting at line 3 in the new
+file.  An `<old_count>` of 0 means no lines were removed (pure
+insertion); a `<new_count>` of 0 means no lines were added (pure
+deletion).  For a side with a count of 0 (a pure insertion or
+deletion) the start is the 1-based line the change sits before,
+ranging from 1 to one past the last line (the line count plus 1, to
+place the change at the end of the file); like every start it must
+keep the unchanged runs aligned on both sides (see below), so for a
+given change it takes one specific value, not an arbitrary one.  A
+start of 0 is also accepted and treated as 1, matching the
+empty-file-side form `git diff` emits (e.g. `hunk 0 0 1 5` for a newly
+added file).  A nonzero range must not extend beyond the end of the
+file.  Git ignores any extra
+whitespace-separated tokens after `<new_count>`, so a future protocol
+version can append fields to a hunk line (for example a "moved"
+marker) without older tools rejecting it.
+
+Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
+Hunks must be listed in order and must not overlap.  Any line not
+covered by a hunk is treated as unchanged and is paired, in order,
+with the unchanged lines on the other side.  Each run of unchanged
+lines between two hunks (and the run before the first hunk and
+after the last) must therefore be the same length on both sides,
+not merely equal in total.  For the hunks `1 3 1 5` and `10 2 12 2`
+below, lines 4-9 of the old file and lines 6-11 of the new file are
+both the six unchanged lines between the two hunks.  A response that
+balances only the total unchanged count but misaligns one of these
+runs is rejected, and Git falls back to the builtin diff.
+
+Git does not check that the lines a hunk leaves unchanged are
+byte-for-byte identical between the two sides; it pairs them by
+position and shows the new side as context.  A tool may therefore
+report lines that differ textually (a pure reformatting, say) as
+unchanged, and the diff reflects that judgment.  This is
+the point of a semantic backend, but it means a misbehaving tool can
+produce a diff whose context does not match the old blob; as with
+`git diff -w`, such a patch may not apply against the old content.
+
+-----------------------
+packet:          git< hunk 1 3 1 5
+packet:          git< hunk 10 2 12 2
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
+as changed and feeds them into the standard diff pipeline.  Git may
+still slide or regroup those changes against matching context for
+display, exactly as it compacts its own diffs, so the tool controls
+which lines are reported as changed, not the precise hunk boundaries.
+Patch output features (word diff, function context, color) work
+normally.  Summary formats such as `--stat` still compute their counts
+with the builtin diff for now; see "Which features consult the diff
+process" below for the full picture and the reasoning behind it.
+
+If no hunk lines precede the flush, followed by "success", Git
+treats the files as having no changes: `git diff` produces no output,
+`git diff --exit-code` and `--quiet` report success even though the
+stored blobs differ, and `git blame` skips the commit, attributing
+lines to earlier commits.
+The one exception is a change that only adds or removes the file's
+trailing newline: it cannot be expressed as line hunks, so when the
+line content otherwise matches Git keeps the builtin diff for that
+file (preserving the `\ No newline at end of file` marker) instead of
+treating the two sides as equal.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool returns well-formed but invalid hunks (out of bounds,
+overlapping, or with misaligned unchanged runs), Git warns and falls
+back to the builtin diff for that file; the tool stays available for
+subsequent files.  A malformed hunk line, by contrast (bad syntax, a
+nonzero count paired with a start of 0, or more hunks than the file
+has lines), is a protocol violation: Git stops the process and does
+not send it further requests, as described below.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status.  Git warns and
+falls back to the builtin diff algorithm for this file, treating any
+status other than "success" or "abort" the same way.  The tool
+remains available for subsequent files.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=error
+packet:          git< 0000
+-----------------------
+
+In case the tool cannot or does not want to process the content as
+well as any future content for the lifetime of the Git process, it
+is expected to respond with an "abort" status.  Git silently falls
+back to the builtin diff algorithm for this file and does not send
+further requests to the tool.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=abort
+packet:          git< 0000
+-----------------------
+
+If the tool dies during the communication or does not adhere to the
+protocol then Git will stop the process and fall back to the builtin
+diff algorithm.  Git warns once and does not restart the process for
+subsequent files.
+
+Tools should ignore unknown keys in the per-file request to remain
+forward-compatible.  Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
+
+Which features consult the diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The diff process answers a single question: given two blobs, which
+line ranges differ?  Whether a particular feature consults it follows
+from whether that is the question the feature is really asking.
+
+Features that ask "which lines changed" use the tool's hunks in place
+of the builtin algorithm:
+
+- `git diff` patch output, together with everything layered on it:
+  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
+  headers, and the `-L` line-range display.  These operate on the
+  lines the patch step already emitted, so they reflect the tool's
+  hunks without any further negotiation.
+- `git blame`: a commit whose change the tool reports as equivalent is
+  skipped, and its lines are attributed to an earlier commit.
+
+Features that ask a different question do not consult the process, by
+design:
+
+- The pickaxe `-G<regex>` searches the textual diff for a pattern; it
+  asks "does this string appear in the diff," not "did these lines
+  change."  (`-S` runs at an earlier stage and is likewise unaffected.)
+- `git patch-id` must produce a stable hash for `git rebase` and
+  cherry-pick detection; deriving it from a configured tool would make
+  equal patches hash differently from machine to machine.
+- The merge machinery (`git merge-tree`, `rerere`) computes merge
+  content and conflict signatures rather than display output, so the
+  tool's hunks must not alter its results.
+- `git range-diff` diffs patch text, not source blobs, so source-file
+  hunks do not apply to it.
+- `--check` reports whitespace errors in added lines using the builtin
+  diff's notion of which lines are added, not the tool's.  It can
+  therefore flag (and exit non-zero on) a line the tool treats as
+  unchanged and that `git diff` shows as context.  Whitespace breakage
+  is a property of the literal bytes, so `--check` keeps the builtin
+  partition deliberately; a future change could wire it to the tool if
+  matching `git diff` exactly became desirable.
+- `--raw`, `--name-only`, and `--name-status` compare object ids at
+  the tree level and never run a line-level diff at all.
+
+Some features ask "which lines changed" but still use the builtin
+algorithm for now, and may consult the process in a later change: the
+summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
+commit selection and parent range propagation (as distinct from its
+display, which is covered above); and combined diffs (`--cc` and merge
+diffs), whose protocol would have to be extended from a single old/new
+pair to one comparison per merge parent.
+
+`--diff-algorithm` bypasses the process entirely, for every feature
+listed above.  The whitespace-ignoring options (`-w`,
+`--ignore-space-change`, `--ignore-blank-lines`, and the like),
+`-I<regex>`, and `--anchored` also bypass it for the affected files:
+the tool is never told about these options, so it could not honor
+them, and Git falls back to the builtin diff, which does.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index 1cec251f43..1314c10463 100644
--- a/Makefile
+++ b/Makefile
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o
 TEST_BUILTINS_OBJS += test-date.o
 TEST_BUILTINS_OBJS += test-delete-gpgsig.o
 TEST_BUILTINS_OBJS += test-delta.o
+TEST_BUILTINS_OBJS += test-diff-process-backend.o
 TEST_BUILTINS_OBJS += test-dir-iterator.o
 TEST_BUILTINS_OBJS += test-drop-caches.o
 TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..4c748fdd2a
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,491 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  The tool controls which lines are marked as changed
+ * while the display shows the file content (after any textconv
+ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / [old-oid=<hex>] / [new-oid=<hex>] / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * When the tool returns no hunks with status=success, it considers
+ * the files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
+#include "hex.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *get_or_launch_process(
+		struct userdiff_driver *drv)
+{
+	struct diff_subprocess *entry;
+
+	if (drv->diff_subprocess)
+		return drv->diff_subprocess;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start_command(&entry->subprocess, drv->process,
+				     start_diff_process_fn)) {
+		free(entry);
+		drv->diff_process_failed = 1;
+		return NULL;
+	}
+
+	drv->diff_subprocess = entry;
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret = 0;
+
+	if (size < 0)
+		return -1;
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+/*
+ * A hunk in the diff process's presentation coordinates: the line
+ * numbering it reports over the protocol.  Kept distinct from struct
+ * xdl_hunk (xdiff's coordinates) so that only translated hunks ever
+ * reach the diff algorithm; diff_process_hunk_to_xdl() is the single
+ * crossing point.
+ */
+struct diff_process_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
+/*
+ * Parse one non-negative decimal field of a hunk line into *out and
+ * advance *line past it.  Fields must be plain decimal with no leading
+ * whitespace or sign (isdigit() takes an unsigned char to stay defined
+ * for high-bit bytes).  The first three fields are followed by a single
+ * space; the last (is_last) is followed by end-of-string or a space.
+ * Trailing space-separated tokens after the last field are allowed and
+ * ignored, so a future protocol version can append fields (e.g. a
+ * "moved" marker) without older tools rejecting the line -- mirroring
+ * the request-side rule that tools ignore unknown keys.
+ */
+static int parse_hunk_field(const char **line, long *out, int is_last)
+{
+	const char *p = *line;
+	char *end;
+
+	if (!isdigit((unsigned char)*p))
+		return -1;
+	errno = 0;
+	*out = strtol(p, &end, 10);
+	if (errno || end == p)
+		return -1;
+	if (is_last) {
+		if (*end != '\0' && *end != ' ')
+			return -1;
+	} else {
+		if (*end != ' ')
+			return -1;
+		end++;
+	}
+	*line = end;
+	return 0;
+}
+
+static int parse_hunk_line(const char *line,
+			   struct diff_process_hunk *presented)
+{
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+	if (parse_hunk_field(&line, &presented->old_start, 0) ||
+	    parse_hunk_field(&line, &presented->old_count, 0) ||
+	    parse_hunk_field(&line, &presented->new_start, 0) ||
+	    parse_hunk_field(&line, &presented->new_count, 1))
+		return -1;
+	return 0;
+}
+
+/*
+ * Translate a hunk from the diff process's presentation coordinates
+ * into xdiff's.
+ *
+ * Protocol starts are already 1-based positions (the line a change
+ * sits before), the same numbering xdiff uses, so the only adjustment
+ * is for an empty file side: "git diff" addresses it with a start of 0
+ * and a count of 0 (e.g. "0 0 1 5" adds five lines to an empty old
+ * side), and since xdiff uses start-1 as an array index that 0 becomes
+ * 1 here.  This is NOT the full inverse of xdl_emit_hunk_hdr()
+ * (xdiff/xutils.c): that emitter shifts a count-0 range to start-1 for
+ * the displayed "@@" header, but the protocol keeps the unshifted
+ * 1-based position for a mid-file insert or delete.  This is the single
+ * point where presentation coordinates become xdiff coordinates, so
+ * xdl_populate_hunks_from_external() may assume 1-based starts.
+ *
+ * Returns -1 for a start of 0 paired with a nonzero count, which names
+ * no line in either coordinate system.  (parse_hunk_line() already
+ * guarantees non-negative starts and counts.)
+ */
+static int diff_process_hunk_to_xdl(const struct diff_process_hunk *presented,
+				    struct xdl_hunk *xdl)
+{
+	long old_start = presented->old_start;
+	long new_start = presented->new_start;
+
+	if ((!old_start && presented->old_count) ||
+	    (!new_start && presented->new_count))
+		return -1;
+	if (!old_start)
+		old_start = 1;
+	if (!new_start)
+		new_start = 1;
+
+	xdl->old_start = old_start;
+	xdl->old_count = presented->old_count;
+	xdl->new_start = new_start;
+	xdl->new_count = presented->new_count;
+	return 0;
+}
+
+static enum diff_process_result get_hunks(
+		struct userdiff_driver *drv,
+		const char *path,
+		const char *old_buf, long old_size,
+		const char *new_buf, long new_size,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		struct xdl_hunk **hunks_out,
+		size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct diff_process_hunk presented;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	size_t max_hunks;
+	int len;
+	char *line;
+
+	backend = get_or_launch_process(drv);
+	if (!backend)
+		return DIFF_PROCESS_ERROR;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path))
+		goto comm_error;
+	/*
+	 * old-oid/new-oid let the tool key a cache on the blob pair.  A
+	 * side is sent only when its content is the raw blob (the caller
+	 * passes NULL otherwise, e.g. for textconv'd content), so an oid
+	 * that is present always names the bytes the tool receives.
+	 */
+	if (oid_a &&
+	    packet_write_fmt_gently(fd_in, "old-oid=%s\n", oid_to_hex(oid_a)))
+		goto comm_error;
+	if (oid_b &&
+	    packet_write_fmt_gently(fd_in, "new-oid=%s\n", oid_to_hex(oid_b)))
+		goto comm_error;
+	if (packet_flush_gently(fd_in))
+		goto comm_error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto comm_error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto comm_error;
+
+	/*
+	 * Hunks are non-overlapping and each useful hunk covers at least
+	 * one line, so a valid response cannot contain more hunks than the
+	 * two files have lines, which is bounded by their byte sizes.  Cap
+	 * the accumulation accordingly so a misbehaving tool that floods
+	 * hunk lines cannot drive unbounded memory growth before validation.
+	 */
+	max_hunks = (size_t)old_size + (size_t)new_size + 1;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &presented) < 0)
+			goto comm_error;
+		if (diff_process_hunk_to_xdl(&presented, &hunk) < 0)
+			goto comm_error;
+		if (nr_hunks >= max_hunks) {
+			warning(_("diff process '%s' sent too many hunks"
+				  " for '%s'"), drv->process, path);
+			goto comm_error;
+		}
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto comm_error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		*hunks_out = hunks;
+		*nr_hunks_out = nr_hunks;
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_OK;
+	}
+
+	if (!strcmp(status.buf, "abort")) {
+		/*
+		 * The tool voluntarily withdrew: stop sending requests
+		 * but do not warn (this is not a failure).
+		 */
+		backend->supported_capabilities &= ~CAP_HUNKS;
+		free(hunks);
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_SKIP;
+	}
+
+	/* status=error or unknown status */
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+
+comm_error:
+	/*
+	 * Communication failure (broken pipe, malformed response).
+	 * Tear down the process and mark as failed so we do not
+	 * retry on every subsequent file.
+	 */
+	drv->diff_process_failed = 1;
+	drv->diff_subprocess = NULL;
+	subprocess_stop_command(&backend->subprocess);
+	free(backend);
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+/*
+ * Whether exactly one of the two blobs ends in a newline.  A change
+ * that only adds or removes the trailing newline is not expressible as
+ * line hunks, so a tool comparing lines reports the files as equal.
+ */
+static int eof_newline_differs(const mmfile_t *a, const mmfile_t *b)
+{
+	int a_nl = a->size > 0 && a->ptr[a->size - 1] == '\n';
+	int b_nl = b->size > 0 && b->ptr[b->size - 1] == '\n';
+	return a_nl != b_nl;
+}
+
+/*
+ * Number of lines in a blob, matching xdiff's record count: one per
+ * newline, plus one more if the last line has no trailing newline.
+ */
+static long count_lines(const char *buf, long size)
+{
+	long lines = 0, i;
+
+	for (i = 0; i < size; i++)
+		if (buf[i] == '\n')
+			lines++;
+	if (size > 0 && buf[size - 1] != '\n')
+		lines++;
+	return lines;
+}
+
+/*
+ * Validate the tool's hunks (already in xdiff coordinates) against the
+ * two blobs before they bypass the diff algorithm.  Each hunk must fit
+ * within its file, the hunks must be ordered and non-overlapping, and
+ * the unchanged run before each hunk (and after the last) must be the
+ * same length on both sides -- xdl_build_script() walks the two files
+ * in lockstep over unchanged lines, so a mismatched gap desynchronizes
+ * it and yields a corrupt diff even when the totals balance.  This is
+ * the git layer's job so xdiff stays diagnostic-free; on a bad response
+ * we warn and the caller falls back to the builtin diff.  Returns 0 if
+ * valid, -1 (after warning) otherwise.
+ */
+static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,
+				   long old_lines, long new_lines,
+				   const char *process, const char *path)
+{
+	size_t i;
+	long prev_old_end = 0, prev_new_end = 0;
+
+	for (i = 0; i < nr; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count > old_lines - h->old_start + 1 ||
+		    h->new_count > new_lines - h->new_start + 1) {
+			warning(_("diff process '%s' returned a hunk past the "
+				  "end of '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		if (h->old_start < prev_old_end || h->new_start < prev_new_end) {
+			warning(_("diff process '%s' returned overlapping hunks "
+				  "for '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		if (h->old_start - prev_old_end != h->new_start - prev_new_end) {
+			warning(_("diff process '%s' returned hunks that leave "
+				  "'%s' misaligned; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+	}
+	if (old_lines - prev_old_end != new_lines - prev_new_end) {
+		warning(_("diff process '%s' returned hunks that leave '%s' "
+			  "misaligned; using the builtin diff"),
+			process, path);
+		return -1;
+	}
+	return 0;
+}
+
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+	struct xdl_hunk *ext_hunks = NULL;
+	size_t nr = 0;
+	enum diff_process_result res;
+
+	if (!diffopt || !path)
+		return DIFF_PROCESS_SKIP;
+	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+		return DIFF_PROCESS_SKIP;
+	/*
+	 * Whitespace-ignoring, regex-ignore (-I) and anchored options
+	 * change which lines count as different, but the tool is never
+	 * told about them, so its hunks could not honor them.  Rather
+	 * than silently override the user's request, fall back to the
+	 * builtin diff, which does honor these flags.  Key this off xpp
+	 * (the parameters this diff actually runs with) rather than
+	 * diffopt, so a caller like blame that keeps its flags outside
+	 * diffopt is covered without a separate guard of its own.
+	 */
+	if ((xpp->flags & (XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES)) ||
+	    xpp->ignore_regex_nr || xpp->anchors_nr)
+		return DIFF_PROCESS_SKIP;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return DIFF_PROCESS_SKIP;
+	if (drv->diff_process_failed)
+		return DIFF_PROCESS_SKIP;
+
+	res = get_hunks(drv, path,
+			file_a->ptr, file_a->size,
+			file_b->ptr, file_b->size,
+			oid_a, oid_b,
+			&ext_hunks, &nr);
+	if (res == DIFF_PROCESS_OK) {
+		if (!nr) {
+			free(ext_hunks);
+			/*
+			 * Zero hunks means the tool considers the line
+			 * content identical, but it cannot express a
+			 * trailing-newline-only change.  When that is the
+			 * actual difference, fall back to the builtin diff
+			 * so the "\ No newline at end of file" marker is
+			 * preserved instead of reporting the files equal.
+			 */
+			if (eof_newline_differs(file_a, file_b))
+				return DIFF_PROCESS_SKIP;
+			return DIFF_PROCESS_EQUIVALENT;
+		}
+		if (validate_external_hunks(ext_hunks, nr,
+					    count_lines(file_a->ptr, file_a->size),
+					    count_lines(file_b->ptr, file_b->size),
+					    drv->process, path) < 0) {
+			free(ext_hunks);
+			return DIFF_PROCESS_SKIP;
+		}
+		xpp->external_hunks = ext_hunks;
+		xpp->external_hunks_nr = nr;
+		return DIFF_PROCESS_OK;
+	}
+	if (res == DIFF_PROCESS_ERROR) {
+		warning(_("diff process '%s' failed for '%s',"
+			  " falling back to builtin diff"),
+			drv->process, path);
+		return DIFF_PROCESS_ERROR;
+	}
+	return DIFF_PROCESS_SKIP;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..8d00dafe1d
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,49 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+#include "xdiff/xdiff.h"
+
+struct diff_options;
+struct object_id;
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* failed; caller falls back to builtin */
+	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
+	DIFF_PROCESS_SKIP,       /* process did not apply: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
+/*
+ * Consult the diff process configured for 'path' and populate
+ * xpp->external_hunks with the returned hunks.
+ *
+ * Handles driver lookup, flag checks (--no-ext-diff,
+ * --diff-algorithm), subprocess management, and error reporting.
+ *
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks and
+ * the blobs are not a trailing-newline-only change (files are
+ * considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
+ *
+ * oid_a/oid_b, when non-NULL, are sent to the tool as old-oid/new-oid
+ * so it can key a cache on the blob pair.  Pass NULL for a side whose
+ * content is not the raw blob (e.g. textconv'd) or whose object name is
+ * unknown, so any oid that is sent always names the bytes the tool
+ * receives.
+ */
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 2a9d0d8687..af31072858 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -4055,6 +4056,25 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		/*
+		 * Send the blob oids only for a side whose content is the
+		 * raw blob: textconv rewrites the bytes, and a working-tree
+		 * side has no stored oid, so pass NULL there rather than an
+		 * oid that would not name what the tool receives.
+		 */
+		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
+					    (textconv_one || !one->oid_valid) ? NULL : &one->oid,
+					    (textconv_two || !two->oid_valid) ? NULL : &two->oid,
+					    &xpp)
+		    == DIFF_PROCESS_EQUIVALENT) {
+			if (textconv_one)
+				free(mf1.ptr);
+			if (textconv_two)
+				free(mf2.ptr);
+			goto free_ab_and_return;
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4135,6 +4155,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(xpp.external_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/diff.h b/diff.h
index bb5cddaf34..7dc157968d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/** Disables diff.<driver>.process. */
+	unsigned no_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 3247697f74..aa532f5200 100644
--- a/meson.build
+++ b/meson.build
@@ -328,6 +328,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 3235f10ab8..6abcda4afb 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -12,6 +12,7 @@ test_tool_sources = [
   'test-date.c',
   'test-delete-gpgsig.c',
   'test-delta.c',
+  'test-diff-process-backend.c',
   'test-dir-iterator.c',
   'test-drop-caches.c',
   'test-dump-cache-tree.c',
diff --git a/t/helper/test-diff-process-backend.c b/t/helper/test-diff-process-backend.c
new file mode 100644
index 0000000000..c2ec532c4a
--- /dev/null
+++ b/t/helper/test-diff-process-backend.c
@@ -0,0 +1,381 @@
+/*
+ * Test backend for the long-running diff process protocol
+ * (see diff-process.c and Documentation/gitattributes.adoc).
+ *
+ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
+ *
+ * Implements the server side of the pkt-line handshake and a per-file
+ * response loop.  The --mode= switch selects the response shape
+ * (success, error, abort, crash, malformed hunks).
+ *
+ * Per-file request from Git:
+ *
+ *   packet:          git> command=hunks
+ *   packet:          git> pathname=<path>
+ *   packet:          git> [old-oid=<hex>]   (omitted for textconv/worktree)
+ *   packet:          git> [new-oid=<hex>]
+ *   packet:          git> 0000
+ *   packet:          git> OLD_CONTENT
+ *   packet:          git> 0000
+ *   packet:          git> NEW_CONTENT
+ *   packet:          git> 0000
+ *
+ * Response varies by --mode (default: whole-file):
+ *
+ *   whole-file   packet: git< hunk <1|0> <old_lines> <1|0> <new_lines>
+ *                (start is 0 for an empty side, matching git diff)
+ *   fixed-hunk   packet: git< hunk 5 2 5 2
+ *   no-hunks     (no hunk packets)
+ *   bad-hunk     packet: git< hunk 999 1 999 1
+ *   bad-parse    packet: git< garbage not a hunk
+ *   bad-sync     packet: git< hunk 1 2 1 1
+ *   bad-gap      packet: git< hunk 1 1 3 1
+ *   bad-start    packet: git< hunk 0 1 1 1
+ *   multi-hunk   packet: git< hunk 5 2 5 2
+ *                packet: git< hunk 9 2 9 2
+ *   insert       packet: git< hunk 3 0 3 2   (mid-file count-0 insertion)
+ *   flood        packet: git< hunk 1 1 1 1   (x100000)
+ *   overlap      packet: git< hunk 1 5 1 5
+ *                packet: git< hunk 3 2 3 2
+ *   no-cap       (omits capability=hunks during handshake)
+ *   error        (status=error instead of status=success)
+ *   abort        (status=abort instead of status=success)
+ *   crash        exit(1) before sending any response
+ *
+ * All success modes (not error/abort/crash) end with:
+ *
+ *   packet:          git< 0000
+ *   packet:          git< status=success
+ *   packet:          git< 0000
+ *
+ * Each request is logged to --log as:
+ *
+ *   command=<cmd> pathname=<path> old-oid=<hex> new-oid=<hex> old=<first line> new=<first line>
+ */
+
+#include "test-tool.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "strbuf.h"
+
+static FILE *logfile;
+
+enum mode {
+	MODE_WHOLE_FILE,
+	MODE_FIXED_HUNK,
+	MODE_NO_HUNKS,
+	MODE_BAD_HUNK,
+	MODE_BAD_PARSE,
+	MODE_BAD_SYNC,
+	MODE_BAD_GAP,
+	MODE_BAD_START,
+	MODE_MULTI_HUNK,
+	MODE_INSERT,
+	MODE_FLOOD,
+	MODE_OVERLAP,
+	MODE_NO_CAP,
+	MODE_ERROR,
+	MODE_ABORT,
+	MODE_CRASH,
+};
+
+static enum mode parse_mode(const char *s)
+{
+	if (!strcmp(s, "whole-file"))
+		return MODE_WHOLE_FILE;
+	if (!strcmp(s, "fixed-hunk"))
+		return MODE_FIXED_HUNK;
+	if (!strcmp(s, "no-hunks"))
+		return MODE_NO_HUNKS;
+	if (!strcmp(s, "bad-hunk"))
+		return MODE_BAD_HUNK;
+	if (!strcmp(s, "bad-parse"))
+		return MODE_BAD_PARSE;
+	if (!strcmp(s, "bad-sync"))
+		return MODE_BAD_SYNC;
+	if (!strcmp(s, "bad-gap"))
+		return MODE_BAD_GAP;
+	if (!strcmp(s, "bad-start"))
+		return MODE_BAD_START;
+	if (!strcmp(s, "multi-hunk"))
+		return MODE_MULTI_HUNK;
+	if (!strcmp(s, "insert"))
+		return MODE_INSERT;
+	if (!strcmp(s, "flood"))
+		return MODE_FLOOD;
+	if (!strcmp(s, "overlap"))
+		return MODE_OVERLAP;
+	if (!strcmp(s, "no-cap"))
+		return MODE_NO_CAP;
+	if (!strcmp(s, "error"))
+		return MODE_ERROR;
+	if (!strcmp(s, "abort"))
+		return MODE_ABORT;
+	if (!strcmp(s, "crash"))
+		return MODE_CRASH;
+	die("unknown --mode=%s", s);
+}
+
+/*
+ * Read "key=value" packets up to a flush, capturing "command" and
+ * "pathname".  Returns 1 if a request was read, 0 on EOF.
+ *
+ * The first packet uses the gentle variant so that a clean shutdown
+ * by Git (EOF) does not produce a spurious "the remote end hung up
+ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
+ * variant: once inside a request, truncation is a protocol violation
+ * and dying loudly is the correct response.
+ */
+static int read_request_header(char **command, char **pathname,
+			       char **old_oid, char **new_oid)
+{
+	int first = 1;
+	char *line;
+
+	*command = *pathname = *old_oid = *new_oid = NULL;
+	for (;;) {
+		const char *value;
+
+		if (first) {
+			if (packet_read_line_gently(0, NULL, &line) < 0)
+				return 0;
+			first = 0;
+		} else {
+			line = packet_read_line(0, NULL);
+		}
+		if (!line)
+			break;
+		if (skip_prefix(line, "command=", &value))
+			*command = xstrdup(value);
+		else if (skip_prefix(line, "pathname=", &value))
+			*pathname = xstrdup(value);
+		else if (skip_prefix(line, "old-oid=", &value))
+			*old_oid = xstrdup(value);
+		else if (skip_prefix(line, "new-oid=", &value))
+			*new_oid = xstrdup(value);
+	}
+	return 1;
+}
+
+static size_t count_lines(const struct strbuf *buf)
+{
+	size_t lines = 0;
+
+	for (size_t i = 0; i < buf->len; i++)
+		if (buf->buf[i] == '\n')
+			lines++;
+
+	return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
+}
+
+static void send_status(const char *status)
+{
+	packet_flush(1);
+	packet_write_fmt(1, "%s\n", status);
+	packet_flush(1);
+}
+
+static void respond(enum mode mode,
+		    const struct strbuf *old_buf,
+		    const struct strbuf *new_buf)
+{
+	switch (mode) {
+	case MODE_ERROR:
+		send_status("status=error");
+		return;
+	case MODE_ABORT:
+		send_status("status=abort");
+		return;
+	case MODE_CRASH:
+		exit(1);
+	case MODE_FIXED_HUNK:
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		break;
+	case MODE_BAD_HUNK:
+		packet_write_fmt(1, "hunk 999 1 999 1\n");
+		break;
+	case MODE_BAD_PARSE:
+		packet_write_fmt(1, "garbage not a hunk\n");
+		break;
+	case MODE_BAD_SYNC:
+		packet_write_fmt(1, "hunk 1 2 1 1\n");
+		break;
+	case MODE_BAD_GAP:
+		/*
+		 * Globally balanced (1 changed line on each side, so the
+		 * total unchanged counts match) but the gap before the
+		 * change differs between sides: old line 1 vs new line 3.
+		 * Exercises the per-gap lockstep-alignment check.
+		 */
+		packet_write_fmt(1, "hunk 1 1 3 1\n");
+		break;
+	case MODE_BAD_START:
+		/*
+		 * A start of 0 is valid only for an empty (count 0) range;
+		 * pairing it with a nonzero count names no line in either
+		 * the protocol's or xdiff's coordinates, so the translation
+		 * rejects it and git falls back to the builtin diff.
+		 */
+		packet_write_fmt(1, "hunk 0 1 1 1\n");
+		break;
+	case MODE_MULTI_HUNK:
+		/*
+		 * Two valid, non-overlapping, gap-aligned hunks.  Exercises
+		 * the accepting branch of the per-gap lockstep check with a
+		 * non-zero previous-hunk end (the realistic two-region case).
+		 */
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		packet_write_fmt(1, "hunk 9 2 9 2\n");
+		break;
+	case MODE_INSERT:
+		/*
+		 * A mid-file pure insertion (count 0 on the old side) in the
+		 * protocol's 1-based-position form: 2 lines inserted before
+		 * old line 3.  Exercises the count-0 path, which uses the
+		 * unshifted position (not git diff's "-3,0" display start).
+		 */
+		packet_write_fmt(1, "hunk 3 0 3 2\n");
+		break;
+	case MODE_FLOOD: {
+		/*
+		 * Emit far more hunks than any small file has lines, so Git
+		 * trips its accumulation cap and falls back before reading
+		 * them all.
+		 */
+		int i;
+		for (i = 0; i < 100000; i++)
+			packet_write_fmt(1, "hunk 1 1 1 1\n");
+		break;
+	}
+	case MODE_OVERLAP:
+		packet_write_fmt(1, "hunk 1 5 1 5\n");
+		packet_write_fmt(1, "hunk 3 2 3 2\n");
+		break;
+	case MODE_NO_HUNKS:
+		break;
+	case MODE_NO_CAP:
+	case MODE_WHOLE_FILE: {
+		size_t old_lines = count_lines(old_buf);
+		size_t new_lines = count_lines(new_buf);
+		/*
+		 * Match git diff output: start=0 when count=0
+		 * (empty file side), 1 otherwise.
+		 */
+		packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
+				 " %"PRIuMAX" %"PRIuMAX"\n",
+				 (uintmax_t)(old_lines ? 1 : 0),
+				 (uintmax_t)old_lines,
+				 (uintmax_t)(new_lines ? 1 : 0),
+				 (uintmax_t)new_lines);
+		break;
+	}
+	}
+	send_status("status=success");
+}
+
+static void command_loop(enum mode mode)
+{
+	for (;;) {
+		char *command = NULL, *pathname = NULL;
+		char *old_oid = NULL, *new_oid = NULL;
+		struct strbuf obuf = STRBUF_INIT;
+		struct strbuf nbuf = STRBUF_INIT;
+
+		if (!read_request_header(&command, &pathname,
+					 &old_oid, &new_oid))
+			break; /* EOF: Git closed its end */
+
+		read_packetized_to_strbuf(0, &obuf, 0);
+		read_packetized_to_strbuf(0, &nbuf, 0);
+
+		if (logfile) {
+			fprintf(logfile,
+				"command=%s pathname=%s old-oid=%s new-oid=%s"
+				" old=%.*s new=%.*s\n",
+				command ? command : "(none)",
+				pathname ? pathname : "(none)",
+				old_oid ? old_oid : "(none)",
+				new_oid ? new_oid : "(none)",
+				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
+				obuf.buf,
+				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
+				nbuf.buf);
+			fflush(logfile);
+		}
+
+		respond(mode, &obuf, &nbuf);
+
+		free(command);
+		free(pathname);
+		free(old_oid);
+		free(new_oid);
+		strbuf_release(&obuf);
+		strbuf_release(&nbuf);
+	}
+}
+
+static void handshake(enum mode mode)
+{
+	char *line;
+
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "git-diff-client"))
+		die("bad welcome: '%s'", line ? line : "(eof)");
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "version=1"))
+		die("bad version: '%s'", line ? line : "(eof)");
+	if (packet_read_line(0, NULL))
+		die("expected flush after version");
+
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Drain capabilities advertised by Git */
+	while ((line = packet_read_line(0, NULL)))
+		; /* drain */
+
+	/* Respond with our capabilities (or none for no-cap mode) */
+	if (mode != MODE_NO_CAP)
+		packet_write_fmt(1, "capability=hunks\n");
+	packet_flush(1);
+}
+
+static const char *const usage_str[] = {
+	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
+	NULL
+};
+
+int cmd__diff_process_backend(int argc, const char **argv)
+{
+	const char *mode_str = NULL, *log_path = NULL;
+	enum mode mode = MODE_WHOLE_FILE;
+	struct option options[] = {
+		OPT_STRING(0, "mode", &mode_str, "mode",
+			   "response shape (default whole-file);"
+			   " see the file header for the full list of modes"),
+		OPT_STRING(0, "log", &log_path, "path",
+			   "append per-request summary to this file"),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
+	if (argc)
+		usage_with_options(usage_str, options);
+
+	if (mode_str)
+		mode = parse_mode(mode_str);
+
+	if (log_path) {
+		logfile = fopen(log_path, "a");
+		if (!logfile)
+			die_errno("failed to open log '%s'", log_path);
+	}
+
+	handshake(mode);
+	command_loop(mode);
+
+	if (logfile && fclose(logfile))
+		die_errno("error closing log");
+	return 0;
+}
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index b71a22b43b..3c3f95269c 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = {
 	{ "date", cmd__date },
 	{ "delete-gpgsig", cmd__delete_gpgsig },
 	{ "delta", cmd__delta },
+	{ "diff-process-backend", cmd__diff_process_backend },
 	{ "dir-iterator", cmd__dir_iterator },
 	{ "drop-caches", cmd__drop_caches },
 	{ "dump-cache-tree", cmd__dump_cache_tree },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f2885b33d5..a5bb755516 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);
 int cmd__date(int argc, const char **argv);
 int cmd__delta(int argc, const char **argv);
 int cmd__delete_gpgsig(int argc, const char **argv);
+int cmd__diff_process_backend(int argc, const char **argv);
 int cmd__dir_iterator(int argc, const char **argv);
 int cmd__drop_caches(int argc, const char **argv);
 int cmd__dump_cache_tree(int argc, const char **argv);
diff --git a/t/meson.build b/t/meson.build
index 3219264fe7..6afbfa6a87 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -512,6 +512,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..3b75df082e
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,645 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+# See t/helper/test-diff-process-backend.c for the backend implementation
+# and available --mode= options.
+
+BACKEND="test-tool diff-process-backend"
+
+test_expect_success 'setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# boundary.c: 10 lines, changes at 5-6 and 9-10.
+	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+
+	# worddiff.c: single-line function, value changes 1 -> 999.
+	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+
+	# newfile.c: single-line function, value changes 42 -> 99.
+	# Used by: modified file, --exit-code, multiple drivers.
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 42; }
+	EOF
+	git add newfile.c &&
+
+	# logtest.c: single-line function for log/format-patch tests.
+	# Needs two commits so log -1 has a diff.
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+
+	# one.c/two.c: two-file pair for error/abort/startup-failure tests.
+	cat >one.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >two.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add one.c two.c &&
+
+	git commit -m "initial" &&
+
+	# Second commit for logtest.c (so log -1 has something to show).
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	# Working tree modifications (not committed).
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 99; }
+	EOF
+
+	cat >one.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+
+	cat >two.c <<-\EOF
+	int second(void) { return 20; }
+	EOF
+'
+
+#
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
+test_expect_success 'diff process hunk boundaries affect output' '
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	test_grep ! "^-OLD9" actual &&
+	test_grep ! "^-OLD10" actual &&
+	test_grep ! "^+NEW9" actual &&
+	test_grep ! "^+NEW10" actual
+'
+
+test_expect_success 'diff process accepts valid multi-hunk output' '
+	# multi-hunk reports both changed regions (5-6 and 9-10) as two
+	# gap-aligned hunks.  This exercises the accepting branch of the
+	# per-gap lockstep check (non-zero previous-hunk end) and must
+	# produce a correct two-region diff with the lines between the
+	# hunks kept as context.
+	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "^ line7" actual &&
+	test_grep "^ line8" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process accepts a mid-file count-0 insertion' '
+	# insert mode reports "hunk 3 0 3 2": a pure insertion (count 0 on
+	# the old side) in the protocol 1-based-position form.  Exercises
+	# the count-0 hunk path that the other valid-hunk modes (full
+	# replacements, equal-count modifies) never hit.  Empty stderr is
+	# the discriminator: a mishandled count-0 start would be rejected
+	# by the lockstep check and warn.
+	cat >insert.c <<-\EOF &&
+	a
+	b
+	c
+	d
+	e
+	EOF
+	git add insert.c &&
+	git commit -m "add insert.c" &&
+	cat >insert.c <<-\EOF &&
+	a
+	b
+	X
+	Y
+	c
+	d
+	e
+	EOF
+	git -c diff.cdiff.process="$BACKEND --mode=insert" \
+		diff insert.c >actual 2>stderr &&
+	test_grep "^+X" actual &&
+	test_grep "^+Y" actual &&
+	test_grep "^ c" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with modified file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- newfile.c >actual 2>stderr &&
+	test_grep "return 99" actual &&
+	test_grep "pathname=newfile.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with added file (empty old side)' '
+	cat >added.c <<-\EOF &&
+	int added(void) { return 1; }
+	EOF
+	git add added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "added" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with deleted file (empty new side)' '
+	git add added.c &&
+	git commit -m "commit added.c" &&
+	git rm added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "deleted file" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process skipped for binary files' '
+	printf "\\0binary" >binary.c &&
+	git add binary.c &&
+	git commit -m "add binary" &&
+	printf "\\0changed" >binary.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- binary.c >actual &&
+	test_grep "Binary files" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not consulted for unmatched driver' '
+	echo "not tracked by cdiff" >unmatched.txt &&
+	git add unmatched.txt &&
+	git commit -m "add unmatched.txt" &&
+
+	echo "modified" >unmatched.txt &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- unmatched.txt >actual &&
+	test_grep "modified" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'multiple drivers use separate processes' '
+	echo "*.h diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 1; }
+	EOF
+	git add multi.h &&
+	git commit -m "add multi.h" &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 2; }
+	EOF
+
+	test_when_finished "rm -f backend-c.log backend-h.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+		diff -- newfile.c multi.h >actual 2>stderr &&
+	test_grep "pathname=newfile.c" backend-c.log &&
+	test_grep "pathname=multi.h" backend-h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works alongside textconv' '
+	write_script uppercase-filter <<-\EOF &&
+	tr "a-z" "A-Z" <"$1"
+	EOF
+
+	cat >textconv.c <<-\EOF &&
+	hello world
+	EOF
+	git add textconv.c &&
+	git commit -m "add textconv.c" &&
+
+	cat >textconv.c <<-\EOF &&
+	goodbye world
+	EOF
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- textconv.c >actual 2>stderr &&
+	# The diff process receives textconv-transformed (uppercase) content.
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=HELLO WORLD" backend.log &&
+	test_grep "new=GOODBYE WORLD" backend.log &&
+	test_must_be_empty stderr
+'
+
+#
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
+test_expect_success 'diff process with --word-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --word-diff worddiff.c >actual 2>stderr &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual &&
+	test_grep "pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with git log -p' '
+	# With no-hunks mode, the tool says the files are equivalent,
+	# so log -p should show the commit but no diff content.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -1 -p -- logtest.c >actual 2>stderr &&
+	test_grep "change logtest.c" actual &&
+	test_grep ! "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process no hunks suppresses diff output' '
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add nohunks.c &&
+	git commit -m "add nohunks.c" &&
+
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff nohunks.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success 'diff process no hunks with --exit-code returns success' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code nohunks.c
+'
+
+test_expect_success 'diff process equivalent commit: --exit-code and --quiet agree' '
+	# A committed blob pair (not a worktree file) whose oids differ but
+	# the tool reports equivalent.  --exit-code and --quiet must agree
+	# with the shown diff (empty) and report success, not fall back to
+	# the byte-level "oids differ" answer.
+	cat >ecq.c <<-\EOF &&
+	alpha
+	EOF
+	git add ecq.c &&
+	git commit -m "ecq v1" &&
+	cat >ecq.c <<-\EOF &&
+	beta
+	EOF
+	git add ecq.c &&
+	git commit -m "ecq v2" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code HEAD^ HEAD -- ecq.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --quiet HEAD^ HEAD -- ecq.c
+'
+
+test_expect_success 'diff process falls back for trailing-newline-only change' '
+	test_when_finished "rm -f backend.log" &&
+	printf "a\nb\nc\n" >eofnl.c &&
+	git add eofnl.c &&
+	git commit -m "add eofnl.c" &&
+	printf "a\nb\nc" >eofnl.c &&
+	# Same lines, only the final newline removed.  The tool reports
+	# no hunks (it sees identical lines), but that change is not
+	# expressible as hunks, so git falls back to the builtin diff
+	# rather than treating the files as equivalent.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff eofnl.c >actual 2>stderr &&
+	test_grep "No newline at end of file" actual &&
+	test_grep "pathname=eofnl.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process falls back for added file (empty old side)' '
+	test_when_finished "rm -f backend.log" &&
+	printf "x\ny\nz\n" >addnl.c &&
+	git add addnl.c &&
+	# The empty old side has no trailing newline while the new side
+	# does, so the newline fallback shows the addition rather than
+	# letting no-hunks suppress the whole new file.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --cached addnl.c >actual 2>stderr &&
+	test_grep "^+x" actual &&
+	test_grep "pathname=addnl.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process with --exit-code and hunks returns failure' '
+	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+		diff --exit-code newfile.c
+'
+
+#
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
+test_expect_success 'diff process bypassed by --diff-algorithm' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
+	test_when_finished "rm -f backend.log" &&
+	printf "a\nb\nc\n" >wsbypass.c &&
+	git add wsbypass.c &&
+	git commit -m "add wsbypass.c" &&
+	printf "a\n  b  \nc\n" >wsbypass.c &&
+	# The tool is never told about these options and could not honor
+	# them, so git bypasses the process for each (covering the whole
+	# XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES mask, not just -w).
+	for opt in -w -b --ignore-space-at-eol --ignore-blank-lines
+	do
+		rm -f backend.log &&
+		git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+			diff $opt wsbypass.c >actual 2>stderr &&
+		test_path_is_missing backend.log &&
+		test_must_be_empty stderr ||
+		return 1
+	done &&
+	# -w additionally suppresses the whitespace-only change via the
+	# builtin diff that now runs.
+	git -c diff.cdiff.process="$BACKEND" diff -w wsbypass.c >actual &&
+	test_must_be_empty actual
+'
+
+#
+# Error handling and fallback.
+#
+
+test_expect_success 'diff process fallback on tool error status' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process error keeps tool available for next file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Unlike abort, error keeps the tool available: both files
+	# are sent to the tool (and both fall back).
+	test_grep "pathname=one.c" backend.log &&
+	test_grep "pathname=two.c" backend.log &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process abort disables for session' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool.
+	test_grep "pathname=one.c" backend.log &&
+	test_grep ! "pathname=two.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Crash is a communication failure, so a warning is emitted.
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process startup failure only warns once' '
+	git -c diff.cdiff.process="/nonexistent/tool" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# Sentinel prevents repeated warnings: only one, not one per file.
+	test_grep "diff process.*failed" stderr >warnings &&
+	test_line_count = 1 warnings
+'
+
+
+test_expect_success 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "hunk past the end" stderr
+'
+
+test_expect_success 'diff process fallback on mismatched unchanged totals' '
+	cat >synctest.c <<-\EOF &&
+	line1
+	line2
+	line3
+	EOF
+	git add synctest.c &&
+	git commit -m "add synctest.c" &&
+
+	cat >synctest.c <<-\EOF &&
+	line1
+	changed
+	line3
+	EOF
+
+	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
+	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
+	# The synchronization invariant fails and git falls back.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+		diff synctest.c >actual 2>stderr &&
+	test_grep "changed" actual &&
+	test_grep "misaligned" stderr
+'
+
+test_expect_success 'diff process fallback on misaligned hunk gap' '
+	# bad-gap reports hunk 1 1 3 1 on boundary.c: one changed line
+	# on each side, so the total unchanged counts match, but the
+	# unchanged run before the change differs (old line 1 vs new
+	# line 3).  A global count check would accept this and emit a
+	# corrupt diff; the per-gap lockstep check rejects it and git
+	# falls back to the builtin algorithm.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-gap" \
+		diff boundary.c >actual 2>stderr &&
+	# The builtin fallback shows both changed regions as additions
+	# (a corrupt-accepted hunk would show NEW5 only as context).
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "misaligned" stderr
+'
+
+test_expect_success 'diff process fallback on overlapping hunks' '
+	# boundary.c has 10 lines, so both hunks are in bounds
+	# but they overlap at lines 3-4, triggering the ordering check.
+	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "NEW5" actual &&
+	test_grep "overlapping hunks" stderr
+'
+
+test_expect_success 'diff process fallback on malformed hunk line' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual
+'
+
+test_expect_success 'diff process fallback on start 0 with nonzero count' '
+	# bad-start reports hunk 0 1 1 1.  A start of 0 is valid only for
+	# an empty (count 0) range, so the presentation-to-xdiff
+	# translation rejects it and git falls back to the builtin diff
+	# instead of handing xdiff an out-of-range start.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-start" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process caps a flood of hunks and falls back' '
+	# flood emits far more hunks than the file has lines.  Git must
+	# stop accumulating and fall back to the builtin diff rather than
+	# grow memory without bound.
+	git -c diff.cdiff.process="$BACKEND --mode=flood" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "too many hunks" stderr
+'
+
+test_expect_success 'diff process skipped when tool omits capability' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-cap --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Builtin diff runs: all changes appear, including lines 9-10
+	# that a tool-provided hunk would have narrowed away.
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD9" actual &&
+	# The process launched (creating the log) but was
+	# never sent a per-file request, so no hunks command is logged.
+	test_path_is_file backend.log &&
+	test_grep ! "command=hunks" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process receives old-oid and new-oid for a blob pair' '
+	test_when_finished "rm -f backend.log" &&
+	cat >oidpair.c <<-\EOF &&
+	int f(void) { return 1; }
+	EOF
+	git add oidpair.c &&
+	git commit -m "oidpair v1" &&
+	old=$(git rev-parse HEAD:oidpair.c) &&
+
+	cat >oidpair.c <<-\EOF &&
+	int f(void) { return 2; }
+	EOF
+	git add oidpair.c &&
+	git commit -m "oidpair v2" &&
+	new=$(git rev-parse HEAD:oidpair.c) &&
+
+	# Both sides are stored blobs, so their object names are sent.
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff HEAD^ HEAD -- oidpair.c >actual 2>stderr &&
+	test_grep "old-oid=$old new-oid=$new" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process omits old-oid and new-oid for textconv content' '
+	test_when_finished "rm -f backend.log" &&
+	write_script oidcat <<-\EOF &&
+	cat "$1"
+	EOF
+	cat >oidtc.c <<-\EOF &&
+	alpha
+	EOF
+	git add oidtc.c &&
+	git commit -m "oidtc v1" &&
+	cat >oidtc.c <<-\EOF &&
+	beta
+	EOF
+	git add oidtc.c &&
+	git commit -m "oidtc v2" &&
+
+	# textconv rewrites the bytes, so the raw-blob object name that
+	# would otherwise identify each side is omitted.
+	git -c diff.cdiff.textconv="./oidcat" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff HEAD^ HEAD -- oidtc.c >actual 2>stderr &&
+	test_grep "pathname=oidtc.c" backend.log &&
+	test_grep "old-oid=(none) new-oid=(none)" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_done
diff --git a/userdiff.h b/userdiff.h
index 51c26e0d41..a98eabe377 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -3,6 +3,7 @@
 
 #include "notes-cache.h"
 
+struct diff_subprocess;
 struct index_state;
 struct repository;
 
@@ -33,6 +34,8 @@ struct userdiff_driver {
 	int textconv_want_cache;
 	const char *process;
 	char *process_owned;
+	struct diff_subprocess *diff_subprocess;
+	unsigned diff_process_failed : 1;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v5 6/9] diff: bypass diff process with --no-ext-diff and in format-patch

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Make --no-ext-diff disable diff.<driver>.process in addition to
diff.<driver>.command.  Although the two mechanisms work differently
(command replaces Git's output, process feeds hunks back into the
pipeline), both invoke external tools and --no-ext-diff means
"no external tools."

Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that
sets both allow_external and no_diff_process, so a single option
controls both.  Passing --ext-diff explicitly clears
no_diff_process, so a later --ext-diff overrides an earlier
--no-ext-diff.

Disable the diff process unconditionally in format-patch so that
generated patches are always based on the builtin diff algorithm
and can be applied reliably by recipients who do not have the
external tool.

Document that --diff-algorithm also bypasses the diff process,
since it forces the builtin algorithm.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/diff-algorithm-option.adoc |  3 +++
 Documentation/diff-options.adoc          |  4 +++-
 Documentation/gitattributes.adoc         |  6 +++---
 builtin/log.c                            |  7 +++++++
 diff.c                                   | 16 ++++++++++++++--
 diff.h                                   |  5 ++++-
 t/t4080-diff-process.sh                  | 16 ++++++++++++++++
 7 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..4d7e2ec35f 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index c8242e2462..a884445211 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -833,7 +833,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f4ca4a8c7e..a03fb9deb1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -1073,9 +1073,9 @@ display, which is covered above); and combined diffs (`--cc` and merge
 diffs), whose protocol would have to be extended from a single old/new
 pair to one comparison per merge parent.
 
-`--diff-algorithm` bypasses the process entirely, for every feature
-listed above.  The whitespace-ignoring options (`-w`,
-`--ignore-space-change`, `--ignore-blank-lines`, and the like),
+`--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
+for every feature listed above.  The whitespace-ignoring options
+(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like),
 `-I<regex>`, and `--anchored` also bypass it for the affected files:
 the tool is never told about these options, so it could not honor
 them, and Git falls back to the builtin diff, which does.
diff --git a/builtin/log.c b/builtin/log.c
index d027ce1e0b..7821a61143 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * Disable diff.<driver>.process so that patches generated by
+	 * format-patch are always based on the builtin diff algorithm
+	 * and can be applied reliably.
+	 */
+	rev.diffopt.flags.no_diff_process = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff.c b/diff.c
index af31072858..a9a732629e 100644
--- a/diff.c
+++ b/diff.c
@@ -5948,6 +5948,17 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.no_diff_process = unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6278,8 +6289,9 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index 7dc157968d..ee034d240d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,7 +173,10 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
-	/** Disables diff.<driver>.process. */
+	/**
+	 * Disables diff.<driver>.process.  Set by --no-ext-diff and by
+	 * format-patch.
+	 */
 	unsigned no_diff_process;
 
 	/**
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 3b75df082e..7e71b70ab9 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -398,6 +398,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' '
 	test_path_is_missing backend.log
 '
 
+test_expect_success 'diff process bypassed by --no-ext-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --no-ext-diff worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not used by format-patch' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		format-patch -1 --stdout -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_path_is_missing backend.log
+'
+
 test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
 	test_when_finished "rm -f backend.log" &&
 	printf "a\nb\nc\n" >wsbypass.c &&
-- 
gitgitgadget

[PATCH v5 7/9] blame: consult diff process for no-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

Introduce xdi_diff_process(), a process-aware xdi_diff() that
consults the process, runs xdiff on the tool's hunks or on the
builtin algorithm when it does not apply, frees the hunks, and
reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller
can drop or skip the change.  It is the shared consult-then-diff path
for consumers that work on raw hunks: blame's pass_blame_to_parent()
uses it here, and git log -L reuses it later.  builtin_diff() keeps
consulting the process directly, because it tests for equivalence
early, before its funcname-pattern and word-diff setup, so a
reformat-only file short-circuits without that work.

Blame's -w option is not communicated to the process and it could not
honor it, so blame must fall back to the builtin diff there.  Because
blame keeps its whitespace flags in sb->xdl_opts rather than diffopt,
the process bypass keys off xpp (the flags the diff actually runs
with), which covers blame without a guard of its own.

The subprocess is long-running (one startup cost amortized across the
blame traversal), but each commit in the file's history incurs a
round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                 |  24 +++++++-
 diff-process.c          |  38 ++++++++++++
 diff-process.h          |  26 +++++++++
 t/t4080-diff-process.sh | 126 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 213 insertions(+), 1 deletion(-)

diff --git a/blame.c b/blame.c
index 126e232416..932a04c4e8 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "xdiff-interface.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -1946,6 +1948,9 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *parent, int ignore_diffs)
 {
 	mmfile_t file_p, file_o;
+	xpparam_t xpp = {0};
+	xdemitconf_t xecfg = {0};
+	xdemitcb_t ecb = {NULL};
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
 
@@ -1964,7 +1969,24 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	xpp.flags = sb->xdl_opts;
+	xecfg.hunk_func = blame_chunk_cb;
+	ecb.priv = &d;
+	/*
+	 * Consult the diff process, then attribute the resulting chunks
+	 * via blame_chunk_cb.  It bypasses the process for the whitespace-
+	 * ignoring options it cannot honor (they live in xpp.flags, which
+	 * the consultation checks), and when the process reports the blobs
+	 * equivalent it runs no diff, so blame passes this commit and looks
+	 * past it.  Look up the driver by the parent (old) path, as
+	 * builtin_diff() does with name_a, so a renamed file resolves to the
+	 * same driver across diff, blame, and line-log.  Pass no
+	 * old-oid/new-oid: blame diffs each blob pair once, so the tool gains
+	 * nothing from a per-invocation cache key.
+	 */
+	if (xdi_diff_process(&sb->revs->diffopt, parent->path,
+			     &file_p, &file_o, NULL, NULL, &xpp, &xecfg, &ecb)
+	    == DIFF_PROCESS_ERROR)
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
diff --git a/diff-process.c b/diff-process.c
index 4c748fdd2a..191b2b67b2 100644
--- a/diff-process.c
+++ b/diff-process.c
@@ -37,6 +37,7 @@
 #include "sub-process.h"
 #include "pkt-line.h"
 #include "strbuf.h"
+#include "xdiff-interface.h"
 #include "xdiff/xdiff.h"
 
 #define CAP_HUNKS (1u << 0)
@@ -489,3 +490,40 @@ enum diff_process_result diff_process_fill_hunks(
 	}
 	return DIFF_PROCESS_SKIP;
 }
+
+enum diff_process_result xdi_diff_process(
+		struct diff_options *diffopt,
+		const char *path,
+		mmfile_t *file_a,
+		mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp,
+		xdemitconf_t *xecfg,
+		xdemitcb_t *ecb)
+{
+	enum diff_process_result res;
+
+	/*
+	 * Consult the diff process, then run xdiff either constrained to
+	 * the tool's hunks or, when the process does not apply, computing
+	 * the diff itself as a fallback.  EQUIVALENT short-circuits: the
+	 * caller decides what "no change" means for it (drop the commit,
+	 * skip the file, ...), so xdiff is not run.
+	 *
+	 * A SKIP/ERROR from the process just selects the builtin path
+	 * (its warning, if any, was already emitted), so the result then
+	 * reflects whether xdiff itself succeeded, not the process.
+	 */
+	res = diff_process_fill_hunks(diffopt, path, file_a, file_b,
+				      oid_a, oid_b, xpp);
+	if (res == DIFF_PROCESS_EQUIVALENT)
+		return res;
+
+	res = xdi_diff(file_a, file_b, xpp, xecfg, ecb) < 0
+		? DIFF_PROCESS_ERROR : DIFF_PROCESS_OK;
+
+	FREE_AND_NULL(xpp->external_hunks);
+	xpp->external_hunks_nr = 0;
+	return res;
+}
diff --git a/diff-process.h b/diff-process.h
index 8d00dafe1d..5e5b514b77 100644
--- a/diff-process.h
+++ b/diff-process.h
@@ -46,4 +46,30 @@ enum diff_process_result diff_process_fill_hunks(
 		const struct object_id *oid_b,
 		xpparam_t *xpp);
 
+/*
+ * Process-aware xdi_diff(): consult the diff process for 'path', then
+ * run xdiff either constrained to the tool's hunks or computing the
+ * diff itself when the process does not apply or fails.  Frees any
+ * hunks it obtained before returning.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool
+ * reports the blobs equal, so the caller can drop or skip the change;
+ * DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and
+ * DIFF_PROCESS_ERROR if xdiff itself errored.
+ *
+ * The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as
+ * for a direct xdi_diff() call.  oid_a/oid_b are forwarded to
+ * diff_process_fill_hunks() (see there).
+ */
+enum diff_process_result xdi_diff_process(
+		struct diff_options *diffopt,
+		const char *path,
+		mmfile_t *file_a,
+		mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp,
+		xdemitconf_t *xecfg,
+		xdemitcb_t *ecb);
+
 #endif /* DIFF_PROCESS_H */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 7e71b70ab9..694c94edb2 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -658,4 +658,130 @@ test_expect_success 'diff process omits old-oid and new-oid for textconv content
 	test_must_be_empty stderr
 '
 
+#
+# Blame integration.
+#
+
+test_expect_success 'blame uses tool-provided hunks' '
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "add blame-hunk.c" &&
+	ORIG=$(git rev-parse --short HEAD) &&
+
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "change blame-hunk.c" &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+
+	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
+	# so blame should attribute lines 9-10 to the original commit
+	# even though the builtin diff would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		blame blame-hunk.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	sed -n "6p" actual >line6 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "$CHANGE" line6
+'
+
+test_expect_success 'blame skips commits with no hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void) {
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without no-hunks mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With no-hunks mode, the process considers the files equivalent
+	# and blame skips the reformat commit, attributing to the original.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		blame blame.c >with &&
+	test_grep ! "$BLAME_COMMIT" with &&
+	test_grep "$ORIG_COMMIT" with
+'
+
+test_expect_success 'blame --no-ext-diff bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		blame --no-ext-diff blame.c >actual &&
+	# Without the process, blame attributes the reformat commit normally.
+	test_grep "$BLAME_COMMIT" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame --no-ext-diff uses builtin hunks' '
+	# fixed-hunk mode would narrow blame to lines 5-6, but
+	# --no-ext-diff should bypass it and use the builtin diff.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		blame --no-ext-diff blame-hunk.c >actual &&
+	# Builtin diff attributes lines 9-10 to the change commit.
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame -w bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	printf "alpha\nbeta\ngamma\n" >blamew.c &&
+	git add blamew.c &&
+	git commit -m "add blamew.c" &&
+	orig=$(git rev-parse --short HEAD) &&
+	printf "alpha\n   beta   \ngamma\n" >blamew.c &&
+	git commit -am "reindent beta" &&
+	reindent=$(git rev-parse --short HEAD) &&
+	# blame -w must ignore the whitespace-only change and attribute
+	# beta to the original commit, not the reindent commit.  The tool
+	# is never told about -w, so blame must bypass it (not let tool
+	# hunks override -w).
+	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=backend.log" \
+		blame -w blamew.c >actual &&
+	sed -n "2p" actual >line2 &&
+	test_grep "$orig" line2 &&
+	test_grep ! "$reindent" line2 &&
+	test_path_is_missing backend.log
+'
+
 test_done
-- 
gitgitgadget

[PATCH v5 8/9] diff: consult diff process for --stat counts

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

builtin_diff() already consults a configured diff.<driver>.process: a
file the tool reports as equivalent emits no patch, and otherwise the
tool's hunks drive the output.  builtin_diffstat() ran its own xdiff
and ignored the process, so "git diff --stat" still counted a
byte-level change for a file that "git diff" showed as unchanged.

Consult diff_process_fill_hunks() before the stat xdiff, as
builtin_diff() does.  On DIFF_PROCESS_EQUIVALENT, skip the xdiff so
the file keeps its zero inserted and deleted counts and the existing
"nothing changed" pruning drops it, matching the empty patch.
Otherwise the tool's hunks, or the builtin fallback, feed the counts
through the shared xpparam_t.

Like the builtin summary path, builtin_diffstat() does not apply
textconv, so the process is consulted on the raw blob content here,
unlike builtin_diff() which sends textconv'd content.  This keeps
"git diff --stat" counting raw lines as it does today; the asymmetry
between patch output and summary counts under textconv predates this
change.  Because the content is the raw blob, the stat path sends the
blob object names to the tool (old-oid/new-oid) for any stored blob,
where the patch path omits the oid under textconv.

Move the summary formats out of the "not yet wired" group of the
"Which features consult the diff process" documentation and into the
list of features that use the tool's hunks, noting the raw,
non-textconv content they receive.  Document that the line-counting
--dirstat=lines follows these counts while the default --dirstat does
not, and that summary formats and blame (only under --textconv) differ
from patch output in whether they textconv the content the tool sees.

Add tests covering counts from the tool's hunks (--numstat,
--shortstat), an equivalent file producing no stat line, --stat
--exit-code, the raw non-textconv content the tool receives, a
multi-file mix of equivalent and changed files, and a mode-only
change.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc |  33 +++++---
 diff.c                           |  20 ++++-
 t/t4080-diff-process.sh          | 130 +++++++++++++++++++++++++++++++
 3 files changed, 172 insertions(+), 11 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index a03fb9deb1..7cdede6b21 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -874,7 +874,10 @@ a flush packet, followed by the old and new file content as packetized
 data, each terminated with a flush packet.  The pathname is relative
 to the repository root.  When `diff.<name>.textconv` is also set,
 the tool receives the textconv-transformed content rather than the
-raw blob.  Git does not send binary files to the diff process.
+raw blob, matching what the consuming feature itself diffs: patch
+output is textconv'd, the summary formats (noted below) are not, and
+`git blame` applies textconv only under `--textconv`.  Git does not
+send binary files to the diff process.
 
 -----------------------
 packet:          git> command=hunks
@@ -960,8 +963,8 @@ still slide or regroup those changes against matching context for
 display, exactly as it compacts its own diffs, so the tool controls
 which lines are reported as changed, not the precise hunk boundaries.
 Patch output features (word diff, function context, color) work
-normally.  Summary formats such as `--stat` still compute their counts
-with the builtin diff for now; see "Which features consult the diff
+normally, as do summary formats like `--stat`.  Not every feature
+consults the process, though; see "Which features consult the diff
 process" below for the full picture and the reasoning behind it.
 
 If no hunk lines precede the flush, followed by "success", Git
@@ -1040,6 +1043,17 @@ of the builtin algorithm:
   hunks without any further negotiation.
 - `git blame`: a commit whose change the tool reports as equivalent is
   skipped, and its lines are attributed to an earlier commit.
+- `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted
+  counts come from the tool's hunks, so a file the tool calls
+  equivalent contributes no stat line, matching the empty patch that
+  `git diff` produces for it.  These summary formats do not apply
+  textconv (just as the builtin summary path does not), so the tool
+  is consulted on the raw blob content even when a `textconv` is also
+  configured for patch output; this mirrors how builtin `--stat`
+  already counts raw lines rather than the textconv'd view.  The
+  line-counting `--dirstat=lines` uses these same counts; the default
+  `--dirstat`, which weighs byte changes, is computed on its own and
+  does not consult the tool.
 
 Features that ask a different question do not consult the process, by
 design:
@@ -1065,13 +1079,12 @@ design:
 - `--raw`, `--name-only`, and `--name-status` compare object ids at
   the tree level and never run a line-level diff at all.
 
-Some features ask "which lines changed" but still use the builtin
-algorithm for now, and may consult the process in a later change: the
-summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
-commit selection and parent range propagation (as distinct from its
-display, which is covered above); and combined diffs (`--cc` and merge
-diffs), whose protocol would have to be extended from a single old/new
-pair to one comparison per merge parent.
+Two cases ask "which lines changed" but still use the builtin
+algorithm, and may consult the process in a later change: `git log
+-L`'s commit selection and parent range propagation (as distinct from
+its display, which is covered above), and combined diffs (`--cc` and
+merge diffs), whose protocol would have to be extended from a single
+old/new pair to one comparison per merge parent.
 
 `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
 for every feature listed above.  The whitespace-ignoring options
diff --git a/diff.c b/diff.c
index a9a732629e..f3c8267a39 100644
--- a/diff.c
+++ b/diff.c
@@ -4269,9 +4269,27 @@ static void builtin_diffstat(const char *name_a, const char *name_b,
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_NO_HUNK_HDR;
-		if (xdi_diff_outf(&mf1, &mf2, NULL,
+		/*
+		 * Consult the diff process so --stat reflects the
+		 * tool's view of which lines changed rather than the
+		 * builtin line diff.  --stat never applies textconv, so
+		 * the tool is fed the same raw mmfiles the stat itself
+		 * diffs (unlike builtin_diff, which consults the process
+		 * on textconv'd content).
+		 * When the tool reports the files as equivalent we skip
+		 * xdiff entirely, leaving added and deleted at zero so
+		 * the file is pruned below, just as builtin_diff() emits
+		 * no patch for an equivalent file.
+		 */
+		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
+					    one->oid_valid ? &one->oid : NULL,
+					    two->oid_valid ? &two->oid : NULL,
+					    &xpp)
+		    != DIFF_PROCESS_EQUIVALENT &&
+		    xdi_diff_outf(&mf1, &mf2, NULL,
 				  diffstat_consume, diffstat, &xpp, &xecfg))
 			die("unable to generate diffstat for %s", one->path);
+		free(xpp.external_hunks);
 
 		if (DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two)) {
 			struct diffstat_file *file =
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 694c94edb2..e1c7256747 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -282,6 +282,21 @@ test_expect_success 'diff process works alongside textconv' '
 	test_must_be_empty stderr
 '
 
+test_expect_success 'diff process --stat is fed raw, not textconv, content' '
+	# Reuses textconv.c from the previous test (committed "hello
+	# world", modified to "goodbye world").  Unlike patch output,
+	# --stat does not apply textconv, so the tool sees raw lowercase
+	# content here even with a textconv configured.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --stat -- textconv.c >actual 2>stderr &&
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=hello world" backend.log &&
+	test_grep "new=goodbye world" backend.log &&
+	test_must_be_empty stderr
+'
+
 #
 # Downstream features: word diff, log, equivalent files, exit code.
 #
@@ -386,6 +401,121 @@ test_expect_success 'diff process with --exit-code and hunks returns failure' '
 		diff --exit-code newfile.c
 '
 
+test_expect_success 'diff process feeds --numstat counts' '
+	# fixed-hunk reports only lines 5-6 as changed, so the stat
+	# counts come from the tool (2/2), not the builtin diff (4/4).
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		diff --numstat boundary.c >actual 2>stderr &&
+	printf "2\t2\tboundary.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process --numstat sums multi-hunk counts' '
+	# multi-hunk reports both 2-line regions (5-6 and 9-10), so the
+	# counts add up across both hunks: 4 inserted, 4 deleted.  This
+	# exercises the two-region hunk path through builtin_diffstat.
+	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
+		diff --numstat boundary.c >actual &&
+	printf "4\t4\tboundary.c\n" >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success 'diff process equivalent files produce no --stat line' '
+	# A file the tool calls equivalent contributes no stat line,
+	# matching the empty patch that git diff produces for it.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --stat worddiff.c >actual 2>stderr &&
+	test_must_be_empty actual &&
+	test_grep "command=hunks pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process feeds --shortstat counts' '
+	# fixed-hunk reports lines 5-6 only, so the summary counts come
+	# from the tool (2 insertions, 2 deletions), not builtin (4/4).
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff --shortstat boundary.c >actual &&
+	test_grep "2 insertions" actual &&
+	test_grep "2 deletions" actual
+'
+
+test_expect_success 'diff process equivalent file makes --stat --exit-code succeed' '
+	# The tool reports worddiff.c equivalent, so --exit-code reports
+	# no change (0); the builtin diff would report a change (1).
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --stat --exit-code worddiff.c &&
+	test_expect_code 1 git diff --no-ext-diff --stat --exit-code worddiff.c
+'
+
+test_expect_success 'diff process --numstat with mixed equivalent and changed files' '
+	test_when_finished "rm -f c.log h.log" &&
+	# Self-contained fixtures: *.c uses whole-file (changed); *.mh
+	# uses no-hunks (equivalent).
+	echo "*.mh diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+	printf "int a(void) { return 1; }\n" >mixed.c &&
+	printf "int b(void) { return 1; }\n" >mixed.mh &&
+	git add mixed.c mixed.mh &&
+	git commit -m "add mixed fixtures" &&
+	printf "int a(void) { return 2; }\n" >mixed.c &&
+	printf "int b(void) { return 2; }\n" >mixed.mh &&
+	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=c.log" \
+	    -c diff.hdiff.process="$BACKEND --mode=no-hunks --log=h.log" \
+		diff --numstat mixed.c mixed.mh >actual 2>stderr &&
+	test_grep "mixed.c" actual &&
+	test_grep ! "mixed.mh" actual &&
+	test_grep "pathname=mixed.c" c.log &&
+	test_grep "pathname=mixed.mh" h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success POSIXPERM 'diff process keeps mode-only change in --stat' '
+	test_when_finished "rm -f backend.log" &&
+	cat >modeonly.c <<-\EOF &&
+	int m(void) { return 1; }
+	EOF
+	git add modeonly.c &&
+	git commit -m "add modeonly.c" &&
+	cat >modeonly.c <<-\EOF &&
+	int m(void) { return 2; }
+	EOF
+	git add modeonly.c &&
+	test_chmod +x modeonly.c &&
+	git commit -m "edit and chmod modeonly.c" &&
+	# Content and mode both changed, but no-hunks reports the content
+	# equivalent.  The tool is consulted (counts are zero, not the
+	# builtin 1/1), yet the mode change keeps the file from being
+	# pruned.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --stat HEAD^ HEAD >actual 2>stderr &&
+	test_grep "modeonly.c" actual &&
+	test_grep "command=hunks pathname=modeonly.c" backend.log &&
+	test_grep ! "1 insertion" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process not consulted for default --dirstat' '
+	# The default (change-based) --dirstat algorithm counts via its
+	# own path and never contacts the tool (here --dirstat=0 just
+	# sets a 0% threshold), so the change is still reported even
+	# though no-hunks would call it equivalent.  --dirstat=lines
+	# instead uses the process-aware stat path.
+	test_when_finished "rm -f backend.log" &&
+	mkdir -p dsub &&
+	printf "a\nb\nc\n" >dsub/d.c &&
+	git add dsub/d.c &&
+	git commit -m "add dsub/d.c" &&
+	printf "a\nB\nc\n" >dsub/d.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --dirstat=0 dsub/d.c >actual &&
+	test_grep "dsub" actual &&
+	test_path_is_missing backend.log
+'
+
 #
 # Bypass mechanisms: flags and commands that skip the diff process.
 #
-- 
gitgitgadget

[PATCH v5 9/9] line-log: consult diff process for range tracking

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

git log -L tracks line ranges by diffing each commit against its
parent in collect_diff().  This pass used the builtin diff while the
displayed diff (builtin_diff()) consults a configured
diff.<driver>.process, so the two could disagree: a reformat-only
commit selected by builtin tracking was then rendered with an empty
diff because the tool reported the files equivalent.

Consult the process in collect_diff() too, mirroring the blame
integration.  When the tool reports the files equivalent, collect no
ranges; the tracked range then maps across unchanged and the commit
drops out of the log, matching what is displayed.  Like the summary
formats, the tracking pass diffs raw content, so the tool is consulted
on the raw blobs here.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc | 20 ++++++++++---------
 line-log.c                       | 33 ++++++++++++++++++++++++++++----
 t/t4080-diff-process.sh          | 33 ++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 13 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 7cdede6b21..8021dc8e39 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -1037,12 +1037,16 @@ Features that ask "which lines changed" use the tool's hunks in place
 of the builtin algorithm:
 
 - `git diff` patch output, together with everything layered on it:
-  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
-  headers, and the `-L` line-range display.  These operate on the
-  lines the patch step already emitted, so they reflect the tool's
-  hunks without any further negotiation.
+  word diff, function context (`-W`), `--color-moved`, and the `@@`
+  hunk headers.  These operate on the lines the patch step already
+  emitted, so they reflect the tool's hunks without any further
+  negotiation.
 - `git blame`: a commit whose change the tool reports as equivalent is
   skipped, and its lines are attributed to an earlier commit.
+- `git log -L`: both the line-range display and the underlying range
+  tracking consult the tool, so a commit it reports as equivalent is
+  dropped from the log (its tracked range maps across unchanged)
+  rather than selected and then shown with an empty diff.
 - `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted
   counts come from the tool's hunks, so a file the tool calls
   equivalent contributes no stat line, matching the empty patch that
@@ -1079,11 +1083,9 @@ design:
 - `--raw`, `--name-only`, and `--name-status` compare object ids at
   the tree level and never run a line-level diff at all.
 
-Two cases ask "which lines changed" but still use the builtin
-algorithm, and may consult the process in a later change: `git log
--L`'s commit selection and parent range propagation (as distinct from
-its display, which is covered above), and combined diffs (`--cc` and
-merge diffs), whose protocol would have to be extended from a single
+Combined diffs (`--cc` and merge diffs) ask "which lines changed" but
+still use the builtin algorithm, and may consult the process in a
+later change; their protocol would have to be extended from a single
 old/new pair to one comparison per merge parent.
 
 `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
diff --git a/line-log.c b/line-log.c
index 5fc75ae275..97b3e0a31d 100644
--- a/line-log.c
+++ b/line-log.c
@@ -7,11 +7,11 @@
 #include "tag.h"
 #include "tree.h"
 #include "diff.h"
+#include "diff-process.h"
 #include "commit.h"
 #include "decorate.h"
 #include "repository.h"
 #include "revision.h"
-#include "xdiff-interface.h"
 #include "strbuf.h"
 #include "line-log.h"
 #include "setup.h"
@@ -330,12 +330,15 @@ static int collect_diff_cb(long start_a, long count_a,
 	return 0;
 }
 
-static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges *out)
+static int collect_diff(struct diff_options *diffopt, const char *path,
+			mmfile_t *parent, mmfile_t *target,
+			struct diff_ranges *out)
 {
 	struct collect_diff_cbdata cbdata = {NULL};
 	xpparam_t xpp;
 	xdemitconf_t xecfg;
 	xdemitcb_t ecb;
+	int ret = 0;
 
 	memset(&xpp, 0, sizeof(xpp));
 	memset(&xecfg, 0, sizeof(xecfg));
@@ -345,7 +348,23 @@ static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges *
 	xecfg.hunk_func = collect_diff_cb;
 	memset(&ecb, 0, sizeof(ecb));
 	ecb.priv = &cbdata;
-	return xdi_diff(parent, target, &xpp, &xecfg, &ecb);
+
+	/*
+	 * Consult the diff process so range tracking agrees with the
+	 * diff that will be shown.  When the tool reports the files as
+	 * equivalent we collect no ranges, so the tracked range maps
+	 * across unchanged and the commit drops out of the log, rather
+	 * than being selected here but rendered with an empty diff by
+	 * the process-aware builtin_diff().  Blob oids are not threaded to
+	 * this path yet, so pass NULL and send no old-oid/new-oid (a later
+	 * change can supply the pair, where they would let the tool cache
+	 * across the range-tracking and display passes over the same
+	 * commit).
+	 */
+	if (xdi_diff_process(diffopt, path, parent, target,
+			     NULL, NULL, &xpp, &xecfg, &ecb) == DIFF_PROCESS_ERROR)
+		ret = -1;
+	return ret;
 }
 
 /*
@@ -927,7 +946,13 @@ static int process_diff_filepair(struct rev_info *rev,
 	}
 
 	diff_ranges_init(&diff);
-	if (collect_diff(&file_parent, &file_target, &diff))
+	/*
+	 * Select the driver by the old (parent) path, as builtin_diff() does
+	 * with name_a, so a renamed file resolves to the same driver for
+	 * range tracking as for the diff that is shown.
+	 */
+	if (collect_diff(&rev->diffopt, pair->one->path,
+			 &file_parent, &file_target, &diff))
 		die("unable to generate diff for %s", pair->one->path);
 
 	/* NEEDSWORK should apply some heuristics to prevent mismatches */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index e1c7256747..c20ab15ecd 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -914,4 +914,37 @@ test_expect_success 'blame -w bypasses diff process' '
 	test_path_is_missing backend.log
 '
 
+#
+# Line-log (git log -L) range tracking.
+#
+
+test_expect_success 'diff process drops equivalent commit from log -L' '
+	test_when_finished "rm -f backend.log" &&
+	cat >linelog.c <<-\EOF &&
+	int tracked(void) { return 1; }
+	EOF
+	git add linelog.c &&
+	git commit -m "add linelog.c" &&
+
+	cat >linelog.c <<-\EOF &&
+	int tracked(void) { return 2; }
+	EOF
+	git commit -am "change tracked line" &&
+
+	# Builtin line tracking selects the change commit.
+	git log --no-ext-diff -L1,1:linelog.c --format="%s" >builtin &&
+	test_grep "change tracked line" builtin &&
+
+	# With the tool reporting the change as equivalent, tracking
+	# drops the commit (the range maps across unchanged) instead of
+	# selecting it and rendering an empty diff.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -L1,1:linelog.c --format="%s" >actual &&
+	test_grep ! "change tracked line" actual &&
+	# The creating commit still appears, so the change commit was
+	# selectively dropped rather than the whole log going empty.
+	test_grep "add linelog.c" actual &&
+	test_grep "command=hunks pathname=linelog.c" backend.log
+'
+
 test_done
-- 
gitgitgadget