[PATCH 00/10] Convert remaining hooks to hook.h

[PATCH 00/10] Convert remaining hooks to hook.h

[Adrian Ratiu]

Hello everyone,

This is a continuation of Emily and Aevar's work to convert remaining hooks
to the hook.h interface, by adding and using two new run-command/hook APIs:
 * feeding hook stdin via a callback
 * capturing server-side collated outputs

I've tried to keep the implementations as simple as possible and avoid any
unnecessary copying by feeding the data directly to the hook stdin fds and
even batching the writes of pre/post-receive so we achieve similar perf/data/
syscall efficiency as we had before the callback conversion.

As suggested by Aevar [1], I've removed the string_list API, the extra copies
and the $'\n' assumptions on the data, however I did not go the full zero-copy
route with mmap-ing because I think that will break backwards compatbility. We
could explore that in a future series as an efficientization of the current IPC,
this patch series basically aims for parity with the existing implementation.

This series also unblocks config-based hooks and hooks parallelization which will
follow up in a separate series.

The patch series is based on the master branch, I've pushed it to github [2] and
it also passes CI runs. [3]. Also merged and tested against next with no conflicts.

1: https://lore.kernel.org/git/230209.86y1p7y4fa.gmgdl@evledraar.gmail.com/
2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v1
3: https://github.com/10ne1/git/actions/runs/18006589297

Big warm thank you,
Adrian

Adrian Ratiu (1):
  reference-transaction: use hook.h to run hooks

Emily Shaffer (9):
  run-command: add stdin callback for parallelization
  hook: provide stdin via callback
  hook: convert 'post-rewrite' hook in sequencer.c to hook.h
  transport: convert pre-push hook to hook.h
  run-command: allow capturing of collated output
  hooks: allow callers to capture output
  receive-pack: convert 'update' hook to hook.h
  post-update: use hook.h library
  receive-pack: convert receive hooks to hook.h

 builtin/fetch.c             |   2 +-
 builtin/receive-pack.c      | 310 +++++++++++++++++++-----------------
 builtin/submodule--helper.c |   2 +-
 hook.c                      |  11 +-
 hook.h                      |  30 ++++
 refs.c                      |  61 ++++---
 run-command.c               | 115 +++++++++++--
 run-command.h               |  44 ++++-
 sequencer.c                 |  62 +++++---
 submodule.c                 |   2 +-
 t/helper/test-run-command.c |  67 +++++++-
 t/t0061-run-command.sh      |  37 +++++
 transport.c                 |  79 ++++-----
 13 files changed, 547 insertions(+), 275 deletions(-)

-- 
2.49.1

[PATCH 01/10] run-command: add stdin callback for parallelization

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

If a user of the run_processes_parallel() API wants to pipe a large
amount of information to the stdin of each parallel command, that
data could exceed the pipe buffer of the process's stdin and can be
too big to store in-memory via strbuf & friends or to slurp to a file.

Generally this is solved by repeatedly writing to child_process.in
between calls to start_command() and finish_command(). For a specific
pre-existing example of this, see transport.c:run_pre_push_hook().

This adds a generic callback API to run_processes_parallel() to do
exactly that in a unified manner, similar to the existing callback APIs,
which can then be used by hooks.h to convert the remaining hooks to the
new, simpler parallel interface.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 84 +++++++++++++++++++++++++++++++++----
 run-command.h               | 22 ++++++++++
 t/helper/test-run-command.c | 52 ++++++++++++++++++++++-
 t/t0061-run-command.sh      | 30 +++++++++++++
 4 files changed, 179 insertions(+), 9 deletions(-)

diff --git a/run-command.c b/run-command.c
index ed9575bd6a..6c455a0e43 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
+static void pp_buffer_stdin(struct parallel_processes *pp,
+			    const struct run_process_parallel_opts *opts)
+{
+	/* Buffer stdin for each pipe. */
+	for (int i = 0; i < opts->processes; i++) {
+		struct child_process *proc = &pp->children[i].process;
+		int ret;
+
+		if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)
+			continue;
+
+		/**
+		 * child input is provided via path_to_stdin when the feed_pipe cb is
+		 * missing, so we just signal an EOF.
+		 */
+		if (!opts->feed_pipe) {
+			close(proc->in);
+			proc->in = 0;
+			continue;
+		}
+
+		/**
+		 * Feed the pipe:
+		 *   ret < 0 means error
+		 *   ret == 0 means there is more data to be fed
+		 *   ret > 0 means feeding finished
+		 */
+		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
+		if (ret < 0)
+			die_errno("feed_pipe");
+
+		if (ret == 1) {
+			close(proc->in);
+			proc->in = 0;
+		}
+	}
+}
+
 static void pp_buffer_stderr(struct parallel_processes *pp,
 			     const struct run_process_parallel_opts *opts,
 			     int output_timeout)
@@ -1722,6 +1760,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		pp->children[i].state = GIT_CP_FREE;
 		if (pp->pfd)
 			pp->pfd[i].fd = -1;
+		pp->children[i].process.in = 0;
 		child_process_init(&pp->children[i].process);
 
 		if (opts->ungroup) {
@@ -1756,6 +1795,33 @@ static int pp_collect_finished(struct parallel_processes *pp,
 	return result;
 }
 
+static void pp_handle_child_IO(struct parallel_processes *pp,
+				const struct run_process_parallel_opts *opts,
+				int output_timeout)
+{
+	/*
+	 * First push input, if any (it might no-op), to child tasks to avoid them blocking
+	 * after input. This also prevents deadlocks when ungrouping below, if a child blocks
+	 * while the parent also waits for them to finish.
+	 */
+	pp_buffer_stdin(pp, opts);
+
+	if (opts->ungroup) {
+		for (size_t i = 0; i < opts->processes; i++) {
+			int child_ready_for_cleanup =
+				pp->children[i].state == GIT_CP_WORKING &&
+				pp->children[i].process.in == 0;
+
+			if (child_ready_for_cleanup)
+				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
+		}
+		return;
+	}
+
+	pp_buffer_stderr(pp, opts, output_timeout);
+	pp_output(pp);
+}
+
 void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
@@ -1775,6 +1841,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	/*
+	 * Child tasks might receive input via stdin, terminating early (or not), so
+	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
+	 * actually writes the data to children stdin fds.
+	 */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
 	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (i = 0;
@@ -1792,13 +1865,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts->ungroup) {
-			for (size_t i = 0; i < opts->processes; i++)
-				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
-		} else {
-			pp_buffer_stderr(&pp, opts, output_timeout);
-			pp_output(&pp);
-		}
+		pp_handle_child_IO(&pp, opts, output_timeout);
 		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
@@ -1809,8 +1876,11 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 
 	pp_cleanup(&pp, opts);
 
+	sigchain_pop(SIGPIPE);
+
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
+
 }
 
 int prepare_auto_maintenance(int quiet, struct child_process *maint)
diff --git a/run-command.h b/run-command.h
index 0df25e445f..4679987c8e 100644
--- a/run-command.h
+++ b/run-command.h
@@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * This callback is repeatedly called on every child process who requests
+ * start_command() to create a pipe by setting child_process.in < 0.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel, and
+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.
+ * The contents of 'send' will be read into the pipe and passed to the pipe.
+ *
+ * Returns < 0 for error
+ * Returns == 0 when there is more data to be fed (will be called again)
+ * Returns > 0 when finished (child closes fd or no more data to be fed)
+ */
+typedef int (*feed_pipe_fn)(int child_in,
+				void *pp_cb,
+				void *pp_task_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -473,6 +489,12 @@ struct run_process_parallel_opts
 	 */
 	start_failure_fn start_failure;
 
+	/*
+	 * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any
+	 * special handling.
+	 */
+	feed_pipe_fn feed_pipe;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 3719f23cc2..dfdb03b3ab 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -23,19 +23,26 @@ static int number_callbacks;
 static int parallel_next(struct child_process *cp,
 			 struct strbuf *err,
 			 void *cb,
-			 void **task_cb UNUSED)
+			 void **task_cb)
 {
 	struct child_process *d = cb;
 	if (number_callbacks >= 4)
 		return 0;
 
 	strvec_pushv(&cp->args, d->args.v);
+	cp->in = d->in;
+	cp->no_stdin = d->no_stdin;
 	if (err)
 		strbuf_addstr(err, "preloaded output of a child\n");
 	else
 		fprintf(stderr, "preloaded output of a child\n");
 
 	number_callbacks++;
+
+	/* test_stdin callback will use this to count remaining lines */
+	*task_cb = xmalloc(sizeof(int));
+	*(int*)(*task_cb) = 2;
+
 	return 1;
 }
 
@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
-			 void *pp_task_cb UNUSED)
+			 void *pp_task_cb)
 {
 	if (err)
 		strbuf_addstr(err, "asking for a quick stop\n");
 	else
 		fprintf(stderr, "asking for a quick stop\n");
+	if (pp_task_cb)
+		FREE_AND_NULL(pp_task_cb);
 	return 1;
 }
 
+static int task_finished_quiet(int result UNUSED,
+			       struct strbuf *err UNUSED,
+			       void *pp_cb UNUSED,
+			       void *pp_task_cb)
+{
+	if (pp_task_cb)
+		FREE_AND_NULL(pp_task_cb);
+	return 0;
+}
+
+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)
+{
+	int *lines_remaining = task_cb;
+
+	if (*lines_remaining) {
+		struct strbuf buf = STRBUF_INIT;
+		strbuf_addf(&buf, "sample stdin %d\n", --(*lines_remaining));
+		if (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {
+			if (errno == EPIPE) {
+				/* child closed stdin, nothing more to do */
+				strbuf_release(&buf);
+				return 1;
+			}
+			die_errno("write");
+		}
+		strbuf_release(&buf);
+	}
+
+	return !(*lines_remaining);
+}
+
 struct testsuite {
 	struct string_list tests, failed;
 	int next;
@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)
 	struct run_process_parallel_opts opts = {
 		.get_next_task = next_test,
 		.start_failure = test_failed,
+		.feed_pipe = test_stdin_pipe_feed,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
 		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		opts.get_next_task = no_job;
 		opts.task_finished = task_finished;
+	} else if (!strcmp(argv[1], "run-command-stdin")) {
+		proc.in = -1;
+		proc.no_stdin = 0;
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
+		opts.feed_pipe = test_stdin_pipe_feed;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 76d4936a87..282afecefc 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,36 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+cat >expect <<-EOF
+preloaded output of a child
+listening for stdin:
+sample stdin 1
+sample stdin 0
+preloaded output of a child
+listening for stdin:
+sample stdin 1
+sample stdin 0
+preloaded output of a child
+listening for stdin:
+sample stdin 1
+sample stdin 0
+preloaded output of a child
+listening for stdin:
+sample stdin 1
+sample stdin 0
+EOF
+
+test_expect_success 'run_command listens to stdin' '
+	write_script stdin-script <<-\EOF &&
+	echo "listening for stdin:"
+	while read line; do
+		echo "$line"
+	done
+	EOF
+	test-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&
+	test_cmp expect actual
+'
+
 cat >expect <<-EOF
 preloaded output of a child
 asking for a quick stop
-- 
2.49.1

[PATCH 02/10] hook: provide stdin via callback

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This adds a callback mechanism for feeding stdin to hooks alongside
the existing path_to_stdin (which slurps a file's content to stdin).

The advantage of this new callback is that it can feed stdin without
going through the FS layer. This helps when feeding large amount of
data and uses the run-command parallel stdin callback introduced in
the preceding commit.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 hook.c |  8 ++++++++
 hook.h | 22 ++++++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/hook.c b/hook.c
index b3de1048bf..54568d5bc0 100644
--- a/hook.c
+++ b/hook.c
@@ -69,6 +69,10 @@ static int pick_next_hook(struct child_process *cp,
 	if (hook_cb->options->path_to_stdin) {
 		cp->no_stdin = 0;
 		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
+	} else if (hook_cb->options->feed_pipe) {
+		cp->no_stdin = 0;
+		/* start_command() will allocate a pipe / stdin fd for us */
+		cp->in = -1;
 	}
 	cp->stdout_to_stderr = 1;
 	cp->trace2_hook_name = hook_cb->hook_name;
@@ -140,6 +144,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
+		.feed_pipe = options->feed_pipe,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
@@ -148,6 +153,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
 
+	if (options->path_to_stdin && options->feed_pipe)
+		BUG("choose only one method to populate hook stdin");
+
 	if (options->invoked_hook)
 		*options->invoked_hook = 0;
 
diff --git a/hook.h b/hook.h
index 11863fa734..8fdbc8c673 100644
--- a/hook.h
+++ b/hook.h
@@ -1,6 +1,7 @@
 #ifndef HOOK_H
 #define HOOK_H
 #include "strvec.h"
+#include "run-command.h"
 
 struct repository;
 
@@ -37,6 +38,24 @@ struct run_hooks_opt
 	 * Path to file which should be piped to stdin for each hook.
 	 */
 	const char *path_to_stdin;
+
+	/**
+	 * Callback to ask for more content to pipe to each hook stdin.
+	 *
+	 * If a hook needs to consume large quantities of data (e.g. a list of all refs received in a
+	 * client push), feeding data via in-memory strings or slurping to/from files via path_to_stdin
+	 * will not be efficient, so this callback allows for piecemeal reading and writing.
+	 *
+	 * Add initalization context to hook.feed_pipe_ctx.
+	 */
+	feed_pipe_fn feed_pipe;
+	void *feed_pipe_ctx;
+
+	/**
+	 * Use this to keep internal state for your feed_pipe_fn callback.
+	 * Only useful if you are using run_hooks_opt.feed_pipe. Otherwise, ignore it.
+	 */
+	void *feed_pipe_cb_data;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
@@ -44,6 +63,9 @@ struct run_hooks_opt
 	.args = STRVEC_INIT, \
 }
 
+/**
+ * Callback data provided to feed_pipe_fn.
+ */
 struct hook_cb_data {
 	/* rc reflects the cumulative failure state */
 	int rc;
-- 
2.49.1

[PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

By using 'hook.h' for 'post-rewrite', we simplify hook invocations by
not needing to put together our own 'struct child_process'.

The signal handling that's being removed by this commit now takes
place in run-command.h:run_processes_parallel(), so it is OK to remove
them here.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 sequencer.c | 62 ++++++++++++++++++++++++++++++++---------------------
 1 file changed, 38 insertions(+), 24 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 9ae40a91b2..93cd6ab1f2 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1298,32 +1298,46 @@ int update_head_with_reflog(const struct commit *old_head,
 	return ret;
 }
 
+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
+{
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
+	int ret;
+
+	if (!to_pipe || !to_pipe->len)
+		return 1; /* nothing to feed */
+
+	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);
+	if (ret < 0) {
+		if (errno == EPIPE) {
+			return 1; /* child closed pipe, nothing more to feed */
+		}
+		return ret;
+	}
+
+	/* Reset the input buffer to avoid sending it again */
+	strbuf_reset(to_pipe);
+	return ret;
+}
+
 static int run_rewrite_hook(const struct object_id *oldoid,
 			    const struct object_id *newoid)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 	int code;
 	struct strbuf sb = STRBUF_INIT;
-	const char *hook_path = find_hook(the_repository, "post-rewrite");
 
-	if (!hook_path)
-		return 0;
+	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
 
-	strvec_pushl(&proc.args, hook_path, "amend", NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "post-rewrite";
+	opt.feed_pipe_ctx = &sb;
+	opt.feed_pipe = pipe_from_strbuf;
+
+	strvec_push(&opt.args, "amend");
+
+	code = run_hooks_opt(the_repository, "post-rewrite", &opt);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
-	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
-	sigchain_push(SIGPIPE, SIG_IGN);
-	write_in_full(proc.in, sb.buf, sb.len);
-	close(proc.in);
 	strbuf_release(&sb);
-	sigchain_pop(SIGPIPE);
-	return finish_command(&proc);
+	return code;
 }
 
 void commit_post_rewrite(struct repository *r,
@@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,
 		flush_rewritten_pending();
 		if (!stat(rebase_path_rewritten_list(), &st) &&
 				st.st_size > 0) {
-			struct child_process child = CHILD_PROCESS_INIT;
+			struct child_process notes_cp = CHILD_PROCESS_INIT;
 			struct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;
 
-			child.in = open(rebase_path_rewritten_list(), O_RDONLY);
-			child.git_cmd = 1;
-			strvec_push(&child.args, "notes");
-			strvec_push(&child.args, "copy");
-			strvec_push(&child.args, "--for-rewrite=rebase");
+			notes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);
+			notes_cp.git_cmd = 1;
+			strvec_push(&notes_cp.args, "notes");
+			strvec_push(&notes_cp.args, "copy");
+			strvec_push(&notes_cp.args, "--for-rewrite=rebase");
 			/* we don't care if this copying failed */
-			run_command(&child);
+			run_command(&notes_cp);
 
 			hook_opt.path_to_stdin = rebase_path_rewritten_list();
 			strvec_push(&hook_opt.args, "rebase");
-- 
2.49.1

[PATCH 04/10] transport: convert pre-push hook to hook.h

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Move the pre-push hook away from run-command.h to and over to
the new hook.h library.

In addition to moving the signal and process handling, we use
a custom stdin feeding callback which preserves the original
behaviour of feeding stdin line by line, to avoid buffering
too much data in memory.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 transport.c | 79 ++++++++++++++++++++++-------------------------------
 1 file changed, 32 insertions(+), 47 deletions(-)

diff --git a/transport.c b/transport.c
index 6ac8aa402b..241314806f 100644
--- a/transport.c
+++ b/transport.c
@@ -1316,67 +1316,52 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
 	die(_("Aborting."));
 }
 
-static int run_pre_push_hook(struct transport *transport,
-			     struct ref *remote_refs)
+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	int ret = 0, x;
-	struct ref *r;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct strbuf buf;
-	const char *hook_path = find_hook(the_repository, "pre-push");
-
-	if (!hook_path)
-		return 0;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct ref *r = hook_cb->options->feed_pipe_ctx;
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, transport->remote->name);
-	strvec_push(&proc.args, transport->url);
+	if (r) {
+		struct strbuf buf = STRBUF_INIT;
+		int ret = 0;
+		hook_cb->options->feed_pipe_ctx = r->next;
 
-	proc.in = -1;
-	proc.trace2_hook_name = "pre-push";
+		if (!r->peer_ref) return 0;
+		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;
+		if (r->status == REF_STATUS_REJECT_STALE) return 0;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;
+		if (r->status == REF_STATUS_UPTODATE) return 0;
 
-	if (start_command(&proc)) {
-		finish_command(&proc);
-		return -1;
-	}
+		strbuf_addf(&buf, "%s %s %s %s\n",
+			    r->peer_ref->name, oid_to_hex(&r->new_oid),
+			    r->name, oid_to_hex(&r->old_oid));
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+		ret = write_in_full(hook_stdin_fd, buf.buf, buf.len);
 
-	strbuf_init(&buf, 256);
+		strbuf_release(&buf);
 
-	for (r = remote_refs; r; r = r->next) {
-		if (!r->peer_ref) continue;
-		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
-		if (r->status == REF_STATUS_REJECT_STALE) continue;
-		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
-		if (r->status == REF_STATUS_UPTODATE) continue;
+		/* We do not mind if a hook does not read all refs. */
+		if (ret < 0 && errno != EPIPE)
+			return ret;
 
-		strbuf_reset(&buf);
-		strbuf_addf( &buf, "%s %s %s %s\n",
-			 r->peer_ref->name, oid_to_hex(&r->new_oid),
-			 r->name, oid_to_hex(&r->old_oid));
-
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			/* We do not mind if a hook does not read all refs. */
-			if (errno != EPIPE)
-				ret = -1;
-			break;
-		}
+		return 0;
 	}
 
-	strbuf_release(&buf);
+	return 1; /* we ran out of refs: no more input to feed */
+}
 
-	x = close(proc.in);
-	if (!ret)
-		ret = x;
+static int run_pre_push_hook(struct transport *transport,
+			     struct ref *remote_refs)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
-	sigchain_pop(SIGPIPE);
+	strvec_push(&opt.args, transport->remote->name);
+	strvec_push(&opt.args, transport->url);
 
-	x = finish_command(&proc);
-	if (!ret)
-		ret = x;
+	opt.feed_pipe = pre_push_hook_feed_stdin;
+	opt.feed_pipe_ctx = remote_refs;
 
-	return ret;
+	return run_hooks_opt(the_repository, "pre-push", &opt);
 }
 
 int transport_push(struct repository *r,
-- 
2.49.1

[PATCH 05/10] reference-transaction: use hook.h to run hooks

[Adrian Ratiu]

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 refs.c | 61 ++++++++++++++++++++++++++++------------------------------
 1 file changed, 29 insertions(+), 32 deletions(-)

diff --git a/refs.c b/refs.c
index 4ff55cf24f..5a2b6ad1fc 100644
--- a/refs.c
+++ b/refs.c
@@ -2377,31 +2377,16 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-static int run_transaction_hook(struct ref_transaction *transaction,
-				const char *state)
+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct run_hooks_opt *opt = hook_cb->options;
+	struct ref_transaction *transaction = opt->feed_pipe_ctx;
 	struct strbuf buf = STRBUF_INIT;
-	const char *hook;
-	int ret = 0, i;
-
-	hook = find_hook(transaction->ref_store->repo, "reference-transaction");
-	if (!hook)
-		return ret;
-
-	strvec_pushl(&proc.args, hook, state, NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "reference-transaction";
-
-	ret = start_command(&proc);
-	if (ret)
-		return ret;
-
-	sigchain_push(SIGPIPE, SIG_IGN);
 
-	for (i = 0; i < transaction->nr; i++) {
+	for (int i = 0; i < transaction->nr; i++) {
 		struct ref_update *update = transaction->updates[i];
+		int ret;
 
 		if (update->flags & REF_LOG_ONLY)
 			continue;
@@ -2424,22 +2409,34 @@ static int run_transaction_hook(struct ref_transaction *transaction,
 
 		strbuf_addf(&buf, "%s\n", update->refname);
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			if (errno != EPIPE) {
-				/* Don't leak errno outside this API */
-				errno = 0;
-				ret = -1;
-			}
-			break;
+		ret = write_in_full(hook_stdin_fd, buf.buf, buf.len);
+		if (ret < 0) {
+			if (errno == EPIPE)
+				ret = 1; /* child hook closed stdin, we're done */
+
+			strbuf_release(&buf);
+			return ret; /* run-command will handle the error */
 		}
 	}
 
-	close(proc.in);
-	sigchain_pop(SIGPIPE);
 	strbuf_release(&buf);
+	return 1; /* no more input to feed */
+}
 
-	ret |= finish_command(&proc);
-	return ret;
+static int run_transaction_hook(struct ref_transaction *transaction,
+				const char *state)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+
+	if (!hook_exists(transaction->ref_store->repo, "reference-transaction"))
+		return 0;
+
+	strvec_push(&opt.args, state);
+
+	opt.feed_pipe = transaction_hook_feed_stdin;
+	opt.feed_pipe_ctx = transaction;
+
+	return run_hooks_opt(transaction->ref_store->repo, "reference-transaction", &opt);
 }
 
 int ref_transaction_prepare(struct ref_transaction *transaction,
-- 
2.49.1

[PATCH 06/10] run-command: allow capturing of collated output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some callers, for example server-side hooks which wish to relay hook
output to clients across a transport, want to capture what would
normally print to stderr and do something else with it. Allow that via a
callback.

By calling the callback regardless of whether there's output available,
we allow clients to send e.g. a keepalive if necessary.

Because we expose a strbuf, not a fd or FILE*, there's no need to create
a temporary pipe or similar - we can just skip the print to stderr and
instead hand it to the caller.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/fetch.c             |  2 +-
 builtin/submodule--helper.c |  2 +-
 hook.c                      |  2 +-
 run-command.c               | 33 ++++++++++++++++++++++++---------
 run-command.h               | 22 +++++++++++++++++++++-
 submodule.c                 |  2 +-
 t/helper/test-run-command.c | 15 +++++++++++++++
 t/t0061-run-command.sh      |  7 +++++++
 8 files changed, 71 insertions(+), 14 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 24645c4653..53bd5552c4 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -2129,7 +2129,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
 
 	if (max_children != 1 && list->nr != 1) {
 		struct parallel_fetch_state state = { argv.v, list, 0, 0, config };
-		const struct run_process_parallel_opts opts = {
+		struct run_process_parallel_opts opts = {
 			.tr2_category = "fetch",
 			.tr2_label = "parallel/fetch",
 
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 07a1935cbe..76cae9f015 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -2700,7 +2700,7 @@ static int update_submodules(struct update_data *update_data)
 {
 	int i, ret = 0;
 	struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;
-	const struct run_process_parallel_opts opts = {
+	struct run_process_parallel_opts opts = {
 		.tr2_category = "submodule",
 		.tr2_label = "parallel/update",
 
diff --git a/hook.c b/hook.c
index 54568d5bc0..199c210b97 100644
--- a/hook.c
+++ b/hook.c
@@ -135,7 +135,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	};
 	const char *const hook_path = find_hook(r, hook_name);
 	int ret = 0;
-	const struct run_process_parallel_opts opts = {
+	struct run_process_parallel_opts opts = {
 		.tr2_category = "hook",
 		.tr2_label = hook_name,
 
diff --git a/run-command.c b/run-command.c
index 6c455a0e43..c0ef771ff4 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1578,7 +1578,10 @@ static void pp_cleanup(struct parallel_processes *pp,
 	 * When get_next_task added messages to the buffer in its last
 	 * iteration, the buffered output is non empty.
 	 */
-	strbuf_write(&pp->buffered_output, stderr);
+	if (opts->consume_sideband)
+		opts->consume_sideband(&pp->buffered_output, opts->data);
+	else
+		strbuf_write(&pp->buffered_output, stderr);
 	strbuf_release(&pp->buffered_output);
 
 	sigchain_pop_common();
@@ -1717,13 +1720,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 	}
 }
 
-static void pp_output(const struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp,
+		      const struct run_process_parallel_opts *opts)
 {
 	size_t i = pp->output_owner;
 
 	if (pp->children[i].state == GIT_CP_WORKING &&
 	    pp->children[i].err.len) {
-		strbuf_write(&pp->children[i].err, stderr);
+		if (opts->consume_sideband)
+			opts->consume_sideband(&pp->children[i].err, opts->data);
+		else
+			strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
 	}
 }
@@ -1771,11 +1778,15 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		} else {
 			const size_t n = opts->processes;
 
-			strbuf_write(&pp->children[i].err, stderr);
+			/* Output errors, then all other finished child processes */
+			if (opts->consume_sideband) {
+				opts->consume_sideband(&pp->children[i].err, opts->data);
+				opts->consume_sideband(&pp->buffered_output, opts->data);
+			} else {
+				strbuf_write(&pp->children[i].err, stderr);
+				strbuf_write(&pp->buffered_output, stderr);
+			}
 			strbuf_reset(&pp->children[i].err);
-
-			/* Output all other finished child processes */
-			strbuf_write(&pp->buffered_output, stderr);
 			strbuf_reset(&pp->buffered_output);
 
 			/*
@@ -1819,10 +1830,10 @@ static void pp_handle_child_IO(struct parallel_processes *pp,
 	}
 
 	pp_buffer_stderr(pp, opts, output_timeout);
-	pp_output(pp);
+	pp_output(pp, opts);
 }
 
-void run_processes_parallel(const struct run_process_parallel_opts *opts)
+void run_processes_parallel(struct run_process_parallel_opts *opts)
 {
 	int i, code;
 	int output_timeout = 100;
@@ -1841,6 +1852,10 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	/* ungroup and reading sideband are mutualy exclusive, so disable ungroup */
+	if (opts->ungroup && opts->consume_sideband)
+		opts->ungroup = 0;
+
 	/*
 	 * Child tasks might receive input via stdin, terminating early (or not), so
 	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
diff --git a/run-command.h b/run-command.h
index 4679987c8e..ad0bab14b0 100644
--- a/run-command.h
+++ b/run-command.h
@@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * If this callback is provided, instead of collating process output to stderr,
+ * they will be collated into a new pipe. consume_sideband_fn will be called
+ * repeatedly. When output is available on that pipe, it will be contained in
+ * 'output'. But it will be called with an empty 'output' too, to allow for
+ * keepalives or similar operations if necessary.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel.
+ *
+ * Since this callback is provided with the collated output, no task cookie is
+ * provided.
+ */
+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -495,6 +509,12 @@ struct run_process_parallel_opts
 	 */
 	feed_pipe_fn feed_pipe;
 
+	/*
+	 * consume_sideband: see consume_sideband_fn() above. This can be NULL
+	 * to omit any special handling.
+	 */
+	consume_sideband_fn consume_sideband;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
@@ -529,7 +549,7 @@ struct run_process_parallel_opts
  * emitting their own output, including dealing with any race
  * conditions due to writing in parallel to stdout and stderr.
  */
-void run_processes_parallel(const struct run_process_parallel_opts *opts);
+void run_processes_parallel(struct run_process_parallel_opts *opts);
 
 /**
  * Convenience function which prepares env for a command to be run in a
diff --git a/submodule.c b/submodule.c
index fff3c75570..8ff4418e41 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1821,7 +1821,7 @@ int fetch_submodules(struct repository *r,
 {
 	int i;
 	struct submodule_parallel_fetch spf = SPF_INIT;
-	const struct run_process_parallel_opts opts = {
+	struct run_process_parallel_opts opts = {
 		.tr2_category = "submodule",
 		.tr2_label = "parallel/fetch",
 
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index dfdb03b3ab..95152a0395 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,
 	return 0;
 }
 
+static void test_consume_sideband(struct strbuf *output, void *cb UNUSED)
+{
+	FILE *sideband;
+
+	sideband = fopen("./sideband", "a");
+
+	strbuf_write(output, sideband);
+	fclose(sideband);
+}
+
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)
 		.get_next_task = next_test,
 		.start_failure = test_failed,
 		.feed_pipe = test_stdin_pipe_feed,
+		.consume_sideband = test_consume_sideband,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished_quiet;
 		opts.feed_pipe = test_stdin_pipe_feed;
+	} else if (!strcmp(argv[1], "run-command-sideband")) {
+		opts.get_next_task = parallel_next;
+		opts.consume_sideband = test_consume_sideband;
+		opts.task_finished = task_finished_quiet;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 282afecefc..38249e4798 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command can divert output' '
+	test_when_finished rm sideband &&
+	test-tool run-command run-command-sideband 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
+	test_must_be_empty actual &&
+	test_cmp expect sideband
+'
+
 cat >expect <<-EOF
 preloaded output of a child
 listening for stdin:
-- 
2.49.1

[PATCH 07/10] hooks: allow callers to capture output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some server-side hooks will require capturing output to send over
sideband instead of printing directly to stderr. Expose that capability.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c | 1 +
 hook.h | 8 ++++++++
 2 files changed, 9 insertions(+)

diff --git a/hook.c b/hook.c
index 199c210b97..e15ee08f60 100644
--- a/hook.c
+++ b/hook.c
@@ -145,6 +145,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
 		.feed_pipe = options->feed_pipe,
+		.consume_sideband = options->consume_sideband,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
diff --git a/hook.h b/hook.h
index 8fdbc8c673..a862858c1b 100644
--- a/hook.h
+++ b/hook.h
@@ -56,6 +56,14 @@ struct run_hooks_opt
 	 * Only useful if you are using run_hooks_opt.feed_pipe. Otherwise, ignore it.
 	 */
 	void *feed_pipe_cb_data;
+
+	/*
+	 * Populate this to capture output and prevent it from being printed to
+	 * stderr. This will be passed directly through to
+	 * run_command:run_parallel_processes(). See t/helper/test-run-command.c
+	 * for an example.
+	 */
+	consume_sideband_fn consume_sideband;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.49.1

[PATCH 08/10] receive-pack: convert 'update' hook to hook.h

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This makes use of the new sideband API in hook.h added in the
preceding commit.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/receive-pack.c | 60 +++++++++++++++++++++++++++++-------------
 1 file changed, 41 insertions(+), 19 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 1113137a6f..d5192ce132 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -939,31 +939,53 @@ static int run_receive_hook(struct command *commands,
 	return status;
 }
 
-static int run_update_hook(struct command *cmd)
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	int code;
-	const char *hook_path = find_hook(the_repository, "update");
+	int keepalive_active = 0;
 
-	if (!hook_path)
-		return 0;
+	if (keepalive_in_sec <= 0)
+		use_keepalive = KEEPALIVE_NEVER;
+	if (use_keepalive == KEEPALIVE_ALWAYS)
+		keepalive_active = 1;
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, cmd->ref_name);
-	strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
-	strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
+	/* send a keepalive if there is no data to write */
+	if (keepalive_active && !output->len) {
+		static const char buf[] = "0005\1";
+		write_or_die(1, buf, sizeof(buf) - 1);
+		return;
+	}
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "update";
+	if (use_keepalive == KEEPALIVE_AFTER_NUL && !keepalive_active) {
+		const char *first_null = memchr(output->buf, '\0', output->len);
+		if (first_null) {
+			/* The null bit is excluded. */
+			size_t before_null = first_null - output->buf;
+			size_t after_null = output->len - (before_null + 1);
+			keepalive_active = 1;
+			send_sideband(1, 2, output->buf, before_null, use_sideband);
+			send_sideband(1, 2, first_null + 1, after_null, use_sideband);
+
+			return;
+		}
+	}
+
+	send_sideband(1, 2, output->buf, output->len, use_sideband);
+}
+
+static int run_update_hook(struct command *cmd)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+
+	strvec_pushl(&opt.args,
+		     cmd->ref_name,
+		     oid_to_hex(&cmd->old_oid),
+		     oid_to_hex(&cmd->new_oid),
+		     NULL);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
 	if (use_sideband)
-		copy_to_sideband(proc.err, -1, NULL);
-	return finish_command(&proc);
+		opt.consume_sideband = hook_output_to_sideband;
+
+	return run_hooks_opt(the_repository, "update", &opt);
 }
 
 static struct command *find_command_by_refname(struct command *list,
-- 
2.49.1

[PATCH 09/10] post-update: use hook.h library

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/receive-pack.c | 25 ++++++-------------------
 1 file changed, 6 insertions(+), 19 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index d5192ce132..78d4df349e 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1662,33 +1662,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 static void run_update_post_hook(struct command *commands)
 {
 	struct command *cmd;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	const char *hook;
-
-	hook = find_hook(the_repository, "post-update");
-	if (!hook)
-		return;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		if (cmd->error_string || cmd->did_not_exist)
 			continue;
-		if (!proc.args.nr)
-			strvec_push(&proc.args, hook);
-		strvec_push(&proc.args, cmd->ref_name);
+		strvec_push(&opt.args, cmd->ref_name);
 	}
-	if (!proc.args.nr)
+	if (!opt.args.nr)
 		return;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "post-update";
+	if (use_sideband)
+		opt.consume_sideband = hook_output_to_sideband;
 
-	if (!start_command(&proc)) {
-		if (use_sideband)
-			copy_to_sideband(proc.err, -1, NULL);
-		finish_command(&proc);
-	}
+	run_hooks_opt(the_repository, "post-update", &opt);
 }
 
 static void check_aliased_update_internal(struct command *cmd,
-- 
2.49.1

[PATCH 10/10] receive-pack: convert receive hooks to hook.h

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 243 +++++++++++++++++++++--------------------
 1 file changed, 122 insertions(+), 121 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 78d4df349e..5e5ce34371 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)
 	return retval;
 }
 
-static void prepare_push_cert_sha1(struct child_process *proc)
+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)
 {
 	static int already_done;
 
@@ -775,168 +775,126 @@ static void prepare_push_cert_sha1(struct child_process *proc)
 		nonce_status = check_nonce(sigcheck.payload);
 	}
 	if (!is_null_oid(&push_cert_oid)) {
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT=%s",
 			     oid_to_hex(&push_cert_oid));
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_SIGNER=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_SIGNER=%s",
 			     sigcheck.signer ? sigcheck.signer : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_KEY=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_KEY=%s",
 			     sigcheck.key ? sigcheck.key : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_STATUS=%c",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_STATUS=%c",
 			     sigcheck.result);
 		if (push_cert_nonce) {
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE=%s",
 				     push_cert_nonce);
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE_STATUS=%s",
 				     nonce_status);
 			if (nonce_status == NONCE_SLOP)
-				strvec_pushf(&proc->env,
+				strvec_pushf(&opt->env,
 					     "GIT_PUSH_CERT_NONCE_SLOP=%ld",
 					     nonce_stamp_slop);
 		}
 	}
 }
 
+struct receive_hook_feed_context {
+	struct command *cmd;
+	int skip_broken;
+};
+
 struct receive_hook_feed_state {
 	struct command *cmd;
 	struct ref_push_report *report;
 	int skip_broken;
 	struct strbuf buf;
-	const struct string_list *push_options;
 };
 
-typedef int (*feed_fn)(void *, const char **, size_t *);
-static int run_and_feed_hook(const char *hook_name, feed_fn feed,
-			     struct receive_hook_feed_state *feed_state)
+static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct async muxer;
-	int code;
-	const char *hook_path = find_hook(the_repository, hook_name);
+	struct command *cmd = state->cmd;
 
-	if (!hook_path)
-		return 0;
+	strbuf_reset(&state->buf);
 
-	strvec_push(&proc.args, hook_path);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = hook_name;
-
-	if (feed_state->push_options) {
-		size_t i;
-		for (i = 0; i < feed_state->push_options->nr; i++)
-			strvec_pushf(&proc.env,
-				     "GIT_PUSH_OPTION_%"PRIuMAX"=%s",
-				     (uintmax_t)i,
-				     feed_state->push_options->items[i].string);
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
-			     (uintmax_t)feed_state->push_options->nr);
-	} else
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
+	/* batch lines to avoid going through run-command's ppoll for each line */
+	for (int i = 0; i < lines_batch_size; i++) {
+		while (cmd &&
+		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
+			cmd = cmd->next;
 
-	if (tmp_objdir)
-		strvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));
+		if (!cmd)
+			break;  /* no more commands left */
 
-	if (use_sideband) {
-		memset(&muxer, 0, sizeof(muxer));
-		muxer.proc = copy_to_sideband;
-		muxer.in = -1;
-		code = start_async(&muxer);
-		if (code)
-			return code;
-		proc.err = muxer.in;
-	}
+		if (!state->report)
+			state->report = cmd->report;
 
-	prepare_push_cert_sha1(&proc);
+		if (state->report) {
+			struct object_id *old_oid;
+			struct object_id *new_oid;
+			const char *ref_name;
 
-	code = start_command(&proc);
-	if (code) {
-		if (use_sideband)
-			finish_async(&muxer);
-		return code;
-	}
+			old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
+			new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
+			ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(old_oid), oid_to_hex(new_oid),
+				    ref_name);
 
-	while (1) {
-		const char *buf;
-		size_t n;
-		if (feed(feed_state, &buf, &n))
-			break;
-		if (write_in_full(proc.in, buf, n) < 0)
-			break;
+			state->report = state->report->next;
+			if (!state->report)
+				cmd = cmd->next;
+		} else {
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
+				    cmd->ref_name);
+			cmd = cmd->next;
+		}
 	}
-	close(proc.in);
-	if (use_sideband)
-		finish_async(&muxer);
 
-	sigchain_pop(SIGPIPE);
+	state->cmd = cmd;
+
+	if (state->buf.len > 0) {
+		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
+		if (ret < 0) {
+			if (errno == EPIPE)
+				return 1; /* child closed pipe */
+			return ret;
+		}
+	}
 
-	return finish_command(&proc);
+	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
 }
 
-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	struct receive_hook_feed_state *state = state_;
-	struct command *cmd = state->cmd;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;
 
-	while (cmd &&
-	       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
-		cmd = cmd->next;
-	if (!cmd)
-		return -1; /* EOF */
-	if (!bufp)
-		return 0; /* OK, can feed something. */
-	strbuf_reset(&state->buf);
-	if (!state->report)
-		state->report = cmd->report;
-	if (state->report) {
-		struct object_id *old_oid;
-		struct object_id *new_oid;
-		const char *ref_name;
-
-		old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
-		new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
-		ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(old_oid), oid_to_hex(new_oid),
-			    ref_name);
-		state->report = state->report->next;
-		if (!state->report)
-			state->cmd = cmd->next;
-	} else {
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
-			    cmd->ref_name);
-		state->cmd = cmd->next;
-	}
-	if (bufp) {
-		*bufp = state->buf.buf;
-		*sizep = state->buf.len;
+	/* first-time setup */
+	if (!hook_cb->options->feed_pipe_cb_data) {
+		struct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;
+		if (!ctx)
+			BUG("run_hooks_opt.feed_pipe_ctx required for receive hook");
+
+		hook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));
+		feed_state = hook_cb->options->feed_pipe_cb_data;
+		strbuf_init(&feed_state->buf, 0);
+		feed_state->cmd = ctx->cmd;
+		feed_state->skip_broken = ctx->skip_broken;
+		feed_state->report = NULL;
 	}
-	return 0;
-}
 
-static int run_receive_hook(struct command *commands,
-			    const char *hook_name,
-			    int skip_broken,
-			    const struct string_list *push_options)
-{
-	struct receive_hook_feed_state state;
-	int status;
+	/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */
+	if (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)
+		return 0; /* still have more data to feed */
 
-	strbuf_init(&state.buf, 0);
-	state.cmd = commands;
-	state.skip_broken = skip_broken;
-	state.report = NULL;
-	if (feed_receive_hook(&state, NULL, NULL))
-		return 0;
-	state.cmd = commands;
-	state.push_options = push_options;
-	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
-	strbuf_release(&state.buf);
-	return status;
+	strbuf_release(&feed_state->buf);
+
+	if (hook_cb->options->feed_pipe_cb_data)
+		FREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);
+
+	return 1; /* done feeding, run-command can close pipe */
 }
 
 static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
@@ -972,6 +930,49 @@ static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
 	send_sideband(1, 2, output->buf, output->len, use_sideband);
 }
 
+static int run_receive_hook(struct command *commands,
+			    const char *hook_name,
+			    int skip_broken,
+			    const struct string_list *push_options)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct receive_hook_feed_context ctx;
+	struct command *iter = commands;
+
+	/* if there are no valid commands, don't invoke the hook at all. */
+	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
+		iter = iter->next;
+	if (!iter)
+		return 0;
+
+	if (push_options) {
+		int i;
+		for (i = 0; i < push_options->nr; i++)
+			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
+				     push_options->items[i].string);
+		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
+					     (uintmax_t)push_options->nr);
+	} else
+		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+
+	if (tmp_objdir)
+		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+
+	prepare_push_cert_sha1(&opt);
+
+	/* set up sideband printer */
+	if (use_sideband)
+		opt.consume_sideband = hook_output_to_sideband;
+
+	/* set up stdin callback */
+	ctx.cmd = commands;
+	ctx.skip_broken = skip_broken;
+	opt.feed_pipe = feed_receive_hook_cb;
+	opt.feed_pipe_ctx = &ctx;
+
+	return run_hooks_opt(the_repository, hook_name, &opt);
+}
+
 static int run_update_hook(struct command *cmd)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
-- 
2.49.1

Re: [PATCH 04/10] transport: convert pre-push hook to hook.h

[D. Ben Knoble]

On Thu, Sep 25, 2025 at 8:54 AM Adrian Ratiu <adrian.ratiu@collabora.com> wrote:
>
> From: Emily Shaffer <emilyshaffer@google.com>
>
> Move the pre-push hook away from run-command.h to and over to
> the new hook.h library.

Perhaps s/to and// ?

Re: [PATCH 02/10] hook: provide stdin via callback

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> @@ -69,6 +69,10 @@ static int pick_next_hook(struct child_process *cp,
>  	if (hook_cb->options->path_to_stdin) {
>  		cp->no_stdin = 0;
>  		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
> +	} else if (hook_cb->options->feed_pipe) {
> +		cp->no_stdin = 0;
> +		/* start_command() will allocate a pipe / stdin fd for us */
> +		cp->in = -1;
>  	}
>  	cp->stdout_to_stderr = 1;
>  	cp->trace2_hook_name = hook_cb->hook_name;

OK, so when feed_pipe is defined, just like when path_to_stdin is
specified, we stop saying there is nothing coming from the standard
input, and intead set cp->in so that the child process would read
from there.  Unlike path_to_stdin case it is not pointing at a file
descriptor that is opened for a filesystem entity.  ".in = -1" is a
standard signal to run-command.[ch] machinery that a pipe to that
child is to be prepared.

> @@ -37,6 +38,24 @@ struct run_hooks_opt
>  	 * Path to file which should be piped to stdin for each hook.
>  	 */
>  	const char *path_to_stdin;
> +
> +	/**
> +	 * Callback to ask for more content to pipe to each hook stdin.
> +	 *
> +	 * If a hook needs to consume large quantities of data (e.g. a list of all refs received in a
> +	 * client push), feeding data via in-memory strings or slurping to/from files via path_to_stdin
> +	 * will not be efficient, so this callback allows for piecemeal reading and writing.
> +	 *
> +	 * Add initalization context to hook.feed_pipe_ctx.
> +	 */
> +	feed_pipe_fn feed_pipe;
> +	void *feed_pipe_ctx;

The comment for the member is a bit too wide.  More importantly,
this does not seem to capture the fact that this is completely
ignored when path_to_stdin is already in effect.  We should at least
document it if we wanted to leave the behaviour as is, but I wonder
if we want to detect and flag it as BUG() if both feed_pipe and
path_to_stdin are not NULL.  There is no inherent reason why the
data prepared in a file must take precedence over data coming over a
pipe.

Thanks.

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> From: Emily Shaffer <emilyshaffer@google.com>
>
> By using 'hook.h' for 'post-rewrite', we simplify hook invocations by

This "By using 'hook.h" is somewhat a strange thing to say.
<hook.h> has been in use by the file (evidenced by the fact that
there is no new "#include <hook.h>" in the patch).  I haven't
carefully read other steps in this series, but from my quick
skimming of them, I got an impression that this comment may apply
equally to other steps as well.

What the commit does is not "use hook.h"; it is to replace a custom
run-command call with a call to run_hooks_opt().  The shared API
service function may happen to be declared in <hook.h>, that that is
secondary piece of information.

> not needing to put together our own 'struct child_process'.

Imperative?  I think just dropping "we" would be sufficient.

> The signal handling that's being removed by this commit now takes
> place in run-command.h:run_processes_parallel(), so it is OK to remove
> them here.

Phrase it more positively, instead of "it is OK" (which sounds like
it is also OK to leave it there).  Perhaps say something like:

    Another benefit we gain from using run_hook_opt() instead of a
    custom start_command()/finish_command() invocations is that the
    hook API handles with sigpipe itself, so we no longer need to
    toggle signals ourselves.

or something like that, perhaps.

Thanks.

Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
> ---
>  refs.c | 61 ++++++++++++++++++++++++++++------------------------------
>  1 file changed, 29 insertions(+), 32 deletions(-)

Please describe what is done, why, and what benefit we are reaping,
just like you did for the previous few steps.

Re: [PATCH 06/10] run-command: allow capturing of collated output

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> From: Emily Shaffer <emilyshaffer@google.com>
>
> Some callers, for example server-side hooks which wish to relay hook
> output to clients across a transport, want to capture what would
> normally print to stderr and do something else with it. Allow that via a
> callback.
>
> By calling the callback regardless of whether there's output available,
> we allow clients to send e.g. a keepalive if necessary.
>
> Because we expose a strbuf, not a fd or FILE*, there's no need to create
> a temporary pipe or similar - we can just skip the print to stderr and
> instead hand it to the caller.
>
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
> ---
>  builtin/fetch.c             |  2 +-
>  builtin/submodule--helper.c |  2 +-
>  hook.c                      |  2 +-
>  run-command.c               | 33 ++++++++++++++++++++++++---------
>  run-command.h               | 22 +++++++++++++++++++++-
>  submodule.c                 |  2 +-
>  t/helper/test-run-command.c | 15 +++++++++++++++
>  t/t0061-run-command.sh      |  7 +++++++
>  8 files changed, 71 insertions(+), 14 deletions(-)
>
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 24645c4653..53bd5552c4 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -2129,7 +2129,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
>  
>  	if (max_children != 1 && list->nr != 1) {
>  		struct parallel_fetch_state state = { argv.v, list, 0, 0, config };
> -		const struct run_process_parallel_opts opts = {
> +		struct run_process_parallel_opts opts = {
>  			.tr2_category = "fetch",
>  			.tr2_label = "parallel/fetch",

This ...

> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index 07a1935cbe..76cae9f015 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -2700,7 +2700,7 @@ static int update_submodules(struct update_data *update_data)
>  {
>  	int i, ret = 0;
>  	struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;
> -	const struct run_process_parallel_opts opts = {
> +	struct run_process_parallel_opts opts = {
>  		.tr2_category = "submodule",
>  		.tr2_label = "parallel/update",

... and this ...

>  
> diff --git a/hook.c b/hook.c
> index 54568d5bc0..199c210b97 100644
> --- a/hook.c
> +++ b/hook.c
> @@ -135,7 +135,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
>  	};
>  	const char *const hook_path = find_hook(r, hook_name);
>  	int ret = 0;
> -	const struct run_process_parallel_opts opts = {
> +	struct run_process_parallel_opts opts = {
>  		.tr2_category = "hook",
>  		.tr2_label = hook_name,

... and this are curious changes that are not explained in the
proposed log message.

> @@ -1841,6 +1852,10 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  					   "max:%"PRIuMAX,
>  					   (uintmax_t)opts->processes);
>  
> +	/* ungroup and reading sideband are mutualy exclusive, so disable ungroup */
> +	If (opts->ungroup && opts->consume_sideband)
> +		opts->ungroup = 0;

Make it a BUG(""), which may help avoid unintended bugs, especially ...

> diff --git a/run-command.h b/run-command.h
> index 4679987c8e..ad0bab14b0 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,
>  				void *pp_cb,
>  				void *pp_task_cb);
>  
> +/**
> + * If this callback is provided, instead of collating process output to stderr,
> + * they will be collated into a new pipe. consume_sideband_fn will be called
> + * repeatedly. When output is available on that pipe, it will be contained in
> + * 'output'. But it will be called with an empty 'output' too, to allow for
> + * keepalives or similar operations if necessary.
> + *
> + * pp_cb is the callback cookie as passed into run_processes_parallel.
> + *
> + * Since this callback is provided with the collated output, no task cookie is
> + * provided.
> + */
> +typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);

> +
>  /**
>   * This callback is called on every child process that finished processing.
>   *
> @@ -495,6 +509,12 @@ struct run_process_parallel_opts
>  	 */
>  	feed_pipe_fn feed_pipe;
>  
> +	/*
> +	 * consume_sideband: see consume_sideband_fn() above. This can be NULL
> +	 * to omit any special handling.
> +	 */
> +	consume_sideband_fn consume_sideband;

... because which one between this and ungroup gets precedence.
Document that they are mutually exclusive, and help the callers
with a BUG("") message when both are set.

> @@ -529,7 +549,7 @@ struct run_process_parallel_opts
>   * emitting their own output, including dealing with any race
>   * conditions due to writing in parallel to stdout and stderr.
>   */
> -void run_processes_parallel(const struct run_process_parallel_opts *opts);
> +void run_processes_parallel(struct run_process_parallel_opts *opts);

This is the same unexplained curiousity I touched earlier.

Thanks.

Re: [PATCH 08/10] receive-pack: convert 'update' hook to hook.h

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> From: Emily Shaffer <emilyshaffer@google.com>
>
> This makes use of the new sideband API in hook.h added in the
> preceding commit.

... to achieve what?

Re: [PATCH 02/10] hook: provide stdin via callback

[Adrian Ratiu]

On Thu, 25 Sep 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Adrian Ratiu <adrian.ratiu@collabora.com> writes: 
> 
>> @@ -69,6 +69,10 @@ static int pick_next_hook(struct 
>> child_process *cp, 
>>  	if (hook_cb->options->path_to_stdin) { cp->no_stdin = 0; 
>>  cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY); 
>> +	} else if (hook_cb->options->feed_pipe) { + 
>> cp->no_stdin = 0; +		/* start_command() will allocate a 
>> pipe / stdin fd for us */ +		cp->in = -1; 
>>  	} cp->stdout_to_stderr = 1; cp->trace2_hook_name = 
>>  hook_cb->hook_name; 
> 
> OK, so when feed_pipe is defined, just like when path_to_stdin 
> is specified, we stop saying there is nothing coming from the 
> standard input, and intead set cp->in so that the child process 
> would read from there.  Unlike path_to_stdin case it is not 
> pointing at a file descriptor that is opened for a filesystem 
> entity.  ".in = -1" is a standard signal to run-command.[ch] 
> machinery that a pipe to that child is to be prepared. 

Yes, that is all correct.

 
>> @@ -37,6 +38,24 @@ struct run_hooks_opt 
>>  	 * Path to file which should be piped to stdin for each 
>>  hook.  */ const char *path_to_stdin; 
>> + +	/** +	 * Callback to ask for more content to pipe to 
>> each hook stdin.  +	 * +	 * If a hook needs to consume 
>> large quantities of data (e.g. a list of all refs received in a 
>> +	 * client push), feeding data via in-memory strings or 
>> slurping to/from files via path_to_stdin +	 * will not be 
>> efficient, so this callback allows for piecemeal reading and 
>> writing.  +	 * +	 * Add initalization context to 
>> hook.feed_pipe_ctx.  +	 */ +	feed_pipe_fn feed_pipe; + 
>> void *feed_pipe_ctx; 
> 
> The comment for the member is a bit too wide.  More importantly, 
> this does not seem to capture the fact that this is completely 
> ignored when path_to_stdin is already in effect.  We should at 
> least document it if we wanted to leave the behaviour as is, but 
> I wonder if we want to detect and flag it as BUG() if both 
> feed_pipe and path_to_stdin are not NULL.  There is no inherent 
> reason why the data prepared in a file must take precedence over 
> data coming over a pipe. 

Very good points. I agree we should flag a BUG() when both are 
provided, because the stdin feed methods are mutually exclusive.
 
This is actually something I thought about while coding/testing v1 
and just forgot to add in.

Will do in v2 and also make the comment less wide.

Many thanks for spotting this!

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Adrian Ratiu]

On Thu, 25 Sep 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Adrian Ratiu <adrian.ratiu@collabora.com> writes: 
> 
>> From: Emily Shaffer <emilyshaffer@google.com> 
>> 
>> By using 'hook.h' for 'post-rewrite', we simplify hook 
>> invocations by 
> 
> This "By using 'hook.h" is somewhat a strange thing to say. 
> <hook.h> has been in use by the file (evidenced by the fact that 
> there is no new "#include <hook.h>" in the patch).  I haven't 
> carefully read other steps in this series, but from my quick 
> skimming of them, I got an impression that this comment may 
> apply equally to other steps as well. 
> 
> What the commit does is not "use hook.h"; it is to replace a 
> custom run-command call with a call to run_hooks_opt().  The 
> shared API service function may happen to be declared in 
> <hook.h>, that that is secondary piece of information. 
> 
>> not needing to put together our own 'struct child_process'. 
> 
> Imperative?  I think just dropping "we" would be sufficient. 
> 
>> The signal handling that's being removed by this commit now 
>> takes place in run-command.h:run_processes_parallel(), so it is 
>> OK to remove them here. 
> 
> Phrase it more positively, instead of "it is OK" (which sounds 
> like it is also OK to leave it there).  Perhaps say something 
> like: 
> 
>     Another benefit we gain from using run_hook_opt() instead of 
>     a custom start_command()/finish_command() invocations is 
>     that the hook API handles with sigpipe itself, so we no 
>     longer need to toggle signals ourselves. 
> 
> or something like that, perhaps. 

Ack, will reword in v2. I actually inherited this message and thought
about rewording it as well :). Some others I already reworded in v1.

Re: [PATCH 04/10] transport: convert pre-push hook to hook.h

[Adrian Ratiu]

On Thu, 25 Sep 2025, "D. Ben Knoble" <ben.knoble@gmail.com> wrote:
> On Thu, Sep 25, 2025 at 8:54 AM Adrian Ratiu 
> <adrian.ratiu@collabora.com> wrote: 
>> 
>> From: Emily Shaffer <emilyshaffer@google.com> 
>> 
>> Move the pre-push hook away from run-command.h to and over to 
>> the new hook.h library. 
> 
> Perhaps s/to and// ? 

Ack, will reword in v2.

Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks

[Adrian Ratiu]

On Thu, 25 Sep 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Adrian Ratiu <adrian.ratiu@collabora.com> writes: 
> 
>> Signed-off-by: Emily Shaffer <emilyshaffer@google.com> 
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> 
>> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com> --- 
>>  refs.c | 61 
>>  ++++++++++++++++++++++++++++------------------------------ 1 
>>  file changed, 29 insertions(+), 32 deletions(-) 
> 
> Please describe what is done, why, and what benefit we are 
> reaping, just like you did for the previous few steps. 

Ack, will do in v2.

Re: [PATCH 04/10] transport: convert pre-push hook to hook.h

[Phillip Wood]

Hi Adrian

On 25/09/2025 13:53, Adrian Ratiu wrote:
>   
> -static int run_pre_push_hook(struct transport *transport,
> -			     struct ref *remote_refs)
> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
>   {
> -	int ret = 0, x;
> -	struct ref *r;
> -	struct child_process proc = CHILD_PROCESS_INIT;
> -	struct strbuf buf;
> -	const char *hook_path = find_hook(the_repository, "pre-push");
> -
> -	if (!hook_path)
> -		return 0;
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct ref *r = hook_cb->options->feed_pipe_ctx;
>   
> -	strvec_push(&proc.args, hook_path);
> -	strvec_push(&proc.args, transport->remote->name);
> -	strvec_push(&proc.args, transport->url);
> +	if (r) {
> +		struct strbuf buf = STRBUF_INIT;

If we passed the strbuf in as part of the context and called 
strbuf_reset() before using it each time we'd avoid allocating a new 
buffer for each ref just as the current code does.

Thanks

Phillip

> +		int ret = 0;
> +		hook_cb->options->feed_pipe_ctx = r->next;
>   
> -	proc.in = -1;
> -	proc.trace2_hook_name = "pre-push";
> +		if (!r->peer_ref) return 0;
> +		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;
> +		if (r->status == REF_STATUS_REJECT_STALE) return 0;
> +		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;
> +		if (r->status == REF_STATUS_UPTODATE) return 0;
>   
> -	if (start_command(&proc)) {
> -		finish_command(&proc);
> -		return -1;
> -	}
> +		strbuf_addf(&buf, "%s %s %s %s\n",
> +			    r->peer_ref->name, oid_to_hex(&r->new_oid),
> +			    r->name, oid_to_hex(&r->old_oid));
>   
> -	sigchain_push(SIGPIPE, SIG_IGN);
> +		ret = write_in_full(hook_stdin_fd, buf.buf, buf.len);
>   
> -	strbuf_init(&buf, 256);
> +		strbuf_release(&buf);
>   
> -	for (r = remote_refs; r; r = r->next) {
> -		if (!r->peer_ref) continue;
> -		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
> -		if (r->status == REF_STATUS_REJECT_STALE) continue;
> -		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
> -		if (r->status == REF_STATUS_UPTODATE) continue;
> +		/* We do not mind if a hook does not read all refs. */
> +		if (ret < 0 && errno != EPIPE)
> +			return ret;
>   
> -		strbuf_reset(&buf);
> -		strbuf_addf( &buf, "%s %s %s %s\n",
> -			 r->peer_ref->name, oid_to_hex(&r->new_oid),
> -			 r->name, oid_to_hex(&r->old_oid));
> -
> -		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
> -			/* We do not mind if a hook does not read all refs. */
> -			if (errno != EPIPE)
> -				ret = -1;
> -			break;
> -		}
> +		return 0;
>   	}
>   
> -	strbuf_release(&buf);
> +	return 1; /* we ran out of refs: no more input to feed */
> +}
>   
> -	x = close(proc.in);
> -	if (!ret)
> -		ret = x;
> +static int run_pre_push_hook(struct transport *transport,
> +			     struct ref *remote_refs)
> +{
> +	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
>   
> -	sigchain_pop(SIGPIPE);
> +	strvec_push(&opt.args, transport->remote->name);
> +	strvec_push(&opt.args, transport->url);
>   
> -	x = finish_command(&proc);
> -	if (!ret)
> -		ret = x;
> +	opt.feed_pipe = pre_push_hook_feed_stdin;
> +	opt.feed_pipe_ctx = remote_refs;
>   
> -	return ret;
> +	return run_hooks_opt(the_repository, "pre-push", &opt);
>   }
>   
>   int transport_push(struct repository *r,

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Phillip Wood]

Hi Adrian

Thanks for working on this, it would be really good to be able to run 
hooks in parallel.

On 25/09/2025 13:53, Adrian Ratiu wrote:
> From: Emily Shaffer <emilyshaffer@google.com>
> 
> By using 'hook.h' for 'post-rewrite', we simplify hook invocations by
> not needing to put together our own 'struct child_process'.

Right so instead we use the new api to feed an strbuf into the hook's 
stdin, sounds reasonable.

> The signal handling that's being removed by this commit now takes
> place in run-command.h:run_processes_parallel(), so it is OK to remove
> them here.
> 
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
> ---
>   sequencer.c | 62 ++++++++++++++++++++++++++++++++---------------------
>   1 file changed, 38 insertions(+), 24 deletions(-)
> 
> diff --git a/sequencer.c b/sequencer.c
> index 9ae40a91b2..93cd6ab1f2 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -1298,32 +1298,46 @@ int update_head_with_reflog(const struct commit *old_head,
>   	return ret;
>   }
>   
> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
> +{
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
> +	int ret;
> +
> +	if (!to_pipe || !to_pipe->len)
> +		return 1; /* nothing to feed */

Why are we running the hook if there is nothing to pass to it?

> +	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);

This will block until the hook has read all of the input. Unless the 
hook drains and closes stdin before it does anything else it will block 
the parallel execution of other hooks.

> +	if (ret < 0) {
> +		if (errno == EPIPE) {
> +			return 1; /* child closed pipe, nothing more to feed */
> +		}

Style: we don't use braces for single statement bodies.

> +		return ret;
> +	}
> +
> +	/* Reset the input buffer to avoid sending it again */
> +	strbuf_reset(to_pipe);

Shouldn't the return value do that?
> +	return ret;
> +}

The changes to run_rewrite_hook() look fine. I'm not sure whats 
happening in commit_post_rewrite() below though - am I missing something 
or have you just renamed "child" -> "notes_cp". If so I don't see what 
that has to do with using the new api.

Thanks

Phillip

>   void commit_post_rewrite(struct repository *r,
> @@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,
>   		flush_rewritten_pending();
>   		if (!stat(rebase_path_rewritten_list(), &st) &&
>   				st.st_size > 0) {
> -			struct child_process child = CHILD_PROCESS_INIT;
> +			struct child_process notes_cp = CHILD_PROCESS_INIT;
>   			struct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;
>   
> -			child.in = open(rebase_path_rewritten_list(), O_RDONLY);
> -			child.git_cmd = 1;
> -			strvec_push(&child.args, "notes");
> -			strvec_push(&child.args, "copy");
> -			strvec_push(&child.args, "--for-rewrite=rebase");
> +			notes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);
> +			notes_cp.git_cmd = 1;
> +			strvec_push(&notes_cp.args, "notes");
> +			strvec_push(&notes_cp.args, "copy");
> +			strvec_push(&notes_cp.args, "--for-rewrite=rebase");
>   			/* we don't care if this copying failed */
> -			run_command(&child);
> +			run_command(&notes_cp);
>   
>   			hook_opt.path_to_stdin = rebase_path_rewritten_list();
>   			strvec_push(&hook_opt.args, "rebase");

Re: [PATCH 06/10] run-command: allow capturing of collated output

[Adrian Ratiu]

On Thu, 25 Sep 2025, Junio C Hamano <gitster@pobox.com> wrote: 
>> diff --git a/builtin/fetch.c b/builtin/fetch.c index 
>> 24645c4653..53bd5552c4 100644 --- a/builtin/fetch.c +++ 
>> b/builtin/fetch.c @@ -2129,7 +2129,7 @@ static int 
>> fetch_multiple(struct string_list *list, int max_children, 
>>   if (max_children != 1 && list->nr != 1) { struct 
>>  parallel_fetch_state state = { argv.v, list, 0, 0, config }; 
>> -		const struct run_process_parallel_opts opts = { + 
>> struct run_process_parallel_opts opts = { 
>>  			.tr2_category = "fetch", .tr2_label = 
>>  "parallel/fetch", 
> 
> This ... 
> 
>> diff --git a/builtin/submodule--helper.c 
>> b/builtin/submodule--helper.c index 07a1935cbe..76cae9f015 
>> 100644 --- a/builtin/submodule--helper.c +++ 
>> b/builtin/submodule--helper.c @@ -2700,7 +2700,7 @@ static int 
>> update_submodules(struct update_data *update_data) 
>>  { int i, ret = 0; struct submodule_update_clone suc = 
>>  SUBMODULE_UPDATE_CLONE_INIT; 
>> -	const struct run_process_parallel_opts opts = { + 
>> struct run_process_parallel_opts opts = { 
>>  		.tr2_category = "submodule", .tr2_label = 
>>  "parallel/update", 
> 
> ... and this ... 
> 
>>   
>> diff --git a/hook.c b/hook.c index 54568d5bc0..199c210b97 
>> 100644 --- a/hook.c +++ b/hook.c @@ -135,7 +135,7 @@ int 
>> run_hooks_opt(struct repository *r, const char *hook_name, 
>>  	}; const char *const hook_path = find_hook(r, hook_name); 
>>  int ret = 0; 
>> -	const struct run_process_parallel_opts opts = { + 
>> struct run_process_parallel_opts opts = { 
>>  		.tr2_category = "hook", .tr2_label = hook_name, 
> 
> ... and this are curious changes that are not explained in the 
> proposed log message. 

Yes and sorry for not explaining these better. The only reason I 
had to remove the const is to be able to set opts->ungroup = 0 
below.

If I can find a way to do what you propose, then 100% I will drop 
all these hunks. I agree that is the best way forward in v2.

> 
>> @@ -1841,6 +1852,10 @@ void run_processes_parallel(const struct 
>> run_process_parallel_opts *opts) 
>>  					   "max:%"PRIuMAX, 
>>  (uintmax_t)opts->processes);  
>> +	/* ungroup and reading sideband are mutualy exclusive, so 
>> disable ungroup */ +	If (opts->ungroup && 
>> opts->consume_sideband) +		opts->ungroup = 0; 
> 
> Make it a BUG(""), which may help avoid unintended bugs, 
> especially ... 

I did exactly this and got test failures because some tests end up 
setting both ungroup and consume_sideband. Since the original code 
I got from Emily and Aevar just defaulted to setting ungroup = 0 
and removing the const I went with that instead of actually fixing 
the tests, assuming the tests actually need to do that. :)

Now I know better and for v2 I will fix the tests and add a BUG() 
here, with proper reasoning for the test modification.

An example of test which fails because it sets both is:
t1416-ref-transaction-hooks.sh
not ok 7 - interleaving hook calls succeed
 
>> diff --git a/run-command.h b/run-command.h index 
>> 4679987c8e..ad0bab14b0 100644 --- a/run-command.h +++ 
>> b/run-command.h @@ -436,6 +436,20 @@ typedef int 
>> (*feed_pipe_fn)(int child_in, 
>>  				void *pp_cb, void *pp_task_cb);  
>> +/** + * If this callback is provided, instead of collating 
>> process output to stderr, + * they will be collated into a new 
>> pipe. consume_sideband_fn will be called + * repeatedly. When 
>> output is available on that pipe, it will be contained in + * 
>> 'output'. But it will be called with an empty 'output' too, to 
>> allow for + * keepalives or similar operations if necessary.  + 
>> * + * pp_cb is the callback cookie as passed into 
>> run_processes_parallel.  + * + * Since this callback is 
>> provided with the collated output, no task cookie is + * 
>> provided.  + */ +typedef void (*consume_sideband_fn)(struct 
>> strbuf *output, void *pp_cb); + 
>>  /** 
>>   * This callback is called on every child process that 
>>   finished processing.  * 
>> @@ -495,6 +509,12 @@ struct run_process_parallel_opts 
>>  	 */ feed_pipe_fn feed_pipe;  
>> +	/* +	 * consume_sideband: see consume_sideband_fn() 
>> above. This can be NULL +	 * to omit any special handling. 
>> +	 */ +	consume_sideband_fn consume_sideband; 
> 
> ... because which one between this and ungroup gets precedence. 
> Document that they are mutually exclusive, and help the callers 
> with a BUG("") message when both are set.

Agreed, will do in v2.
 
> 
>> @@ -529,7 +549,7 @@ struct run_process_parallel_opts 
>>   * emitting their own output, including dealing with any race 
>>   * conditions due to writing in parallel to stdout and stderr. 
>>   */ 
>> -void run_processes_parallel(const struct 
>> run_process_parallel_opts *opts); +void 
>> run_processes_parallel(struct run_process_parallel_opts *opts); 
> 
> This is the same unexplained curiousity I touched earlier. 

Yes, I promise I'll drop all these in v2.

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Adrian Ratiu]

On Fri, 26 Sep 2025, Phillip Wood <phillip.wood123@gmail.com> 
wrote:
> Hi Adrian 
> 
> Thanks for working on this, it would be really good to be able 
> to run  hooks in parallel. 

Hi Phillip and thank you for your feedback! It is very valuable 
and appreciated on all my patch series.

> 
> On 25/09/2025 13:53, Adrian Ratiu wrote: 
>> From: Emily Shaffer <emilyshaffer@google.com>  By using 
>> 'hook.h' for 'post-rewrite', we simplify hook invocations by 
>> not needing to put together our own 'struct child_process'. 
> 
> Right so instead we use the new api to feed an strbuf into the 
> hook's  stdin, sounds reasonable. 

That is the high level idea yes, maybe I can improve the commit 
msg a bit in v2 to make it clearer (those are not actually my 
words :).

Slightly unrelated:

I actually thought about putting pipe_from_strbuf() into hook.c or 
someplace similar because it's a generic utility function, however 
this is the only hook which needs it, so I've left it in 
sequencer.c.
 
>> The signal handling that's being removed by this commit now 
>> takes place in run-command.h:run_processes_parallel(), so it is 
>> OK to remove them here.   Signed-off-by: Emily Shaffer 
>> <emilyshaffer@google.com> Signed-off-by: Ævar Arnfjörð 
>> Bjarmason <avarab@gmail.com> Signed-off-by: Adrian Ratiu 
>> <adrian.ratiu@collabora.com> --- 
>>   sequencer.c | 62 
>>   ++++++++++++++++++++++++++++++++--------------------- 1 file 
>>   changed, 38 insertions(+), 24 deletions(-) 
>>  diff --git a/sequencer.c b/sequencer.c index 
>> 9ae40a91b2..93cd6ab1f2 100644 --- a/sequencer.c +++ 
>> b/sequencer.c @@ -1298,32 +1298,46 @@ int 
>> update_head_with_reflog(const struct commit *old_head, 
>>   	return ret; }  
>> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, 
>> void *pp_task_cb UNUSED) +{ +	struct hook_cb_data 
>> *hook_cb = pp_cb; +	struct strbuf *to_pipe = 
>> hook_cb->options->feed_pipe_ctx; +	int ret; + +	if 
>> (!to_pipe || !to_pipe->len) +		return 1; /* 
>> nothing to feed */ 
> 
> Why are we running the hook if there is nothing to pass to it? 

run-commands has a ppoll loop which calls the stdin feed callback 
(pipe_from_stdbuf in this case) repeatedly until it signals 
reading is finished by return 1;

Now that I look again at this code, I made the mistake of assuming 
it needs to work recursively:
    1. write the strbuf
    2. reset the strbuf (return 0)
    3. next callback sees the strbuf is empty and stops the loop 
    (return 1)

The line you're asking asking here is actually step 3. :)

This all can be simplified by writing just once and returning 1 
immediately since it's just a simple strbuf to write to stdin.

We still need to keep the pointer null check though, just in case.
 
>> +	ret = write_in_full(hook_stdin_fd, to_pipe->buf, 
>> to_pipe->len); 
> 
> This will block until the hook has read all of the input. Unless 
> the  hook drains and closes stdin before it does anything else 
> it will block  the parallel execution of other hooks. 

I double checked the write_in_full / xwrite implementations. :)

Sorry for the wall of text btw, I try to explain as best I can.

I think it blocks only if/when the stdin fd pipe is full, which 
can't happen in this specific instance because the strbuf data is 
small, so the write_in_full() call just writes all it has, then 
returns.

In other words, the most important aspect I think is how much data 
we are writing to the pipe in every single callback call.

The idea of these callbacks is to write small chunks of data at a 
time, then switch context: it's usually the child hook processes 
which end up blocking for their stdin input which is fed by the 
parent which multiplexes between the parallel processes.

Of course, a balance needs to be found: I've noticed, in other 
hooks, that if we write too small chunks of data in each callback, 
we unnecessarily increase wait times for hooks.

This can be seen especially in hooks like post-receive which can 
get a lot of data: if we feed one line at a time and let's say 
run-command's ppoll loop adds 100ms delay between callbacks, then 
from 7-800 ms we end up with something like 90 seconds!

Of course that is a pathological case and the solution there is to 
batch more data in a single callback write. I've actually batched 
500 lines in every write for those update hooks (we can do more or 
less).

None of this is set in stone and can be changed. What I tried is 
to get similar performance with what we had before these 
callbacks.

If you notice any specific degradation or unnecessary blocking, 
please raise it up and we can address it.
 
>> +	if (ret < 0) { +		if (errno == EPIPE) { + 
>> return 1; /* child closed pipe, nothing more to feed */ + 
>> } 
> 
> Style: we don't use braces for single statement bodies. 

Ack, will fix in v2.
 
>> +		return ret; +	} + +	/* Reset the input buffer 
>> to avoid sending it again */ +	strbuf_reset(to_pipe); 
> 
> Shouldn't the return value do that?

Yes, as explained above with my 1 2 3 recursive steps above we 
don't actually need this and the function can be simplified.

I'll do that in v2.
 
>> +	return ret; +} 
> 
> The changes to run_rewrite_hook() look fine. I'm not sure whats 
> happening in commit_post_rewrite() below though - am I missing 
> something  or have you just renamed "child" -> "notes_cp". If so 
> I don't see what  that has to do with using the new api.

Thanks for spotting this!

It's actually a leftover from Emily and Aevar's string_list API 
implementation which I've rewritten / removed and this hunk fell 
through the cracks. :)

Will drop it in v2.

>
> Thanks
>
> Phillip
>
>>   void commit_post_rewrite(struct repository *r,
>> @@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,
>>   		flush_rewritten_pending();
>>   		if (!stat(rebase_path_rewritten_list(), &st) &&
>>   				st.st_size > 0) {
>> -			struct child_process child = CHILD_PROCESS_INIT;
>> +			struct child_process notes_cp = CHILD_PROCESS_INIT;
>>   			struct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;
>>   
>> -			child.in = open(rebase_path_rewritten_list(), O_RDONLY);
>> -			child.git_cmd = 1;
>> -			strvec_push(&child.args, "notes");
>> -			strvec_push(&child.args, "copy");
>> -			strvec_push(&child.args, "--for-rewrite=rebase");
>> +			notes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);
>> +			notes_cp.git_cmd = 1;
>> +			strvec_push(&notes_cp.args, "notes");
>> +			strvec_push(&notes_cp.args, "copy");
>> +			strvec_push(&notes_cp.args, "--for-rewrite=rebase");
>>   			/* we don't care if this copying failed */
>> -			run_command(&child);
>> +			run_command(&notes_cp);
>>   
>>   			hook_opt.path_to_stdin = rebase_path_rewritten_list();
>>   			strvec_push(&hook_opt.args, "rebase");

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

>> +	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);
>
> This will block until the hook has read all of the input. Unless the
> hook drains and closes stdin before it does anything else it will
> block the parallel execution of other hooks.

Ouch.

>> +	if (ret < 0) {
>> +		if (errno == EPIPE) {
>> +			return 1; /* child closed pipe, nothing more to feed */
>> +		}
>
> Style: we don't use braces for single statement bodies.
>
>> +		return ret;
>> +	}
>> +
>> +	/* Reset the input buffer to avoid sending it again */
>> +	strbuf_reset(to_pipe);
>
> Shouldn't the return value do that?

Sorry, I do not understand this comment, but did you mean to_pipe
strbuf is left with some buffered bytes when we take the early-return
path when we got an error above?

This part of the new code makes me wonder what the lifetime rules
for the to_pipe message are?

In the original code before this rewrite, it was clear that the
caller of this function was responsible to allocate the strbuf, to
feed it to the subprocess, and to release the resources it held.
Now, what is the rule?  The caller still prepares the strbuf, but
the called machinery using the hook API will release the resources?

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Adrian Ratiu]

On Fri, 26 Sep 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Phillip Wood <phillip.wood123@gmail.com> writes: 
> 
>>> +	ret = write_in_full(hook_stdin_fd, to_pipe->buf, 
>>> to_pipe->len); 
>> 
>> This will block until the hook has read all of the 
>> input. Unless the hook drains and closes stdin before it does 
>> anything else it will block the parallel execution of other 
>> hooks. 
> 
> Ouch. 

I replied separately to this, because I do not think it is 
correct. :)

Please see my other reply and if there are further questions or 
concerns we can continue the discussion there, happy to address 
them.
 
>>> +	if (ret < 0) { +		if (errno == EPIPE) { + 
>>> return 1; /* child closed pipe, nothing more to feed */ + 
>>> } 
>> 
>> Style: we don't use braces for single statement bodies. 
>> 
>>> +		return ret; +	} + +	/* Reset the input buffer 
>>> to avoid sending it again */ +	strbuf_reset(to_pipe); 
>> 
>> Shouldn't the return value do that? 
> 
> Sorry, I do not understand this comment, but did you mean 
> to_pipe strbuf is left with some buffered bytes when we take the 
> early-return path when we got an error above?

I described in the other reply how this function works. I messed 
up by making it more complicated than necessary. I'll simplify it 
in v2.

Basically I assumed this write works recursively and the reset 
stops the loop, so the next call exits early to stops loop.

Since we write just 1 strbuf once, there is no need for 
recursiveness and we can just return based on the return value, as 
Phillip suggested.
 
> 
> This part of the new code makes me wonder what the lifetime 
> rules for the to_pipe message are? 
> 
> In the original code before this rewrite, it was clear that the 
> caller of this function was responsible to allocate the strbuf, 
> to feed it to the subprocess, and to release the resources it 
> held.  Now, what is the rule?  The caller still prepares the 
> strbuf, but the called machinery using the hook API will release 
> the resources? 

It's the same as before: the caller owns the strbuf and releases 
it (see the strbuf_release(&sb); in this patch's context, I did 
not modify that).

I'll document this in addition to simplifying the function logic in v2.

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Phillip Wood]

Hi Adrian

On 26/09/2025 16:53, Adrian Ratiu wrote:
> On Fri, 26 Sep 2025, Phillip Wood <phillip.wood123@gmail.com> wrote:
>> On 25/09/2025 13:53, Adrian Ratiu wrote:
> 
> I actually thought about putting pipe_from_strbuf() into hook.c or 
> someplace similar because it's a generic utility function, however this 
> is the only hook which needs it, so I've left it in sequencer.c.

Yes, I was surprised that was the only place we ended up needing that 
function. I think it makes sense to leave it where it is if that's the 
only place that needs it.

>> Why are we running the hook if there is nothing to pass to it? 
> 
> run-commands has a ppoll loop which calls the stdin feed callback 
> (pipe_from_stdbuf in this case) repeatedly until it signals reading is 
> finished by return 1;
> 
> Now that I look again at this code, I made the mistake of assuming it 
> needs to work recursively:
>     1. write the strbuf
>     2. reset the strbuf (return 0)
>     3. next callback sees the strbuf is empty and stops the loop    
> (return 1)
> 
> The line you're asking asking here is actually step 3. :)
> 
> This all can be simplified by writing just once and returning 1 
> immediately since it's just a simple strbuf to write to stdin.
> 
> We still need to keep the pointer null check though, just in case.

We should make that a BUG() I think

>>> +    ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len); 
>>
>> This will block until the hook has read all of the input. Unless the  
>> hook drains and closes stdin before it does anything else it will 
>> block  the parallel execution of other hooks. 
> 
> I double checked the write_in_full / xwrite implementations. :)
> 
> Sorry for the wall of text btw, I try to explain as best I can.
> 
> I think it blocks only if/when the stdin fd pipe is full, which can't 
> happen in this specific instance because the strbuf data is small, so 
> the write_in_full() call just writes all it has, then returns.

Oh sorry, I'd forgotten this was just used on a short buffer, and not 
used when we're rebasing. Good point, it should be fine.

> In other words, the most important aspect I think is how much data we 
> are writing to the pipe in every single callback call.
> 
> The idea of these callbacks is to write small chunks of data at a time, 
> then switch context: it's usually the child hook processes which end up 
> blocking for their stdin input which is fed by the parent which 
> multiplexes between the parallel processes.
> 
> Of course, a balance needs to be found: I've noticed, in other hooks, 
> that if we write too small chunks of data in each callback, we 
> unnecessarily increase wait times for hooks.
> 
> This can be seen especially in hooks like post-receive which can get a 
> lot of data: if we feed one line at a time and let's say run-command's 
> ppoll loop adds 100ms delay between callbacks, then from 7-800 ms we end 
> up with something like 90 seconds!
> 
> Of course that is a pathological case and the solution there is to batch 
> more data in a single callback write. I've actually batched 500 lines in 
> every write for those update hooks (we can do more or less).
> 
> None of this is set in stone and can be changed. What I tried is to get 
> similar performance with what we had before these callbacks.

That's great, getting a good balance between not blocking for too long 
and doing a reasonable amount of work in each call to the callback 
sounds like a good plan.

Thanks

Phillip

Re: [PATCH 04/10] transport: convert pre-push hook to hook.h

[Adrian Ratiu]

On Fri, 26 Sep 2025, Phillip Wood <phillip.wood123@gmail.com> 
wrote:
> Hi Adrian
>
> On 25/09/2025 13:53, Adrian Ratiu wrote:
>>   
>> -static int run_pre_push_hook(struct transport *transport,
>> -			     struct ref *remote_refs)
>> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
>>   {
>> -	int ret = 0, x;
>> -	struct ref *r;
>> -	struct child_process proc = CHILD_PROCESS_INIT;
>> -	struct strbuf buf;
>> -	const char *hook_path = find_hook(the_repository, "pre-push");
>> -
>> -	if (!hook_path)
>> -		return 0;
>> +	struct hook_cb_data *hook_cb = pp_cb;
>> +	struct ref *r = hook_cb->options->feed_pipe_ctx;
>>   
>> -	strvec_push(&proc.args, hook_path);
>> -	strvec_push(&proc.args, transport->remote->name);
>> -	strvec_push(&proc.args, transport->url);
>> +	if (r) {
>> +		struct strbuf buf = STRBUF_INIT;
>
> If we passed the strbuf in as part of the context and called 
> strbuf_reset() before using it each time we'd avoid allocating a new 
> buffer for each ref just as the current code does.

That is a good idea. Will do it in v2. Thanks!

Re: [PATCH 01/10] run-command: add stdin callback for parallelization

[Patrick Steinhardt]

On Thu, Sep 25, 2025 at 03:53:44PM +0300, Adrian Ratiu wrote:
> diff --git a/run-command.c b/run-command.c
> index ed9575bd6a..6c455a0e43 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,
>  	return 0;
>  }
>  
> +static void pp_buffer_stdin(struct parallel_processes *pp,
> +			    const struct run_process_parallel_opts *opts)
> +{
> +	/* Buffer stdin for each pipe. */
> +	for (int i = 0; i < opts->processes; i++) {

`opts->processes` is of type `size_t`, so let's use the same type as
iterator.

> +		struct child_process *proc = &pp->children[i].process;
> +		int ret;
> +
> +		if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)
> +			continue;
> +
> +		/**

Nit: multi-line comments should start with "/*", not "/**". This is also
present in multiple other 

> +		 * child input is provided via path_to_stdin when the feed_pipe cb is
> +		 * missing, so we just signal an EOF.
> +		 */
> +		if (!opts->feed_pipe) {
> +			close(proc->in);
> +			proc->in = 0;

Hm. It's curious that we use a valid file descriptor here. Shouldn't we
rather use `-1`? Otherwise I could see that we might try to close this
seemingly valid file descriptor at a later point in time.

> +			continue;
> +		}
> +
> +		/**
> +		 * Feed the pipe:
> +		 *   ret < 0 means error
> +		 *   ret == 0 means there is more data to be fed
> +		 *   ret > 0 means feeding finished
> +		 */
> +		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
> +		if (ret < 0)
> +			die_errno("feed_pipe");
> +
> +		if (ret == 1) {

This condition mismatches the comment: you explicitly check for 1, but
the comment above says `ret > 0` indicates that feeding has finished.

> @@ -1756,6 +1795,33 @@ static int pp_collect_finished(struct parallel_processes *pp,
>  	return result;
>  }
>  
> +static void pp_handle_child_IO(struct parallel_processes *pp,
> +				const struct run_process_parallel_opts *opts,
> +				int output_timeout)

Okay, this function is new and was extracted out of
`run_processes_parallel()`. It's basically the heart of our I/O loop
for our children.

> +{
> +	/*
> +	 * First push input, if any (it might no-op), to child tasks to avoid them blocking
> +	 * after input. This also prevents deadlocks when ungrouping below, if a child blocks
> +	 * while the parent also waits for them to finish.
> +	 */
> +	pp_buffer_stdin(pp, opts);

This part is new, as we now know to also optionally write stdin to the
child process.

> +	if (opts->ungroup) {
> +		for (size_t i = 0; i < opts->processes; i++) {
> +			int child_ready_for_cleanup =
> +				pp->children[i].state == GIT_CP_WORKING &&
> +				pp->children[i].process.in == 0;
> +
> +			if (child_ready_for_cleanup)
> +				pp->children[i].state = GIT_CP_WAIT_CLEANUP;

And this part here has changed, as well. We don't unconditionally set
`GIT_CP_WAIT_CLEANUP` anymore, but wait for `process.in` to be closed.

> +		}
> +		return;

I feel like this return is easy to miss. I think an `else` branch would
be more obvious.

> @@ -1775,6 +1841,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  					   "max:%"PRIuMAX,
>  					   (uintmax_t)opts->processes);
>  
> +	/*
> +	 * Child tasks might receive input via stdin, terminating early (or not), so
> +	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
> +	 * actually writes the data to children stdin fds.
> +	 */
> +	sigchain_push(SIGPIPE, SIG_IGN);
> +
>  	pp_init(&pp, opts, &pp_sig);
>  	while (1) {
>  		for (i = 0;

Yeah, makes sense. I was briefly wondering whether we should rather do
it as part of `pp_buffer_stdin()`, so that it's more contained. But I'm
not sure that buys us anything.

> @@ -1809,8 +1876,11 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  
>  	pp_cleanup(&pp, opts);
>  
> +	sigchain_pop(SIGPIPE);
> +

There are no early exits, so we know this code should be executed.

>  	if (do_trace2)
>  		trace2_region_leave(tr2_category, tr2_label, NULL);
> +
>  }
>  
>  int prepare_auto_maintenance(int quiet, struct child_process *maint)

Nit: stray empty line.

> diff --git a/run-command.h b/run-command.h
> index 0df25e445f..4679987c8e 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,
>  				void *pp_cb,
>  				void *pp_task_cb);
>  
> +/**
> + * This callback is repeatedly called on every child process who requests
> + * start_command() to create a pipe by setting child_process.in < 0.
> + *
> + * pp_cb is the callback cookie as passed into run_processes_parallel, and
> + * pp_task_cb is the callback cookie as passed into get_next_task_fn.
> + * The contents of 'send' will be read into the pipe and passed to the pipe.
> + *
> + * Returns < 0 for error
> + * Returns == 0 when there is more data to be fed (will be called again)
> + * Returns > 0 when finished (child closes fd or no more data to be fed)

s/closes/closed/

> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
> index 3719f23cc2..dfdb03b3ab 100644
> --- a/t/helper/test-run-command.c
> +++ b/t/helper/test-run-command.c

This helper very much looks like it should be converted to a unit test.
Anyway, that is outside of the scope of this patch series.

> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
> index 76d4936a87..282afecefc 100755
> --- a/t/t0061-run-command.sh
> +++ b/t/t0061-run-command.sh
> @@ -164,6 +164,36 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
>  	test_line_count = 4 err
>  '
>  
> +cat >expect <<-EOF
> +preloaded output of a child
> +listening for stdin:
> +sample stdin 1
> +sample stdin 0
> +preloaded output of a child
> +listening for stdin:
> +sample stdin 1
> +sample stdin 0
> +preloaded output of a child
> +listening for stdin:
> +sample stdin 1
> +sample stdin 0
> +preloaded output of a child
> +listening for stdin:
> +sample stdin 1
> +sample stdin 0
> +EOF

This block should be part of the test itself.

> +test_expect_success 'run_command listens to stdin' '
> +	write_script stdin-script <<-\EOF &&
> +	echo "listening for stdin:"
> +	while read line; do

Style nit: let's drop the `;` and move the `do` to the next line.

> +		echo "$line"
> +	done
> +	EOF
> +	test-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&
> +	test_cmp expect actual
> +'

Patrick

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Patrick Steinhardt]

On Thu, Sep 25, 2025 at 03:53:46PM +0300, Adrian Ratiu wrote:
> diff --git a/sequencer.c b/sequencer.c
> index 9ae40a91b2..93cd6ab1f2 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -1298,32 +1298,46 @@ int update_head_with_reflog(const struct commit *old_head,
>  	return ret;
>  }
>  
> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
> +{
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
> +	int ret;
> +
> +	if (!to_pipe || !to_pipe->len)
> +		return 1; /* nothing to feed */
> +
> +	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);

One thing I wondered in previous patches was whether we now have the
potential for deadlocks. If we feed data to a child that exceeds the
buffered I/O size, and that child writes data that is consumed by Git
larger than the buffered I/O size, as well. Wouldn't that mean that we
may now not make any progress at all?

I guess that's no different compared to before though, as we also used
`write_in_full()` there.

> +	if (ret < 0) {
> +		if (errno == EPIPE) {
> +			return 1; /* child closed pipe, nothing more to feed */
> +		}

Style: let's drop the curly braces around single-line statements.

> +		return ret;
> +	}
> +
> +	/* Reset the input buffer to avoid sending it again */
> +	strbuf_reset(to_pipe);

Is this really necessary? I would've expected that we return a positive
value from this callback, and as a consequence the run-command subsystem
should notice that we're done with writing stdin and close the file
descriptor for us. Afterwards, it shouldn't invoke this callback ever
again, shouldn't it?

> @@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,
>  		flush_rewritten_pending();
>  		if (!stat(rebase_path_rewritten_list(), &st) &&
>  				st.st_size > 0) {
> -			struct child_process child = CHILD_PROCESS_INIT;
> +			struct child_process notes_cp = CHILD_PROCESS_INIT;
>  			struct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;
>  
> -			child.in = open(rebase_path_rewritten_list(), O_RDONLY);
> -			child.git_cmd = 1;
> -			strvec_push(&child.args, "notes");
> -			strvec_push(&child.args, "copy");
> -			strvec_push(&child.args, "--for-rewrite=rebase");
> +			notes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);
> +			notes_cp.git_cmd = 1;
> +			strvec_push(&notes_cp.args, "notes");
> +			strvec_push(&notes_cp.args, "copy");
> +			strvec_push(&notes_cp.args, "--for-rewrite=rebase");
>  			/* we don't care if this copying failed */
> -			run_command(&child);
> +			run_command(&notes_cp);
>  
>  			hook_opt.path_to_stdin = rebase_path_rewritten_list();
>  			strvec_push(&hook_opt.args, "rebase");

This change looks completely unrelated to me?

Patrick

Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks

[Patrick Steinhardt]

On Thu, Sep 25, 2025 at 03:53:48PM +0300, Adrian Ratiu wrote:
> diff --git a/refs.c b/refs.c
> index 4ff55cf24f..5a2b6ad1fc 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2377,31 +2377,16 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
>  	return 0;
>  }
>  
> -static int run_transaction_hook(struct ref_transaction *transaction,
> -				const char *state)
> +static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
>  {
> -	struct child_process proc = CHILD_PROCESS_INIT;
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct run_hooks_opt *opt = hook_cb->options;
> +	struct ref_transaction *transaction = opt->feed_pipe_ctx;
>  	struct strbuf buf = STRBUF_INIT;
> -	const char *hook;
> -	int ret = 0, i;
> -
> -	hook = find_hook(transaction->ref_store->repo, "reference-transaction");
> -	if (!hook)
> -		return ret;
> -
> -	strvec_pushl(&proc.args, hook, state, NULL);
> -	proc.in = -1;
> -	proc.stdout_to_stderr = 1;
> -	proc.trace2_hook_name = "reference-transaction";
> -
> -	ret = start_command(&proc);
> -	if (ret)
> -		return ret;
> -
> -	sigchain_push(SIGPIPE, SIG_IGN);
>  
> -	for (i = 0; i < transaction->nr; i++) {
> +	for (int i = 0; i < transaction->nr; i++) {
>  		struct ref_update *update = transaction->updates[i];
> +		int ret;
>  
>  		if (update->flags & REF_LOG_ONLY)
>  			continue;

Hm. In the "pre-push" hook you converted the callback to process one ref
per invocation. Why don't we do the same over here, with one transaction
per invocation?

Not saying that either one of these is better, but it left me puzzled
why we use two different patterns now.

Patrick

Re: [PATCH 01/10] run-command: add stdin callback for parallelization

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>> +	/* Buffer stdin for each pipe. */
>> +	for (int i = 0; i < opts->processes; i++) {
>
> `opts->processes` is of type `size_t`, so let's use the same type as
> iterator.

Good eyes.

>> +		/**
>
> Nit: multi-line comments should start with "/*", not "/**". This is also
> present in multiple other 

True.

Especially for a comment about a specific piece of code and not
about an interface---even in a future where we use some tool to
extract them, we would not place them in documentation.

>> +		 * child input is provided via path_to_stdin when the feed_pipe cb is
>> +		 * missing, so we just signal an EOF.
>> +		 */
>> +		if (!opts->feed_pipe) {
>> +			close(proc->in);
>> +			proc->in = 0;
>
> Hm. It's curious that we use a valid file descriptor here. Shouldn't we
> rather use `-1`? Otherwise I could see that we might try to close this
> seemingly valid file descriptor at a later point in time.

Good eyes.  Does -1 also have special meaning or we have no risk
mistaking this proc->in that was once used with a request to open a
pipe?  Regardless, I agree 0 would be a bad choice here.

>> +		/**
>> +		 * Feed the pipe:
>> +		 *   ret < 0 means error
>> +		 *   ret == 0 means there is more data to be fed
>> +		 *   ret > 0 means feeding finished
>> +		 */
>> +		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
>> +		if (ret < 0)
>> +			die_errno("feed_pipe");
>> +
>> +		if (ret == 1) {
>
> This condition mismatches the comment: you explicitly check for 1, but
> the comment above says `ret > 0` indicates that feeding has finished.

True.  We already handled negative, so we can just do "if (ret)"
here, but "if (0 < ret)" is also fine.

Thanks.

Re: [PATCH 01/10] run-command: add stdin callback for parallelization

[Adrian Ratiu]

Hi Patrick and thanks for review! I'll fix in v2 all the issues 
you pointed out.

On Thu, 02 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
>> +		 * child input is provided via path_to_stdin when 
>> the feed_pipe cb is +		 * missing, so we just 
>> signal an EOF.  +		 */ +		if 
>> (!opts->feed_pipe) { +			close(proc->in); + 
>> proc->in = 0; 
> 
> Hm. It's curious that we use a valid file descriptor 
> here. Shouldn't we rather use `-1`? Otherwise I could see that 
> we might try to close this seemingly valid file descriptor at a 
> later point in time. 
> 

I actually asked myself this while preparing the patches, since -1 
is a better fit.

I only left = 0 for historical reasons, to not modify these 
patches too much. :) However I do 100% agree with both you and 
Junio that -1 should be used here.

Will do in v2.

Re: [PATCH 01/10] run-command: add stdin callback for parallelization

[Adrian Ratiu]

On Thu, 02 Oct 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Patrick Steinhardt <ps@pks.im> writes: 
> 
>>> +	/* Buffer stdin for each pipe. */ +	for (int i = 0; i 
>>> < opts->processes; i++) { 
>> 
>> `opts->processes` is of type `size_t`, so let's use the same 
>> type as iterator. 
> 
> Good eyes. 
> 
>>> +		/** 
>> 
>> Nit: multi-line comments should start with "/*", not 
>> "/**". This is also present in multiple other  
> 
> True. 
> 
> Especially for a comment about a specific piece of code and not 
> about an interface---even in a future where we use some tool to 
> extract them, we would not place them in documentation. 
> 
>>> +		 * child input is provided via path_to_stdin when 
>>> the feed_pipe cb is +		 * missing, so we just 
>>> signal an EOF.  +		 */ +		if 
>>> (!opts->feed_pipe) { +			close(proc->in); + 
>>> proc->in = 0; 
>> 
>> Hm. It's curious that we use a valid file descriptor 
>> here. Shouldn't we rather use `-1`? Otherwise I could see that 
>> we might try to close this seemingly valid file descriptor at a 
>> later point in time. 
> 
> Good eyes.  Does -1 also have special meaning or we have no risk 
> mistaking this proc->in that was once used with a request to 
> open a pipe?  Regardless, I agree 0 would be a bad choice here. 
> 

Ack, will fix all these in v2.

Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h

[Adrian Ratiu]

Hi Patrick!

On Thu, 02 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Thu, Sep 25, 2025 at 03:53:46PM +0300, Adrian Ratiu wrote: 
>> diff --git a/sequencer.c b/sequencer.c index 
>> 9ae40a91b2..93cd6ab1f2 100644 --- a/sequencer.c +++ 
>> b/sequencer.c @@ -1298,32 +1298,46 @@ int 
>> update_head_with_reflog(const struct commit *old_head, 
>>  	return ret; }  
>> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, 
>> void *pp_task_cb UNUSED) +{ +	struct hook_cb_data 
>> *hook_cb = pp_cb; +	struct strbuf *to_pipe = 
>> hook_cb->options->feed_pipe_ctx; +	int ret; + +	if 
>> (!to_pipe || !to_pipe->len) +		return 1; /* 
>> nothing to feed */ + +	ret = write_in_full(hook_stdin_fd, 
>> to_pipe->buf, to_pipe->len); 
> 
> One thing I wondered in previous patches was whether we now have 
> the potential for deadlocks. If we feed data to a child that 
> exceeds the buffered I/O size, and that child writes data that 
> is consumed by Git larger than the buffered I/O size, as 
> well. Wouldn't that mean that we may now not make any progress 
> at all? 
> 
> I guess that's no different compared to before though, as we 
> also used `write_in_full()` there.

That is correct, yes, it's the same as before.

Deadlocks can happen, however they are the result of bugs, for 
example a hook child waits for stdin, the parent doesn't feed 
anything and decides to wait for the child to finish. :)

I hit quite a few of these during development, however they should 
all be fixed (deadlocks are usually easily fixed once detected).

Related, but also important, is thoroughtput when feeding the 
pipes: sending input too granularly (e.g. by calling the callback 
on each line when we send many lines) can add unnecessary 
latencies / delays due to the run-command ppoll mechanism.

It's a balance we must find: for most hooks it doesn't matter 
because the input is small (in this case it's a single write), 
however hooks like post-receive get large amount of data, so there 
I had to batch 300-500 lines to get simliar performance as before 
the callback.

So yes, it's improtant to have no deadlocks and it's also 
important to have roughly the same throughtput through the pipes.
 
>> +	if (ret < 0) { +		if (errno == EPIPE) { + 
>> return 1; /* child closed pipe, nothing more to feed */ + 
>> } 
> 
> Style: let's drop the curly braces around single-line 
> statements. 

Ack, will do, others pointed it out as well.
 
>> +		return ret; +	} + +	/* Reset the input buffer 
>> to avoid sending it again */ +	strbuf_reset(to_pipe); 
> 
> Is this really necessary? I would've expected that we return a 
> positive value from this callback, and as a consequence the 
> run-command subsystem should notice that we're done with writing 
> stdin and close the file descriptor for us. Afterwards, it 
> shouldn't invoke this callback ever again, shouldn't it?

That is correct: it is not necessary. Phillip already made me 
aware that I can significantly simplify this function, which I 
will do in v2 shortly. :)
 
> 
>> @@ -5140,16 +5154,16 @@ static int pick_commits(struct 
>> repository *r, 
>>  		flush_rewritten_pending(); if 
>>  (!stat(rebase_path_rewritten_list(), &st) && st.st_size > 0) { 
>> -			struct child_process child = 
>> CHILD_PROCESS_INIT; +			struct 
>> child_process notes_cp = CHILD_PROCESS_INIT; 
>>  			struct run_hooks_opt hook_opt = 
>>  RUN_HOOKS_OPT_INIT;  
>> -			child.in = 
>> open(rebase_path_rewritten_list(), O_RDONLY); - 
>> child.git_cmd = 1; -			strvec_push(&child.args, 
>> "notes"); -			strvec_push(&child.args, "copy"); 
>> -			strvec_push(&child.args, 
>> "--for-rewrite=rebase"); +			notes_cp.in = 
>> open(rebase_path_rewritten_list(), O_RDONLY); + 
>> notes_cp.git_cmd = 1; + 
>> strvec_push(&notes_cp.args, "notes"); + 
>> strvec_push(&notes_cp.args, "copy"); + 
>> strvec_push(&notes_cp.args, "--for-rewrite=rebase"); 
>>  			/* we don't care if this copying failed */ 
>> -			run_command(&child); + 
>> run_command(&notes_cp); 
>>   hook_opt.path_to_stdin = rebase_path_rewritten_list(); 
>>  strvec_push(&hook_opt.args, "rebase"); 
> 
> This change looks completely unrelated to me? 

Yes, Phillip pointed this out as well, I will drop it in v2.

Thanks!

Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks

[Adrian Ratiu]

Hi Patrick and sorry for the delayed reply!

On Thu, 02 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Thu, Sep 25, 2025 at 03:53:48PM +0300, Adrian Ratiu wrote: 
>> diff --git a/refs.c b/refs.c index 4ff55cf24f..5a2b6ad1fc 
>> 100644 --- a/refs.c +++ b/refs.c @@ -2377,31 +2377,16 @@ static 
>> int ref_update_reject_duplicates(struct string_list *refnames, 
>>  	return 0; }  
>> -static int run_transaction_hook(struct ref_transaction 
>> *transaction, -				const char *state) 
>> +static int transaction_hook_feed_stdin(int hook_stdin_fd, void 
>> *pp_cb, void *pp_task_cb UNUSED) 
>>  { 
>> -	struct child_process proc = CHILD_PROCESS_INIT; + 
>> struct hook_cb_data *hook_cb = pp_cb; +	struct 
>> run_hooks_opt *opt = hook_cb->options; +	struct 
>> ref_transaction *transaction = opt->feed_pipe_ctx; 
>>  	struct strbuf buf = STRBUF_INIT; 
>> -	const char *hook; -	int ret = 0, i; - -	hook = 
>> find_hook(transaction->ref_store->repo, 
>> "reference-transaction"); -	if (!hook) -		return 
>> ret; - -	strvec_pushl(&proc.args, hook, state, NULL); - 
>> proc.in = -1; -	proc.stdout_to_stderr = 1; - 
>> proc.trace2_hook_name = "reference-transaction"; - -	ret = 
>> start_command(&proc); -	if (ret) -		return 
>> ret; - -	sigchain_push(SIGPIPE, SIG_IGN); 
>>   
>> -	for (i = 0; i < transaction->nr; i++) { +	for (int i 
>> = 0; i < transaction->nr; i++) { 
>>  		struct ref_update *update = 
>>  transaction->updates[i]; 
>> +		int ret; 
>>   if (update->flags & REF_LOG_ONLY) continue; 
> 
> Hm. In the "pre-push" hook you converted the callback to process 
> one ref per invocation. Why don't we do the same over here, with 
> one transaction per invocation? 
> 
> Not saying that either one of these is better, but it left me 
> puzzled why we use two different patterns now. 

Good catch! It's a good idea to make them consistent.

We should do it here like we did for pre-push, to avoid processing 
all data at once and risk blocking on the write side (one 
extreme).

However, writing just once/one-line per callback (the other 
extreme) might add unnecessary delay/waiting on the hook child 
side.

So I'll modify both to batch let's say 50-100 in one call, to 
ensure a good balance.

This ties into my previous message about how much data should we 
write in one batch to ensure good throughtput while also not 
blocking for too long. Hope it makes sense.

Will improve this in v2.

Again, many thanks for your careful review, really appreciate it.

Re: [PATCH 02/10] hook: provide stdin via callback

[Emily Shaffer]

On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu <adrian.ratiu@collabora.com> wrote:
>
> From: Emily Shaffer <emilyshaffer@google.com>
>
> This adds a callback mechanism for feeding stdin to hooks alongside
> the existing path_to_stdin (which slurps a file's content to stdin).
>
> The advantage of this new callback is that it can feed stdin without
> going through the FS layer. This helps when feeding large amount of
> data and uses the run-command parallel stdin callback introduced in
> the preceding commit.
>
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
> ---
>  hook.c |  8 ++++++++
>  hook.h | 22 ++++++++++++++++++++++
>  2 files changed, 30 insertions(+)
>
> diff --git a/hook.c b/hook.c
> index b3de1048bf..54568d5bc0 100644
> --- a/hook.c
> +++ b/hook.c
> @@ -69,6 +69,10 @@ static int pick_next_hook(struct child_process *cp,
>         if (hook_cb->options->path_to_stdin) {
>                 cp->no_stdin = 0;
>                 cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
> +       } else if (hook_cb->options->feed_pipe) {
> +               cp->no_stdin = 0;
> +               /* start_command() will allocate a pipe / stdin fd for us */
> +               cp->in = -1;
>         }
>         cp->stdout_to_stderr = 1;
>         cp->trace2_hook_name = hook_cb->hook_name;
> @@ -140,6 +144,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
>
>                 .get_next_task = pick_next_hook,
>                 .start_failure = notify_start_failure,
> +               .feed_pipe = options->feed_pipe,
>                 .task_finished = notify_hook_finished,
>
>                 .data = &cb_data,
> @@ -148,6 +153,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
>         if (!options)
>                 BUG("a struct run_hooks_opt must be provided to run_hooks");
>
> +       if (options->path_to_stdin && options->feed_pipe)
> +               BUG("choose only one method to populate hook stdin");
> +
>         if (options->invoked_hook)
>                 *options->invoked_hook = 0;
>
> diff --git a/hook.h b/hook.h
> index 11863fa734..8fdbc8c673 100644
> --- a/hook.h
> +++ b/hook.h
> @@ -1,6 +1,7 @@
>  #ifndef HOOK_H
>  #define HOOK_H
>  #include "strvec.h"
> +#include "run-command.h"
>
>  struct repository;
>
> @@ -37,6 +38,24 @@ struct run_hooks_opt
>          * Path to file which should be piped to stdin for each hook.
>          */
>         const char *path_to_stdin;
> +
> +       /**
> +        * Callback to ask for more content to pipe to each hook stdin.
> +        *
> +        * If a hook needs to consume large quantities of data (e.g. a list of all refs received in a
> +        * client push), feeding data via in-memory strings or slurping to/from files via path_to_stdin
> +        * will not be efficient, so this callback allows for piecemeal reading and writing.
> +        *
> +        * Add initalization context to hook.feed_pipe_ctx.
> +        */
> +       feed_pipe_fn feed_pipe;
> +       void *feed_pipe_ctx;
> +
> +       /**
> +        * Use this to keep internal state for your feed_pipe_fn callback.
> +        * Only useful if you are using run_hooks_opt.feed_pipe. Otherwise, ignore it.
> +        */
> +       void *feed_pipe_cb_data;
>  };
>
>  #define RUN_HOOKS_OPT_INIT { \
> @@ -44,6 +63,9 @@ struct run_hooks_opt
>         .args = STRVEC_INIT, \
>  }
>
> +/**
> + * Callback data provided to feed_pipe_fn.
> + */

It looks like this comment was maybe a note to yourself? (Or a note to
myself, eons ago?) But hook_cb_data is used in all the parallel hook
callbacks, not just feed_pipe_fn, so I don't think this is accurate.


>  struct hook_cb_data {
>         /* rc reflects the cumulative failure state */
>         int rc;
> --
> 2.49.1
>

Re: [PATCH 08/10] receive-pack: convert 'update' hook to hook.h

[Emily Shaffer]

On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu <adrian.ratiu@collabora.com> wrote:
>
> From: Emily Shaffer <emilyshaffer@google.com>
>
> This makes use of the new sideband API in hook.h added in the
> preceding commit.
>
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  builtin/receive-pack.c | 60 +++++++++++++++++++++++++++++-------------
>  1 file changed, 41 insertions(+), 19 deletions(-)
>
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index 1113137a6f..d5192ce132 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -939,31 +939,53 @@ static int run_receive_hook(struct command *commands,
>         return status;
>  }
>
> -static int run_update_hook(struct command *cmd)
> +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
>  {
> -       struct child_process proc = CHILD_PROCESS_INIT;
> -       int code;
> -       const char *hook_path = find_hook(the_repository, "update");
> +       int keepalive_active = 0;
>
> -       if (!hook_path)
> -               return 0;
> +       if (keepalive_in_sec <= 0)
> +               use_keepalive = KEEPALIVE_NEVER;
> +       if (use_keepalive == KEEPALIVE_ALWAYS)
> +               keepalive_active = 1;

This hook wasn't using the keepalive at all before, right? What's the
reason to use it now? I am worried it might be going to a sideband
consumer who wasn't expecting it because it's not documented in
githooks.


>
> -       strvec_push(&proc.args, hook_path);
> -       strvec_push(&proc.args, cmd->ref_name);
> -       strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
> -       strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
> +       /* send a keepalive if there is no data to write */
> +       if (keepalive_active && !output->len) {
> +               static const char buf[] = "0005\1";
> +               write_or_die(1, buf, sizeof(buf) - 1);
> +               return;
> +       }
>
> -       proc.no_stdin = 1;
> -       proc.stdout_to_stderr = 1;
> -       proc.err = use_sideband ? -1 : 0;
> -       proc.trace2_hook_name = "update";
> +       if (use_keepalive == KEEPALIVE_AFTER_NUL && !keepalive_active) {
> +               const char *first_null = memchr(output->buf, '\0', output->len);
> +               if (first_null) {
> +                       /* The null bit is excluded. */
> +                       size_t before_null = first_null - output->buf;
> +                       size_t after_null = output->len - (before_null + 1);
> +                       keepalive_active = 1;
> +                       send_sideband(1, 2, output->buf, before_null, use_sideband);
> +                       send_sideband(1, 2, first_null + 1, after_null, use_sideband);
> +
> +                       return;
> +               }
> +       }
> +
> +       send_sideband(1, 2, output->buf, output->len, use_sideband);
> +}
> +
> +static int run_update_hook(struct command *cmd)
> +{
> +       struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
> +
> +       strvec_pushl(&opt.args,
> +                    cmd->ref_name,
> +                    oid_to_hex(&cmd->old_oid),
> +                    oid_to_hex(&cmd->new_oid),
> +                    NULL);
>
> -       code = start_command(&proc);
> -       if (code)
> -               return code;
>         if (use_sideband)
> -               copy_to_sideband(proc.err, -1, NULL);
> -       return finish_command(&proc);
> +               opt.consume_sideband = hook_output_to_sideband;
> +
> +       return run_hooks_opt(the_repository, "update", &opt);
>  }
>
>  static struct command *find_command_by_refname(struct command *list,
> --
> 2.49.1
>

Re: [PATCH 00/10] Convert remaining hooks to hook.h

[Emily Shaffer]

On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu <adrian.ratiu@collabora.com> wrote:
>
> Hello everyone,
>
> This is a continuation of Emily and Aevar's work to convert remaining hooks
> to the hook.h interface, by adding and using two new run-command/hook APIs:
>  * feeding hook stdin via a callback
>  * capturing server-side collated outputs
>
> I've tried to keep the implementations as simple as possible and avoid any
> unnecessary copying by feeding the data directly to the hook stdin fds and
> even batching the writes of pre/post-receive so we achieve similar perf/data/
> syscall efficiency as we had before the callback conversion.
>
> As suggested by Aevar [1], I've removed the string_list API, the extra copies
> and the $'\n' assumptions on the data, however I did not go the full zero-copy
> route with mmap-ing because I think that will break backwards compatbility. We
> could explore that in a future series as an efficientization of the current IPC,
> this patch series basically aims for parity with the existing implementation.
>
> This series also unblocks config-based hooks and hooks parallelization which will
> follow up in a separate series.
>
> The patch series is based on the master branch, I've pushed it to github [2] and
> it also passes CI runs. [3]. Also merged and tested against next with no conflicts.
>
> 1: https://lore.kernel.org/git/230209.86y1p7y4fa.gmgdl@evledraar.gmail.com/
> 2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v1
> 3: https://github.com/10ne1/git/actions/runs/18006589297

Thanks Adrian. For the most part I have only small concerns - not
surprising as these patches are mostly (not entirely) logically
equivalent to patches I wrote a few years ago and which we have been
running for Googlers since that time.

Others covered some of the comments I had to send, but I have a few
comments of my own too. The only one I'm truly stumped on is the
`update` hook conversion, we can discuss on that patch.

Otherwise, I'll look forward to the v2.


 - Emily

>
> Big warm thank you,
> Adrian
>
> Adrian Ratiu (1):
>   reference-transaction: use hook.h to run hooks
>
> Emily Shaffer (9):
>   run-command: add stdin callback for parallelization
>   hook: provide stdin via callback
>   hook: convert 'post-rewrite' hook in sequencer.c to hook.h
>   transport: convert pre-push hook to hook.h
>   run-command: allow capturing of collated output
>   hooks: allow callers to capture output
>   receive-pack: convert 'update' hook to hook.h
>   post-update: use hook.h library
>   receive-pack: convert receive hooks to hook.h
>
>  builtin/fetch.c             |   2 +-
>  builtin/receive-pack.c      | 310 +++++++++++++++++++-----------------
>  builtin/submodule--helper.c |   2 +-
>  hook.c                      |  11 +-
>  hook.h                      |  30 ++++
>  refs.c                      |  61 ++++---
>  run-command.c               | 115 +++++++++++--
>  run-command.h               |  44 ++++-
>  sequencer.c                 |  62 +++++---
>  submodule.c                 |   2 +-
>  t/helper/test-run-command.c |  67 +++++++-
>  t/t0061-run-command.sh      |  37 +++++
>  transport.c                 |  79 ++++-----
>  13 files changed, 547 insertions(+), 275 deletions(-)
>
> --
> 2.49.1
>

Re: [PATCH 02/10] hook: provide stdin via callback

[Adrian Ratiu]

On Fri, 10 Oct 2025, Emily Shaffer <nasamuffin@google.com> wrote:
> On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu 
> <adrian.ratiu@collabora.com> wrote: 
>> diff --git a/hook.h b/hook.h index 11863fa734..8fdbc8c673 
>> 100644 --- a/hook.h +++ b/hook.h @@ -1,6 +1,7 @@ 
>>  #ifndef HOOK_H #define HOOK_H #include "strvec.h" 
>> +#include "run-command.h" 
>> 
>>  struct repository; 
>> 
>> @@ -37,6 +38,24 @@ struct run_hooks_opt 
>>          * Path to file which should be piped to stdin for each 
>>          hook.  */ 
>>         const char *path_to_stdin; 
>> + +       /** +        * Callback to ask for more content to 
>> pipe to each hook stdin.  +        * +        * If a hook needs 
>> to consume large quantities of data (e.g. a list of all refs 
>> received in a +        * client push), feeding data via 
>> in-memory strings or slurping to/from files via path_to_stdin + 
>> * will not be efficient, so this callback allows for piecemeal 
>> reading and writing.  +        * +        * Add initalization 
>> context to hook.feed_pipe_ctx.  +        */ + 
>> feed_pipe_fn feed_pipe; +       void *feed_pipe_ctx; + + 
>> /** +        * Use this to keep internal state for your 
>> feed_pipe_fn callback.  +        * Only useful if you are using 
>> run_hooks_opt.feed_pipe. Otherwise, ignore it.  +        */ + 
>> void *feed_pipe_cb_data; 
>>  }; 
>> 
>>  #define RUN_HOOKS_OPT_INIT { \ 
>> @@ -44,6 +63,9 @@ struct run_hooks_opt 
>>         .args = STRVEC_INIT, \ 
>>  } 
>> 
>> +/** + * Callback data provided to feed_pipe_fn.  + */ 
> 
> It looks like this comment was maybe a note to yourself? (Or a 
> note to myself, eons ago?) But hook_cb_data is used in all the 
> parallel hook callbacks, not just feed_pipe_fn, so I don't think 
> this is accurate. 

Nice find! I added the comment while initially understanding the 
code and yes, hook_cb_data is more generic than just its use in 
feed_pipe_fn.

Will drop it in v2. Thanks!

Re: [PATCH 01/10] run-command: add stdin callback for parallelization

[Adrian Ratiu]

Hi again Patrick and Junio,

On Mon, 06 Oct 2025, Adrian Ratiu <adrian.ratiu@collabora.com> 
wrote:
> Hi Patrick and thanks for review! I'll fix in v2 all the issues 
> you pointed out. 
> 
> On Thu, 02 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote: 
>>> +		 * child input is provided via path_to_stdin when 
>>> the feed_pipe cb is +		 * missing, so we just 
>>> signal an EOF.  +		 */ +		if 
>>> (!opts->feed_pipe) { +			close(proc->in); + 
>>> proc->in = 0;  
>>  Hm. It's curious that we use a valid file descriptor 
>> here. Shouldn't we rather use `-1`? Otherwise I could see that 
>> we might try to close this seemingly valid file descriptor at a 
>> later point in time.   
> 
> I actually asked myself this while preparing the patches, since 
> -1  is a better fit. 
> 
> I only left = 0 for historical reasons, to not modify these 
> patches too much. :) However I do 100% agree with both you and 
> Junio that -1 should be used here. 
> 
> Will do in v2. 

This is much harder and riskier than I originally anticipated, so 
I gave up trying to implement it after a few failed attempts.

In a nutshell, we have to change the < 0, 0 and > 0 semantics 
defined in run-command.h for .in, .out, and .err fds across the 
entire source tree.

It's a massive, error-prone and out-of-scope amount of work.

Can we please just keep the current run-command API which uses 0 
for "no fd passed"? I'd very much like to avoid changing this 
run-command API.

Re: [PATCH 08/10] receive-pack: convert 'update' hook to hook.h

[Adrian Ratiu]

Hi Emily and sorry for the delayed response

On Fri, 10 Oct 2025, Emily Shaffer <nasamuffin@google.com> wrote:
> On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu 
> <adrian.ratiu@collabora.com> wrote: 
>> 
>> From: Emily Shaffer <emilyshaffer@google.com> 
>> 
>> This makes use of the new sideband API in hook.h added in the 
>> preceding commit. 
>> 
>> Signed-off-by: Emily Shaffer <emilyshaffer@google.com> 
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> --- 
>>  builtin/receive-pack.c | 60 
>>  +++++++++++++++++++++++++++++------------- 1 file changed, 41 
>>  insertions(+), 19 deletions(-) 
>> 
>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c 
>> index 1113137a6f..d5192ce132 100644 --- 
>> a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ 
>> -939,31 +939,53 @@ static int run_receive_hook(struct command 
>> *commands, 
>>         return status; 
>>  } 
>> 
>> -static int run_update_hook(struct command *cmd) +static void 
>> hook_output_to_sideband(struct strbuf *output, void *cb_data 
>> UNUSED) 
>>  { 
>> -       struct child_process proc = CHILD_PROCESS_INIT; - 
>> int code; -       const char *hook_path = 
>> find_hook(the_repository, "update"); +       int 
>> keepalive_active = 0; 
>> 
>> -       if (!hook_path) -               return 0; +       if 
>> (keepalive_in_sec <= 0) +               use_keepalive = 
>> KEEPALIVE_NEVER; +       if (use_keepalive == KEEPALIVE_ALWAYS) 
>> +               keepalive_active = 1; 
> 
> This hook wasn't using the keepalive at all before, right? 
> What's the reason to use it now? I am worried it might be going 
> to a sideband consumer who wasn't expecting it because it's not 
> documented in githooks. 

Indeed, I just picked this up from the branch I'm basing my work 
on [1] and haven't thought this through enough in v1. There was no 
keepalive before the hook conversion and really there should not 
be any need for it AFAICT (it's a short lived hook).

You raise an excellent point about the behavior change, so I'm 
inclined to remove it in v2. I will obviously test to confirm 
before posting v2.

[1] 
https://github.com/steadmon/git/commit/6d80376bea4e476b1af1d8649fe054cdfd9295dd

[PATCH v2 00/10] Convert remaining hooks to hook.h

[Adrian Ratiu]

Hello everyone,

This is v2 of the series which converts the remaining hooks to the new API.

I addressed all the feedback received in v1, with two small exceptions
(the ones starting with "Opted not to" in the below Changes list).

I had a minor conflict with an upstream change [1] which was trivial to fix.

I added 1 new commit and squashed together two commits (the simplified update
hooks), so in total it's still 10 patches.

The plan is to follow this up with another series which enables config-based
hooks and parallel hook execution, where possible.

As always this is based on the latest master branch, I've pushed it to GitHub
and ran the CI pipeline [3]. The Win+Meson "missing libgitcore.a" and doc
"invalid escape sequence" failures seem to be unrelated, since I get them
without these patches.

1: https://github.com/git/git/commit/22e7bc801cd9c5e5b5c4489b631be28e506fec42
2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v2
3: https://github.com/10ne1/git/actions/runs/18593709082

Changes between v1 -> v2:
* Added a new commit with a mechanism to override ungroup options (Junio)
* Addded a BUG if hook path_to_stdin and feed_pipe are both provided (Junio)
* The feed_pipe cb can be set independently from path_to_stdin (Junio)
* Simplified the post-rewrite callback (Patrick, Phillip and Junio)
* Document that hook caller owns the feed_pipe_ctx (Junio)
* Removed unnecessary "child" -> "notes_cp" renames (Phillip)
* Reuse strbuf inside pre-push cb to avoid multiple alloc (Phillip)
* Simplified pre-push hook cb logic (Phillip)
* Rewrote reference-transaction cb logic to mirror pre-push (Patrick)
* Simplified the update hook cb by removing the keepalive logic (Emily)
* Squashed the simplified update and post-update conversions
* Iterator types, if conditions and other small fixes (Patrick)
* Fixed a conflict in refs.c with an upstream for loop sign compare check
* Opted not to use -1 to signify no fd value instead of 0, because I'd have to
  significantly rework the run-command.h .in/.out/.err API (Patrick)
* Opted not to move sigchain_push(SIGPIPE, SIG_IGN); into pp_buffer_stdin())
  because it will called too many times inside the process loop (Patrick)
* Added Helped-by: Emily Shaffer tag to the reference-transaction coversion
* Comments, typos, stray lines, commit rewordings (Ben, Patrick, Emily, Junio)

Adrian Ratiu (2):
  reference-transaction: use hook API instead of run-command
  hook: allow overriding the ungroup option

Emily Shaffer (8):
  run-command: add stdin callback for parallelization
  hook: provide stdin via callback
  hook: convert 'post-rewrite' hook in sequencer.c to hook API
  transport: convert pre-push to hook API
  run-command: allow capturing of collated output
  hooks: allow callers to capture output
  receive-pack: convert update hooks to new API
  receive-pack: convert receive hooks to hook API

 builtin/hook.c              |   6 +
 builtin/receive-pack.c      | 289 +++++++++++++++++-------------------
 commit.c                    |   3 +
 hook.c                      |  21 ++-
 hook.h                      |  36 +++++
 refs.c                      | 101 +++++++------
 run-command.c               | 110 ++++++++++++--
 run-command.h               |  42 ++++++
 sequencer.c                 |  42 +++---
 t/helper/test-run-command.c |  67 ++++++++-
 t/t0061-run-command.sh      |  38 +++++
 transport.c                 |  83 +++++------
 12 files changed, 557 insertions(+), 281 deletions(-)

-- 
2.49.1

[PATCH v2 01/10] run-command: add stdin callback for parallelization

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

If a user of the run_processes_parallel() API wants to pipe a large
amount of information to the stdin of each parallel command, that
data could exceed the pipe buffer of the process's stdin and can be
too big to store in-memory via strbuf & friends or to slurp to a file.

Generally this is solved by repeatedly writing to child_process.in
between calls to start_command() and finish_command(). For a specific
pre-existing example of this, see transport.c:run_pre_push_hook().

This adds a generic callback API to run_processes_parallel() to do
exactly that in a unified manner, similar to the existing callback APIs,
which can then be used by hooks.h to convert the remaining hooks to the
new, simpler parallel interface.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 82 +++++++++++++++++++++++++++++++++----
 run-command.h               | 22 ++++++++++
 t/helper/test-run-command.c | 52 ++++++++++++++++++++++-
 t/t0061-run-command.sh      | 31 ++++++++++++++
 4 files changed, 178 insertions(+), 9 deletions(-)

diff --git a/run-command.c b/run-command.c
index ed9575bd6a..5bc6db5bb1 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
+static void pp_buffer_stdin(struct parallel_processes *pp,
+			    const struct run_process_parallel_opts *opts)
+{
+	/* Buffer stdin for each pipe. */
+	for (ssize_t i = 0; i < opts->processes; i++) {
+		struct child_process *proc = &pp->children[i].process;
+		int ret;
+
+		if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)
+			continue;
+
+		/*
+		 * child input is provided via path_to_stdin when the feed_pipe cb is
+		 * missing, so we just signal an EOF.
+		 */
+		if (!opts->feed_pipe) {
+			close(proc->in);
+			proc->in = 0;
+			continue;
+		}
+
+		/**
+		 * Feed the pipe:
+		 *   ret < 0 means error
+		 *   ret == 0 means there is more data to be fed
+		 *   ret > 0 means feeding finished
+		 */
+		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
+		if (ret < 0)
+			die_errno("feed_pipe");
+
+		if (ret) {
+			close(proc->in);
+			proc->in = 0;
+		}
+	}
+}
+
 static void pp_buffer_stderr(struct parallel_processes *pp,
 			     const struct run_process_parallel_opts *opts,
 			     int output_timeout)
@@ -1722,6 +1760,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		pp->children[i].state = GIT_CP_FREE;
 		if (pp->pfd)
 			pp->pfd[i].fd = -1;
+		pp->children[i].process.in = 0;
 		child_process_init(&pp->children[i].process);
 
 		if (opts->ungroup) {
@@ -1756,6 +1795,32 @@ static int pp_collect_finished(struct parallel_processes *pp,
 	return result;
 }
 
+static void pp_handle_child_IO(struct parallel_processes *pp,
+				const struct run_process_parallel_opts *opts,
+				int output_timeout)
+{
+	/*
+	 * First push input, if any (it might no-op), to child tasks to avoid them blocking
+	 * after input. This also prevents deadlocks when ungrouping below, if a child blocks
+	 * while the parent also waits for them to finish.
+	 */
+	pp_buffer_stdin(pp, opts);
+
+	if (opts->ungroup) {
+		for (size_t i = 0; i < opts->processes; i++) {
+			int child_ready_for_cleanup =
+				pp->children[i].state == GIT_CP_WORKING &&
+				pp->children[i].process.in == 0;
+
+			if (child_ready_for_cleanup)
+				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
+		}
+	} else {
+		pp_buffer_stderr(pp, opts, output_timeout);
+		pp_output(pp);
+	}
+}
+
 void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
@@ -1775,6 +1840,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	/*
+	 * Child tasks might receive input via stdin, terminating early (or not), so
+	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
+	 * actually writes the data to children stdin fds.
+	 */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
 	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (i = 0;
@@ -1792,13 +1864,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts->ungroup) {
-			for (size_t i = 0; i < opts->processes; i++)
-				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
-		} else {
-			pp_buffer_stderr(&pp, opts, output_timeout);
-			pp_output(&pp);
-		}
+		pp_handle_child_IO(&pp, opts, output_timeout);
 		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
@@ -1809,6 +1875,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 
 	pp_cleanup(&pp, opts);
 
+	sigchain_pop(SIGPIPE);
+
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
diff --git a/run-command.h b/run-command.h
index 0df25e445f..e536ed7544 100644
--- a/run-command.h
+++ b/run-command.h
@@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * This callback is repeatedly called on every child process who requests
+ * start_command() to create a pipe by setting child_process.in < 0.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel, and
+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.
+ * The contents of 'send' will be read into the pipe and passed to the pipe.
+ *
+ * Returns < 0 for error
+ * Returns == 0 when there is more data to be fed (will be called again)
+ * Returns > 0 when finished (child closed fd or no more data to be fed)
+ */
+typedef int (*feed_pipe_fn)(int child_in,
+				void *pp_cb,
+				void *pp_task_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -473,6 +489,12 @@ struct run_process_parallel_opts
 	 */
 	start_failure_fn start_failure;
 
+	/*
+	 * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any
+	 * special handling.
+	 */
+	feed_pipe_fn feed_pipe;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 3719f23cc2..dfdb03b3ab 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -23,19 +23,26 @@ static int number_callbacks;
 static int parallel_next(struct child_process *cp,
 			 struct strbuf *err,
 			 void *cb,
-			 void **task_cb UNUSED)
+			 void **task_cb)
 {
 	struct child_process *d = cb;
 	if (number_callbacks >= 4)
 		return 0;
 
 	strvec_pushv(&cp->args, d->args.v);
+	cp->in = d->in;
+	cp->no_stdin = d->no_stdin;
 	if (err)
 		strbuf_addstr(err, "preloaded output of a child\n");
 	else
 		fprintf(stderr, "preloaded output of a child\n");
 
 	number_callbacks++;
+
+	/* test_stdin callback will use this to count remaining lines */
+	*task_cb = xmalloc(sizeof(int));
+	*(int*)(*task_cb) = 2;
+
 	return 1;
 }
 
@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
-			 void *pp_task_cb UNUSED)
+			 void *pp_task_cb)
 {
 	if (err)
 		strbuf_addstr(err, "asking for a quick stop\n");
 	else
 		fprintf(stderr, "asking for a quick stop\n");
+	if (pp_task_cb)
+		FREE_AND_NULL(pp_task_cb);
 	return 1;
 }
 
+static int task_finished_quiet(int result UNUSED,
+			       struct strbuf *err UNUSED,
+			       void *pp_cb UNUSED,
+			       void *pp_task_cb)
+{
+	if (pp_task_cb)
+		FREE_AND_NULL(pp_task_cb);
+	return 0;
+}
+
+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)
+{
+	int *lines_remaining = task_cb;
+
+	if (*lines_remaining) {
+		struct strbuf buf = STRBUF_INIT;
+		strbuf_addf(&buf, "sample stdin %d\n", --(*lines_remaining));
+		if (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {
+			if (errno == EPIPE) {
+				/* child closed stdin, nothing more to do */
+				strbuf_release(&buf);
+				return 1;
+			}
+			die_errno("write");
+		}
+		strbuf_release(&buf);
+	}
+
+	return !(*lines_remaining);
+}
+
 struct testsuite {
 	struct string_list tests, failed;
 	int next;
@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)
 	struct run_process_parallel_opts opts = {
 		.get_next_task = next_test,
 		.start_failure = test_failed,
+		.feed_pipe = test_stdin_pipe_feed,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
 		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		opts.get_next_task = no_job;
 		opts.task_finished = task_finished;
+	} else if (!strcmp(argv[1], "run-command-stdin")) {
+		proc.in = -1;
+		proc.no_stdin = 0;
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
+		opts.feed_pipe = test_stdin_pipe_feed;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 76d4936a87..2f77fde0d9 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command listens to stdin' '
+	cat >expect <<-\EOF &&
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	EOF
+
+	write_script stdin-script <<-\EOF &&
+	echo "listening for stdin:"
+	while read line
+	do
+		echo "$line"
+	done
+	EOF
+	test-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&
+	test_cmp expect actual
+'
+
 cat >expect <<-EOF
 preloaded output of a child
 asking for a quick stop
-- 
2.49.1

[PATCH v2 02/10] hook: provide stdin via callback

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This adds a callback mechanism for feeding stdin to hooks alongside
the existing path_to_stdin (which slurps a file's content to stdin).

The advantage of this new callback is that it can feed stdin without
going through the FS layer. This helps when feeding large amount of
data and uses the run-command parallel stdin callback introduced in
the preceding commit.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 hook.c | 15 +++++++++++++++
 hook.h | 23 +++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/hook.c b/hook.c
index b3de1048bf..7537cf0f9e 100644
--- a/hook.c
+++ b/hook.c
@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,
 
 	cp->no_stdin = 1;
 	strvec_pushv(&cp->env, hook_cb->options->env.v);
+
+	if (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	/* reopen the file for stdin; run_command closes it. */
 	if (hook_cb->options->path_to_stdin) {
 		cp->no_stdin = 0;
 		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
 	}
+
+	if (hook_cb->options->feed_pipe) {
+		cp->no_stdin = 0;
+		/* start_command() will allocate a pipe / stdin fd for us */
+		cp->in = -1;
+	}
+
 	cp->stdout_to_stderr = 1;
 	cp->trace2_hook_name = hook_cb->hook_name;
 	cp->dir = hook_cb->options->dir;
@@ -140,6 +151,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
+		.feed_pipe = options->feed_pipe,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
@@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
 
+	if (options->path_to_stdin && options->feed_pipe)
+		BUG("choose only one method to populate hook stdin");
+
 	if (options->invoked_hook)
 		*options->invoked_hook = 0;
 
diff --git a/hook.h b/hook.h
index 11863fa734..ebe5dc450e 100644
--- a/hook.h
+++ b/hook.h
@@ -1,6 +1,7 @@
 #ifndef HOOK_H
 #define HOOK_H
 #include "strvec.h"
+#include "run-command.h"
 
 struct repository;
 
@@ -37,6 +38,28 @@ struct run_hooks_opt
 	 * Path to file which should be piped to stdin for each hook.
 	 */
 	const char *path_to_stdin;
+
+	/**
+	 * Callback to ask for more content to pipe to each hook stdin.
+	 *
+	 * If a hook needs to consume large quantities of data (e.g. a
+	 * list of all refs received in a client push), feeding data via
+	 * in-memory strings or slurping to/from files via path_to_stdin
+	 * is inefficient, so this callback allows for piecemeal writes.
+	 *
+	 * Add initalization context to hook.feed_pipe_ctx.
+	 *
+	 * The caller owns hook.feed_pipe_ctx and has to release any
+	 * resources after hooks finish execution.
+	 */
+	feed_pipe_fn feed_pipe;
+	void *feed_pipe_ctx;
+
+	/**
+	 * Use this to keep internal state for your feed_pipe_fn callback.
+	 * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.
+	 */
+	void *feed_pipe_cb_data;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.49.1

[PATCH v2 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Replace the custom run-command calls used by post-rewrite with
the newer and simpler hook_run_opt(), which does not need to
create a custom 'struct child_process' or call find_hook().

Another benefit of using the hook API is that hook_run_opt()
handles the SIGPIPE toggle logic.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 sequencer.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 5476d39ba9..71ed31c774 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,
 	return ret;
 }
 
+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
+{
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
+	int ret;
+
+	if (!to_pipe)
+		BUG("pipe_from_strbuf called without feed_pipe_ctx");
+
+	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
+
+	return 1; /* done writing */
+}
+
 static int run_rewrite_hook(const struct object_id *oldoid,
 			    const struct object_id *newoid)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 	int code;
 	struct strbuf sb = STRBUF_INIT;
-	const char *hook_path = find_hook(the_repository, "post-rewrite");
 
-	if (!hook_path)
-		return 0;
+	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
 
-	strvec_pushl(&proc.args, hook_path, "amend", NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "post-rewrite";
+	opt.feed_pipe_ctx = &sb;
+	opt.feed_pipe = pipe_from_strbuf;
+
+	strvec_push(&opt.args, "amend");
+
+	code = run_hooks_opt(the_repository, "post-rewrite", &opt);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
-	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
-	sigchain_push(SIGPIPE, SIG_IGN);
-	write_in_full(proc.in, sb.buf, sb.len);
-	close(proc.in);
 	strbuf_release(&sb);
-	sigchain_pop(SIGPIPE);
-	return finish_command(&proc);
+	return code;
 }
 
 void commit_post_rewrite(struct repository *r,
-- 
2.49.1

[PATCH v2 04/10] transport: convert pre-push to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Move the pre-push hook from custom run-command invocations to
the new hook API which doesn't require a custom child_process
structure and signal toggling.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 transport.c | 83 ++++++++++++++++++++++++-----------------------------
 1 file changed, 37 insertions(+), 46 deletions(-)

diff --git a/transport.c b/transport.c
index c7f06a7382..67368754bf 100644
--- a/transport.c
+++ b/transport.c
@@ -1316,65 +1316,56 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
 	die(_("Aborting."));
 }
 
-static int run_pre_push_hook(struct transport *transport,
-			     struct ref *remote_refs)
+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	int ret = 0, x;
-	struct ref *r;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct strbuf buf;
-	const char *hook_path = find_hook(the_repository, "pre-push");
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct ref *r = hook_cb->options->feed_pipe_ctx;
+	struct strbuf *buf = hook_cb->options->feed_pipe_cb_data;
+	int ret = 0;
 
-	if (!hook_path)
-		return 0;
+	if (!r)
+		return 1; /* no more refs */
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, transport->remote->name);
-	strvec_push(&proc.args, transport->url);
+	if (!buf)
+		BUG("pipe_task_cb must contain a valid strbuf");
 
-	proc.in = -1;
-	proc.trace2_hook_name = "pre-push";
+	hook_cb->options->feed_pipe_ctx = r->next;
+	strbuf_reset(buf);
 
-	if (start_command(&proc)) {
-		finish_command(&proc);
-		return -1;
-	}
-
-	sigchain_push(SIGPIPE, SIG_IGN);
+	if (!r->peer_ref) return 0;
+	if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;
+	if (r->status == REF_STATUS_REJECT_STALE) return 0;
+	if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;
+	if (r->status == REF_STATUS_UPTODATE) return 0;
 
-	strbuf_init(&buf, 256);
+	strbuf_addf(buf, "%s %s %s %s\n",
+		    r->peer_ref->name, oid_to_hex(&r->new_oid),
+		    r->name, oid_to_hex(&r->old_oid));
 
-	for (r = remote_refs; r; r = r->next) {
-		if (!r->peer_ref) continue;
-		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
-		if (r->status == REF_STATUS_REJECT_STALE) continue;
-		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
-		if (r->status == REF_STATUS_UPTODATE) continue;
+	ret = write_in_full(hook_stdin_fd, buf->buf, buf->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret; /* We do not mind if a hook does not read all refs. */
 
-		strbuf_reset(&buf);
-		strbuf_addf( &buf, "%s %s %s %s\n",
-			 r->peer_ref->name, oid_to_hex(&r->new_oid),
-			 r->name, oid_to_hex(&r->old_oid));
+	return 0;
+}
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			/* We do not mind if a hook does not read all refs. */
-			if (errno != EPIPE)
-				ret = -1;
-			break;
-		}
-	}
+static int run_pre_push_hook(struct transport *transport,
+			     struct ref *remote_refs)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct strbuf buf = STRBUF_INIT;
+	int ret = 0;
 
-	strbuf_release(&buf);
+	strvec_push(&opt.args, transport->remote->name);
+	strvec_push(&opt.args, transport->url);
 
-	x = close(proc.in);
-	if (!ret)
-		ret = x;
+	opt.feed_pipe = pre_push_hook_feed_stdin;
+	opt.feed_pipe_ctx = remote_refs;
+	opt.feed_pipe_cb_data = &buf;
 
-	sigchain_pop(SIGPIPE);
+	ret = run_hooks_opt(the_repository, "pre-push", &opt);
 
-	x = finish_command(&proc);
-	if (!ret)
-		ret = x;
+	strbuf_release(&buf);
 
 	return ret;
 }
-- 
2.49.1

[PATCH v2 05/10] reference-transaction: use hook API instead of run-command

[Adrian Ratiu]

Convert the reference-transaction hook to the new hook API,
so it doesn't need to set up a struct child_process, call
find_hook or toggle the pipe signals.

The stdin feed callback is processing one ref update per
call. I haven't noticed any performance degradation due
to this, however we can batch as many we want in each call,
to ensure a good pipe throughtput (i.e. the child does not
wait after stdin).

Helped-by: Emily Shaffer <nasamuffin@google.com>
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 refs.c | 101 ++++++++++++++++++++++++++++++---------------------------
 1 file changed, 53 insertions(+), 48 deletions(-)

diff --git a/refs.c b/refs.c
index 965381367e..5b042ff89b 100644
--- a/refs.c
+++ b/refs.c
@@ -2405,68 +2405,73 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-static int run_transaction_hook(struct ref_transaction *transaction,
-				const char *state)
+struct transaction_feed_cb_data {
+	size_t index;
+	struct strbuf buf;
+};
+
+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct strbuf buf = STRBUF_INIT;
-	const char *hook;
-	int ret = 0;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct run_hooks_opt *opt = hook_cb->options;
+	struct ref_transaction *transaction = opt->feed_pipe_ctx;
+	struct transaction_feed_cb_data *feed_cb_data = opt->feed_pipe_cb_data;
+	struct strbuf *buf = &feed_cb_data->buf;
+	struct ref_update *update;
+	size_t i = feed_cb_data->index++;
+	int ret;
 
-	hook = find_hook(transaction->ref_store->repo, "reference-transaction");
-	if (!hook)
-		return ret;
+	if (i >= transaction->nr)
+		return 1; /* No more refs to process */
 
-	strvec_pushl(&proc.args, hook, state, NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "reference-transaction";
+	update = transaction->updates[i];
 
-	ret = start_command(&proc);
-	if (ret)
-		return ret;
+	if (update->flags & REF_LOG_ONLY)
+		return 0;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	strbuf_reset(buf);
 
-	for (size_t i = 0; i < transaction->nr; i++) {
-		struct ref_update *update = transaction->updates[i];
+	if (!(update->flags & REF_HAVE_OLD))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->old_target)
+		strbuf_addf(buf, "ref:%s ", update->old_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->old_oid));
 
-		if (update->flags & REF_LOG_ONLY)
-			continue;
+	if (!(update->flags & REF_HAVE_NEW))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->new_target)
+		strbuf_addf(buf, "ref:%s ", update->new_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->new_oid));
 
-		strbuf_reset(&buf);
+	strbuf_addf(buf, "%s\n", update->refname);
 
-		if (!(update->flags & REF_HAVE_OLD))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->old_target)
-			strbuf_addf(&buf, "ref:%s ", update->old_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->old_oid));
+	ret = write_in_full(hook_stdin_fd, buf->buf, buf->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
 
-		if (!(update->flags & REF_HAVE_NEW))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->new_target)
-			strbuf_addf(&buf, "ref:%s ", update->new_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->new_oid));
+	return 0; /* no more input to feed */
+}
+
+static int run_transaction_hook(struct ref_transaction *transaction,
+				const char *state)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct transaction_feed_cb_data feed_ctx = { 0 };
+	int ret = 0;
 
-		strbuf_addf(&buf, "%s\n", update->refname);
+	strvec_push(&opt.args, state);
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			if (errno != EPIPE) {
-				/* Don't leak errno outside this API */
-				errno = 0;
-				ret = -1;
-			}
-			break;
-		}
-	}
+	opt.feed_pipe = transaction_hook_feed_stdin;
+	opt.feed_pipe_ctx = transaction;
+	opt.feed_pipe_cb_data = &feed_ctx;
 
-	close(proc.in);
-	sigchain_pop(SIGPIPE);
-	strbuf_release(&buf);
+	strbuf_init(&feed_ctx.buf, 0);
+
+	ret = run_hooks_opt(transaction->ref_store->repo, "reference-transaction", &opt);
 
-	ret |= finish_command(&proc);
+	strbuf_release(&feed_ctx.buf);
 	return ret;
 }
 
-- 
2.49.1

[PATCH v2 06/10] hook: allow overriding the ungroup option

[Adrian Ratiu]

When calling run_process_parallel() in run_hooks_opt(), the
ungroup option is currently hardcoded to .ungroup = 1.

This causes problems when ungrouping should be disabled, for
example when sideband-reading collated output from child hooks,
because sideband-reading and ungrouping are mutually exclusive.

Thus a new hook.h option is added to allow overriding.

The existing ungroup=1 behavior is preserved in the run_hooks()
API and the "hook run" command. We could modify these to take
an option if necessary, so I added two code comments there.

Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/hook.c | 6 ++++++
 commit.c       | 3 +++
 hook.c         | 5 ++++-
 hook.h         | 5 +++++
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/builtin/hook.c b/builtin/hook.c
index 7afec380d2..73e7b8c2e8 100644
--- a/builtin/hook.c
+++ b/builtin/hook.c
@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,
 	if (!argc)
 		goto usage;
 
+	/*
+	 * All current "hook run" use-cases require ungrouped child output.
+	 * If this changes, a hook run argument can be added to toggle it.
+	 */
+	opt.ungroup = 1;
+
 	/*
 	 * Having a -- for "run" when providing <hook-args> is
 	 * mandatory.
diff --git a/commit.c b/commit.c
index 16d91b2bfc..7da33dde86 100644
--- a/commit.c
+++ b/commit.c
@@ -1965,6 +1965,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,
 		strvec_push(&opt.args, arg);
 	va_end(args);
 
+	/* All commit hook use-cases require ungrouping child output. */
+	opt.ungroup = 1;
+
 	opt.invoked_hook = invoked_hook;
 	return run_hooks_opt(the_repository, name, &opt);
 }
diff --git a/hook.c b/hook.c
index 7537cf0f9e..a325c7cb8c 100644
--- a/hook.c
+++ b/hook.c
@@ -147,7 +147,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.tr2_label = hook_name,
 
 		.processes = 1,
-		.ungroup = 1,
+		.ungroup = options->ungroup,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
@@ -192,6 +192,9 @@ int run_hooks(struct repository *r, const char *hook_name)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
+	/* All use-cases of this API require ungrouping. */
+	opt.ungroup = 1;
+
 	return run_hooks_opt(r, hook_name, &opt);
 }
 
diff --git a/hook.h b/hook.h
index ebe5dc450e..82b3d1dd27 100644
--- a/hook.h
+++ b/hook.h
@@ -34,6 +34,11 @@ struct run_hooks_opt
 	 */
 	int *invoked_hook;
 
+	/**
+	 * Allow hooks to set run_processes_parallel() 'ungroup' behavior.
+	 */
+	unsigned int ungroup:1;
+
 	/**
 	 * Path to file which should be piped to stdin for each hook.
 	 */
-- 
2.49.1

[PATCH v2 07/10] run-command: allow capturing of collated output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some callers, for example server-side hooks which wish to relay hook
output to clients across a transport, want to capture what would
normally print to stderr and do something else with it. Allow that via a
callback.

By calling the callback regardless of whether there's output available,
we allow clients to send e.g. a keepalive if necessary.

Because we expose a strbuf, not a fd or FILE*, there's no need to create
a temporary pipe or similar - we can just skip the print to stderr and
instead hand it to the caller.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 30 ++++++++++++++++++++++--------
 run-command.h               | 20 ++++++++++++++++++++
 t/helper/test-run-command.c | 15 +++++++++++++++
 t/t0061-run-command.sh      |  7 +++++++
 4 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/run-command.c b/run-command.c
index 5bc6db5bb1..f217adcad6 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1578,7 +1578,10 @@ static void pp_cleanup(struct parallel_processes *pp,
 	 * When get_next_task added messages to the buffer in its last
 	 * iteration, the buffered output is non empty.
 	 */
-	strbuf_write(&pp->buffered_output, stderr);
+	if (opts->consume_sideband)
+		opts->consume_sideband(&pp->buffered_output, opts->data);
+	else
+		strbuf_write(&pp->buffered_output, stderr);
 	strbuf_release(&pp->buffered_output);
 
 	sigchain_pop_common();
@@ -1717,13 +1720,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 	}
 }
 
-static void pp_output(const struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp,
+		      const struct run_process_parallel_opts *opts)
 {
 	size_t i = pp->output_owner;
 
 	if (pp->children[i].state == GIT_CP_WORKING &&
 	    pp->children[i].err.len) {
-		strbuf_write(&pp->children[i].err, stderr);
+		if (opts->consume_sideband)
+			opts->consume_sideband(&pp->children[i].err, opts->data);
+		else
+			strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
 	}
 }
@@ -1771,11 +1778,15 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		} else {
 			const size_t n = opts->processes;
 
-			strbuf_write(&pp->children[i].err, stderr);
+			/* Output errors, then all other finished child processes */
+			if (opts->consume_sideband) {
+				opts->consume_sideband(&pp->children[i].err, opts->data);
+				opts->consume_sideband(&pp->buffered_output, opts->data);
+			} else {
+				strbuf_write(&pp->children[i].err, stderr);
+				strbuf_write(&pp->buffered_output, stderr);
+			}
 			strbuf_reset(&pp->children[i].err);
-
-			/* Output all other finished child processes */
-			strbuf_write(&pp->buffered_output, stderr);
 			strbuf_reset(&pp->buffered_output);
 
 			/*
@@ -1817,7 +1828,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,
 		}
 	} else {
 		pp_buffer_stderr(pp, opts, output_timeout);
-		pp_output(pp);
+		pp_output(pp, opts);
 	}
 }
 
@@ -1840,6 +1851,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	if (opts->ungroup && opts->consume_sideband)
+		BUG("ungroup and reading sideband are mutualy exclusive");
+
 	/*
 	 * Child tasks might receive input via stdin, terminating early (or not), so
 	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
diff --git a/run-command.h b/run-command.h
index e536ed7544..2c2484478b 100644
--- a/run-command.h
+++ b/run-command.h
@@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * If this callback is provided, instead of collating process output to stderr,
+ * they will be collated into a new pipe. consume_sideband_fn will be called
+ * repeatedly. When output is available on that pipe, it will be contained in
+ * 'output'. But it will be called with an empty 'output' too, to allow for
+ * keepalives or similar operations if necessary.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel.
+ *
+ * Since this callback is provided with the collated output, no task cookie is
+ * provided.
+ */
+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -495,6 +509,12 @@ struct run_process_parallel_opts
 	 */
 	feed_pipe_fn feed_pipe;
 
+	/*
+	 * consume_sideband: see consume_sideband_fn() above. This can be NULL
+	 * to omit any special handling.
+	 */
+	consume_sideband_fn consume_sideband;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index dfdb03b3ab..95152a0395 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,
 	return 0;
 }
 
+static void test_consume_sideband(struct strbuf *output, void *cb UNUSED)
+{
+	FILE *sideband;
+
+	sideband = fopen("./sideband", "a");
+
+	strbuf_write(output, sideband);
+	fclose(sideband);
+}
+
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)
 		.get_next_task = next_test,
 		.start_failure = test_failed,
 		.feed_pipe = test_stdin_pipe_feed,
+		.consume_sideband = test_consume_sideband,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished_quiet;
 		opts.feed_pipe = test_stdin_pipe_feed;
+	} else if (!strcmp(argv[1], "run-command-sideband")) {
+		opts.get_next_task = parallel_next;
+		opts.consume_sideband = test_consume_sideband;
+		opts.task_finished = task_finished_quiet;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 2f77fde0d9..f133d71783 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command can divert output' '
+	test_when_finished rm sideband &&
+	test-tool run-command run-command-sideband 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
+	test_must_be_empty actual &&
+	test_cmp expect sideband
+'
+
 test_expect_success 'run_command listens to stdin' '
 	cat >expect <<-\EOF &&
 	preloaded output of a child
-- 
2.49.1

[PATCH v2 08/10] hooks: allow callers to capture output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some server-side hooks will require capturing output to send over
sideband instead of printing directly to stderr. Expose that capability.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c | 1 +
 hook.h | 8 ++++++++
 2 files changed, 9 insertions(+)

diff --git a/hook.c b/hook.c
index a325c7cb8c..fb452b5369 100644
--- a/hook.c
+++ b/hook.c
@@ -152,6 +152,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
 		.feed_pipe = options->feed_pipe,
+		.consume_sideband = options->consume_sideband,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
diff --git a/hook.h b/hook.h
index 82b3d1dd27..a84e97db34 100644
--- a/hook.h
+++ b/hook.h
@@ -65,6 +65,14 @@ struct run_hooks_opt
 	 * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.
 	 */
 	void *feed_pipe_cb_data;
+
+	/*
+	 * Populate this to capture output and prevent it from being printed to
+	 * stderr. This will be passed directly through to
+	 * run_command:run_parallel_processes(). See t/helper/test-run-command.c
+	 * for an example.
+	 */
+	consume_sideband_fn consume_sideband;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.49.1

[PATCH v2 09/10] receive-pack: convert update hooks to new API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Use the new hook sideband API introduced in the previous commit.

The hook API avoids creating a custom struct child_process and other
internal hook plumbing (e.g. calling find_hook()) and prepares for
the specification of hooks via configs or running parallel hooks.

Execution is still sequential through the current hook.[ch] via the
run_proces_parallel_opts.processes=1 arg.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 60 +++++++++++++++---------------------------
 1 file changed, 21 insertions(+), 39 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index c9288a9c7e..93b6f28662 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -939,31 +939,26 @@ static int run_receive_hook(struct command *commands,
 	return status;
 }
 
-static int run_update_hook(struct command *cmd)
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	int code;
-	const char *hook_path = find_hook(the_repository, "update");
-
-	if (!hook_path)
-		return 0;
+	if (output && output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
+}
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, cmd->ref_name);
-	strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
-	strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
+static int run_update_hook(struct command *cmd)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "update";
+	strvec_pushl(&opt.args,
+		     cmd->ref_name,
+		     oid_to_hex(&cmd->old_oid),
+		     oid_to_hex(&cmd->new_oid),
+		     NULL);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
 	if (use_sideband)
-		copy_to_sideband(proc.err, -1, NULL);
-	return finish_command(&proc);
+		opt.consume_sideband = hook_output_to_sideband;
+
+	return run_hooks_opt(the_repository, "update", &opt);
 }
 
 static struct command *find_command_by_refname(struct command *list,
@@ -1640,33 +1635,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 static void run_update_post_hook(struct command *commands)
 {
 	struct command *cmd;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	const char *hook;
-
-	hook = find_hook(the_repository, "post-update");
-	if (!hook)
-		return;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		if (cmd->error_string || cmd->did_not_exist)
 			continue;
-		if (!proc.args.nr)
-			strvec_push(&proc.args, hook);
-		strvec_push(&proc.args, cmd->ref_name);
+		strvec_push(&opt.args, cmd->ref_name);
 	}
-	if (!proc.args.nr)
+	if (!opt.args.nr)
 		return;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "post-update";
+	if (use_sideband)
+		opt.consume_sideband = hook_output_to_sideband;
 
-	if (!start_command(&proc)) {
-		if (use_sideband)
-			copy_to_sideband(proc.err, -1, NULL);
-		finish_command(&proc);
-	}
+	run_hooks_opt(the_repository, "post-update", &opt);
 }
 
 static void check_aliased_update_internal(struct command *cmd,
-- 
2.49.1

[PATCH v2 10/10] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This converts the last remaining hooks to the new hook API, for
the same benefits as the previous conversions (no need to toggle
signals, manage custom struct child_process, call find_hook(),
prepares for specifyinig hooks via configs, etc.).

I noticed a performance degradation when processing large amounts
of hook input with just 1 line per callback, due to run-command's
ppoll loop, therefore I batched 500 lines per callback, to ensure
similar pipe throughput as before and to avoid hook child waiting
on stdin.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 239 +++++++++++++++++++++--------------------
 1 file changed, 120 insertions(+), 119 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 93b6f28662..18b5f22d44 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)
 	return retval;
 }
 
-static void prepare_push_cert_sha1(struct child_process *proc)
+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)
 {
 	static int already_done;
 
@@ -775,147 +775,132 @@ static void prepare_push_cert_sha1(struct child_process *proc)
 		nonce_status = check_nonce(sigcheck.payload);
 	}
 	if (!is_null_oid(&push_cert_oid)) {
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT=%s",
 			     oid_to_hex(&push_cert_oid));
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_SIGNER=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_SIGNER=%s",
 			     sigcheck.signer ? sigcheck.signer : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_KEY=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_KEY=%s",
 			     sigcheck.key ? sigcheck.key : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_STATUS=%c",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_STATUS=%c",
 			     sigcheck.result);
 		if (push_cert_nonce) {
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE=%s",
 				     push_cert_nonce);
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE_STATUS=%s",
 				     nonce_status);
 			if (nonce_status == NONCE_SLOP)
-				strvec_pushf(&proc->env,
+				strvec_pushf(&opt->env,
 					     "GIT_PUSH_CERT_NONCE_SLOP=%ld",
 					     nonce_stamp_slop);
 		}
 	}
 }
 
+struct receive_hook_feed_context {
+	struct command *cmd;
+	int skip_broken;
+};
+
 struct receive_hook_feed_state {
 	struct command *cmd;
 	struct ref_push_report *report;
 	int skip_broken;
 	struct strbuf buf;
-	const struct string_list *push_options;
 };
 
-typedef int (*feed_fn)(void *, const char **, size_t *);
-static int run_and_feed_hook(const char *hook_name, feed_fn feed,
-			     struct receive_hook_feed_state *feed_state)
+static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct async muxer;
-	int code;
-	const char *hook_path = find_hook(the_repository, hook_name);
+	struct command *cmd = state->cmd;
 
-	if (!hook_path)
-		return 0;
+	strbuf_reset(&state->buf);
 
-	strvec_push(&proc.args, hook_path);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = hook_name;
-
-	if (feed_state->push_options) {
-		size_t i;
-		for (i = 0; i < feed_state->push_options->nr; i++)
-			strvec_pushf(&proc.env,
-				     "GIT_PUSH_OPTION_%"PRIuMAX"=%s",
-				     (uintmax_t)i,
-				     feed_state->push_options->items[i].string);
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
-			     (uintmax_t)feed_state->push_options->nr);
-	} else
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
+	/* batch lines to avoid going through run-command's ppoll for each line */
+	for (int i = 0; i < lines_batch_size; i++) {
+		while (cmd &&
+		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
+			cmd = cmd->next;
 
-	if (tmp_objdir)
-		strvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));
+		if (!cmd)
+			break;  /* no more commands left */
 
-	if (use_sideband) {
-		memset(&muxer, 0, sizeof(muxer));
-		muxer.proc = copy_to_sideband;
-		muxer.in = -1;
-		code = start_async(&muxer);
-		if (code)
-			return code;
-		proc.err = muxer.in;
-	}
+		if (!state->report)
+			state->report = cmd->report;
 
-	prepare_push_cert_sha1(&proc);
+		if (state->report) {
+			struct object_id *old_oid;
+			struct object_id *new_oid;
+			const char *ref_name;
 
-	code = start_command(&proc);
-	if (code) {
-		if (use_sideband)
-			finish_async(&muxer);
-		return code;
-	}
+			old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
+			new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
+			ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(old_oid), oid_to_hex(new_oid),
+				    ref_name);
 
-	while (1) {
-		const char *buf;
-		size_t n;
-		if (feed(feed_state, &buf, &n))
-			break;
-		if (write_in_full(proc.in, buf, n) < 0)
-			break;
+			state->report = state->report->next;
+			if (!state->report)
+				cmd = cmd->next;
+		} else {
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
+				    cmd->ref_name);
+			cmd = cmd->next;
+		}
 	}
-	close(proc.in);
-	if (use_sideband)
-		finish_async(&muxer);
 
-	sigchain_pop(SIGPIPE);
+	state->cmd = cmd;
 
-	return finish_command(&proc);
+	if (state->buf.len > 0) {
+		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
+		if (ret < 0) {
+			if (errno == EPIPE)
+				return 1; /* child closed pipe */
+			return ret;
+		}
+	}
+
+	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
 }
 
-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	struct receive_hook_feed_state *state = state_;
-	struct command *cmd = state->cmd;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;
 
-	while (cmd &&
-	       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
-		cmd = cmd->next;
-	if (!cmd)
-		return -1; /* EOF */
-	if (!bufp)
-		return 0; /* OK, can feed something. */
-	strbuf_reset(&state->buf);
-	if (!state->report)
-		state->report = cmd->report;
-	if (state->report) {
-		struct object_id *old_oid;
-		struct object_id *new_oid;
-		const char *ref_name;
-
-		old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
-		new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
-		ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(old_oid), oid_to_hex(new_oid),
-			    ref_name);
-		state->report = state->report->next;
-		if (!state->report)
-			state->cmd = cmd->next;
-	} else {
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
-			    cmd->ref_name);
-		state->cmd = cmd->next;
-	}
-	if (bufp) {
-		*bufp = state->buf.buf;
-		*sizep = state->buf.len;
+	/* first-time setup */
+	if (!hook_cb->options->feed_pipe_cb_data) {
+		struct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;
+		if (!ctx)
+			BUG("run_hooks_opt.feed_pipe_ctx required for receive hook");
+
+		hook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));
+		feed_state = hook_cb->options->feed_pipe_cb_data;
+		strbuf_init(&feed_state->buf, 0);
+		feed_state->cmd = ctx->cmd;
+		feed_state->skip_broken = ctx->skip_broken;
+		feed_state->report = NULL;
 	}
-	return 0;
+
+	/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */
+	if (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)
+		return 0; /* still have more data to feed */
+
+	strbuf_release(&feed_state->buf);
+
+	if (hook_cb->options->feed_pipe_cb_data)
+		FREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);
+
+	return 1; /* done feeding, run-command can close pipe */
+}
+
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
+{
+	if (output && output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
 }
 
 static int run_receive_hook(struct command *commands,
@@ -923,26 +908,42 @@ static int run_receive_hook(struct command *commands,
 			    int skip_broken,
 			    const struct string_list *push_options)
 {
-	struct receive_hook_feed_state state;
-	int status;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct receive_hook_feed_context ctx;
+	struct command *iter = commands;
 
-	strbuf_init(&state.buf, 0);
-	state.cmd = commands;
-	state.skip_broken = skip_broken;
-	state.report = NULL;
-	if (feed_receive_hook(&state, NULL, NULL))
+	/* if there are no valid commands, don't invoke the hook at all. */
+	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
+		iter = iter->next;
+	if (!iter)
 		return 0;
-	state.cmd = commands;
-	state.push_options = push_options;
-	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
-	strbuf_release(&state.buf);
-	return status;
-}
 
-static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
-{
-	if (output && output->len)
-		send_sideband(1, 2, output->buf, output->len, use_sideband);
+	if (push_options) {
+		int i;
+		for (i = 0; i < push_options->nr; i++)
+			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
+				     push_options->items[i].string);
+		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
+					     (uintmax_t)push_options->nr);
+	} else
+		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+
+	if (tmp_objdir)
+		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+
+	prepare_push_cert_sha1(&opt);
+
+	/* set up sideband printer */
+	if (use_sideband)
+		opt.consume_sideband = hook_output_to_sideband;
+
+	/* set up stdin callback */
+	ctx.cmd = commands;
+	ctx.skip_broken = skip_broken;
+	opt.feed_pipe = feed_receive_hook_cb;
+	opt.feed_pipe_ctx = &ctx;
+
+	return run_hooks_opt(the_repository, hook_name, &opt);
 }
 
 static int run_update_hook(struct command *cmd)
-- 
2.49.1

Re: [PATCH v2 00/10] Convert remaining hooks to hook.h

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:34PM +0300, Adrian Ratiu wrote:
> Hello everyone,
> 
> This is v2 of the series which converts the remaining hooks to the new API.
> 
> I addressed all the feedback received in v1, with two small exceptions
> (the ones starting with "Opted not to" in the below Changes list).
> 
> I had a minor conflict with an upstream change [1] which was trivial to fix.
> 
> I added 1 new commit and squashed together two commits (the simplified update
> hooks), so in total it's still 10 patches.
> 
> The plan is to follow this up with another series which enables config-based
> hooks and parallel hook execution, where possible.
> 
> As always this is based on the latest master branch, I've pushed it to GitHub
> and ran the CI pipeline [3]. The Win+Meson "missing libgitcore.a" and doc
> "invalid escape sequence" failures seem to be unrelated, since I get them
> without these patches.
> 
> 1: https://github.com/git/git/commit/22e7bc801cd9c5e5b5c4489b631be28e506fec42
> 2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v2
> 3: https://github.com/10ne1/git/actions/runs/18593709082
> 
> Changes between v1 -> v2:
> * Added a new commit with a mechanism to override ungroup options (Junio)
> * Addded a BUG if hook path_to_stdin and feed_pipe are both provided (Junio)
> * The feed_pipe cb can be set independently from path_to_stdin (Junio)
> * Simplified the post-rewrite callback (Patrick, Phillip and Junio)
> * Document that hook caller owns the feed_pipe_ctx (Junio)
> * Removed unnecessary "child" -> "notes_cp" renames (Phillip)
> * Reuse strbuf inside pre-push cb to avoid multiple alloc (Phillip)
> * Simplified pre-push hook cb logic (Phillip)
> * Rewrote reference-transaction cb logic to mirror pre-push (Patrick)
> * Simplified the update hook cb by removing the keepalive logic (Emily)
> * Squashed the simplified update and post-update conversions
> * Iterator types, if conditions and other small fixes (Patrick)
> * Fixed a conflict in refs.c with an upstream for loop sign compare check
> * Opted not to use -1 to signify no fd value instead of 0, because I'd have to
>   significantly rework the run-command.h .in/.out/.err API (Patrick)
> * Opted not to move sigchain_push(SIGPIPE, SIG_IGN); into pp_buffer_stdin())
>   because it will called too many times inside the process loop (Patrick)
> * Added Helped-by: Emily Shaffer tag to the reference-transaction coversion
> * Comments, typos, stray lines, commit rewordings (Ben, Patrick, Emily, Junio)

By the way, it would have helped to have a range-diff here so that it's
easy to see what exactly changed between the two different versions.
Could you maybe include that in subsequent iterations?

Thanks!

Patrick

Re: [PATCH v2 01/10] run-command: add stdin callback for parallelization

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:35PM +0300, Adrian Ratiu wrote:
> diff --git a/run-command.c b/run-command.c
> index ed9575bd6a..5bc6db5bb1 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,
>  	return 0;
>  }
>  
> +static void pp_buffer_stdin(struct parallel_processes *pp,
> +			    const struct run_process_parallel_opts *opts)
> +{
> +	/* Buffer stdin for each pipe. */
> +	for (ssize_t i = 0; i < opts->processes; i++) {

This should use `size_t` to match the type of `opts->processes`.

> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
> index 3719f23cc2..dfdb03b3ab 100644
> --- a/t/helper/test-run-command.c
> +++ b/t/helper/test-run-command.c
> @@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,
>  static int task_finished(int result UNUSED,
>  			 struct strbuf *err,
>  			 void *pp_cb UNUSED,
> -			 void *pp_task_cb UNUSED)
> +			 void *pp_task_cb)
>  {
>  	if (err)
>  		strbuf_addstr(err, "asking for a quick stop\n");
>  	else
>  		fprintf(stderr, "asking for a quick stop\n");
> +	if (pp_task_cb)
> +		FREE_AND_NULL(pp_task_cb);

Tiny nit: the conditional here is not needed.

>  	return 1;
>  }
>  
> +static int task_finished_quiet(int result UNUSED,
> +			       struct strbuf *err UNUSED,
> +			       void *pp_cb UNUSED,
> +			       void *pp_task_cb)
> +{
> +	if (pp_task_cb)
> +		FREE_AND_NULL(pp_task_cb);

Same over here.

Patrick

Re: [PATCH v2 02/10] hook: provide stdin via callback

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:36PM +0300, Adrian Ratiu wrote:
> diff --git a/hook.c b/hook.c
> index b3de1048bf..7537cf0f9e 100644
> --- a/hook.c
> +++ b/hook.c
> @@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,
>  
>  	cp->no_stdin = 1;
>  	strvec_pushv(&cp->env, hook_cb->options->env.v);
> +
> +	if (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)
> +		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
> +
>  	/* reopen the file for stdin; run_command closes it. */
>  	if (hook_cb->options->path_to_stdin) {
>  		cp->no_stdin = 0;
>  		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
>  	}
> +
> +	if (hook_cb->options->feed_pipe) {
> +		cp->no_stdin = 0;
> +		/* start_command() will allocate a pipe / stdin fd for us */
> +		cp->in = -1;
> +	}
> +
>  	cp->stdout_to_stderr = 1;
>  	cp->trace2_hook_name = hook_cb->hook_name;
>  	cp->dir = hook_cb->options->dir;
> @@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
>  	if (!options)
>  		BUG("a struct run_hooks_opt must be provided to run_hooks");
>  
> +	if (options->path_to_stdin && options->feed_pipe)
> +		BUG("choose only one method to populate hook stdin");

Nit: we may want to reuse the same error message here as we use above.

	BUG("options path_to_stdin and feed_pipe are mutually exclusive");

Patrick

Re: [PATCH v2 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:37PM +0300, Adrian Ratiu wrote:
> diff --git a/sequencer.c b/sequencer.c
> index 5476d39ba9..71ed31c774 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,
>  	return ret;
>  }
>  
> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
> +{
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;

Not sure, but shouldn't it be possible to set `opt.feed_pipe_cb_data`
instead and then access the context via `pp_task_cb` here instead of
having to reach into the struct?

Patrick

Re: [PATCH v2 02/10] hook: provide stdin via callback

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:36PM +0300, Adrian Ratiu wrote:
> diff --git a/hook.h b/hook.h
> index 11863fa734..ebe5dc450e 100644
> --- a/hook.h
> +++ b/hook.h
> @@ -37,6 +38,28 @@ struct run_hooks_opt
>  	 * Path to file which should be piped to stdin for each hook.
>  	 */
>  	const char *path_to_stdin;
> +
> +	/**
> +	 * Callback to ask for more content to pipe to each hook stdin.
> +	 *
> +	 * If a hook needs to consume large quantities of data (e.g. a
> +	 * list of all refs received in a client push), feeding data via
> +	 * in-memory strings or slurping to/from files via path_to_stdin
> +	 * is inefficient, so this callback allows for piecemeal writes.
> +	 *
> +	 * Add initalization context to hook.feed_pipe_ctx.
> +	 *
> +	 * The caller owns hook.feed_pipe_ctx and has to release any
> +	 * resources after hooks finish execution.
> +	 */
> +	feed_pipe_fn feed_pipe;
> +	void *feed_pipe_ctx;
> +
> +	/**
> +	 * Use this to keep internal state for your feed_pipe_fn callback.
> +	 * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.
> +	 */
> +	void *feed_pipe_cb_data;

Are these fields used as any of the callback arguments? If so, let's
document which of the fields they correspond to, as it's otherwise hard
to follow.

Patrick

Re: [PATCH v2 04/10] transport: convert pre-push to hook API

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:38PM +0300, Adrian Ratiu wrote:
> diff --git a/transport.c b/transport.c
> index c7f06a7382..67368754bf 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -1316,65 +1316,56 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
>  	die(_("Aborting."));
>  }
>  
> -static int run_pre_push_hook(struct transport *transport,
> -			     struct ref *remote_refs)
> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
>  {
> -	int ret = 0, x;
> -	struct ref *r;
> -	struct child_process proc = CHILD_PROCESS_INIT;
> -	struct strbuf buf;
> -	const char *hook_path = find_hook(the_repository, "pre-push");
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct ref *r = hook_cb->options->feed_pipe_ctx;
> +	struct strbuf *buf = hook_cb->options->feed_pipe_cb_data;

Same question here, isn't `feed_pipe_cb_data` accessible via
`pp_task_cb`? May very well be that I misunderstand the two callback
context and data, I found that part to be a bit hard to follow.

> +	int ret = 0;
>  
> -	if (!hook_path)
> -		return 0;
> +	if (!r)
> +		return 1; /* no more refs */
>  
> -	strvec_push(&proc.args, hook_path);
> -	strvec_push(&proc.args, transport->remote->name);
> -	strvec_push(&proc.args, transport->url);
> +	if (!buf)
> +		BUG("pipe_task_cb must contain a valid strbuf");
>  
> -	proc.in = -1;
> -	proc.trace2_hook_name = "pre-push";
> +	hook_cb->options->feed_pipe_ctx = r->next;

I think that the lines between the "task data" and "task context" are
being blurred here. I understood it so that the task data is what is
specific to the callback, and that data may be changed to keep track of
the state. Subsequent commits do it that way, so shouldn't we also treat
the context as immutable here and instead handle iteration via the data?

> +	strbuf_reset(buf);

Nit: it would make sense to move this reset down a bit close to the
first call that writes to it.

> -	if (start_command(&proc)) {
> -		finish_command(&proc);
> -		return -1;
> -	}
> -
> -	sigchain_push(SIGPIPE, SIG_IGN);
> +	if (!r->peer_ref) return 0;
> +	if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;
> +	if (r->status == REF_STATUS_REJECT_STALE) return 0;
> +	if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;
> +	if (r->status == REF_STATUS_UPTODATE) return 0;

Nit, feel free to ignore: this might read a tiny bit nicer with a switch
statement.

Patrick

Re: [PATCH v2 07/10] run-command: allow capturing of collated output

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:41PM +0300, Adrian Ratiu wrote:
> diff --git a/run-command.h b/run-command.h
> index e536ed7544..2c2484478b 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,
>  				void *pp_cb,
>  				void *pp_task_cb);
>  
> +/**
> + * If this callback is provided, instead of collating process output to stderr,
> + * they will be collated into a new pipe. consume_sideband_fn will be called
> + * repeatedly. When output is available on that pipe, it will be contained in
> + * 'output'. But it will be called with an empty 'output' too, to allow for
> + * keepalives or similar operations if necessary.
> + *
> + * pp_cb is the callback cookie as passed into run_processes_parallel.
> + *
> + * Since this callback is provided with the collated output, no task cookie is
> + * provided.
> + */
> +typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);

I think the interface overall makes sense, but isn't the wording we use
here very specific for hooks?

Taking a step back, what this thing seems to do is to take the hook's
output and customize how exactly we handle this. That isn't really
specific to any kind of "sideband", even though our use case in the hook
code does use this for sidebands.

So maybe we should call this `consume_output_fn` and adapt the rest of
the code accordingly? Because that's what we ultimately do here, IIUC.

Patrick

Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API

[Patrick Steinhardt]

On Fri, Oct 17, 2025 at 05:15:44PM +0300, Adrian Ratiu wrote:
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index 93b6f28662..18b5f22d44 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
[snip]
> +static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)

The `lines_batch_size` can be unsigned.

>  {
> -	struct child_process proc = CHILD_PROCESS_INIT;
> -	struct async muxer;
> -	int code;
> -	const char *hook_path = find_hook(the_repository, hook_name);
> +	struct command *cmd = state->cmd;
>  
> -	if (!hook_path)
> -		return 0;
> +	strbuf_reset(&state->buf);
>  
> -	strvec_push(&proc.args, hook_path);
> -	proc.in = -1;
> -	proc.stdout_to_stderr = 1;
> -	proc.trace2_hook_name = hook_name;
> -
> -	if (feed_state->push_options) {
> -		size_t i;
> -		for (i = 0; i < feed_state->push_options->nr; i++)
> -			strvec_pushf(&proc.env,
> -				     "GIT_PUSH_OPTION_%"PRIuMAX"=%s",
> -				     (uintmax_t)i,
> -				     feed_state->push_options->items[i].string);
> -		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
> -			     (uintmax_t)feed_state->push_options->nr);
> -	} else
> -		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
> +	/* batch lines to avoid going through run-command's ppoll for each line */
> +	for (int i = 0; i < lines_batch_size; i++) {

And if so, the index here should be unsigned, as well.

Patrick

Re: [PATCH v2 02/10] hook: provide stdin via callback

[Adrian Ratiu]

On Tue, 21 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Fri, Oct 17, 2025 at 05:15:36PM +0300, Adrian Ratiu wrote: 
>> diff --git a/hook.h b/hook.h index 11863fa734..ebe5dc450e 
>> 100644 --- a/hook.h +++ b/hook.h @@ -37,6 +38,28 @@ struct 
>> run_hooks_opt 
>>  	 * Path to file which should be piped to stdin for each 
>>  hook.  */ const char *path_to_stdin; 
>> + +	/** +	 * Callback to ask for more content to pipe to 
>> each hook stdin.  +	 * +	 * If a hook needs to consume 
>> large quantities of data (e.g. a +	 * list of all refs 
>> received in a client push), feeding data via +	 * 
>> in-memory strings or slurping to/from files via path_to_stdin + 
>> * is inefficient, so this callback allows for piecemeal writes. 
>> +	 * +	 * Add initalization context to 
>> hook.feed_pipe_ctx.  +	 * +	 * The caller owns 
>> hook.feed_pipe_ctx and has to release any +	 * resources after 
>> hooks finish execution.  +	 */ +	feed_pipe_fn feed_pipe; + 
>> void *feed_pipe_ctx; + +	/** +	 * Use this to keep 
>> internal state for your feed_pipe_fn callback.  +	 * Only 
>> useful when using run_hooks_opt.feed_pipe, otherwise ignore it. 
>> +	 */ +	void *feed_pipe_cb_data; 
> 
> Are these fields used as any of the callback arguments? If so, 
> let's document which of the fields they correspond to, as it's 
> otherwise hard to follow. 
 
I understand your point and will document this better.

They are not the callback args, they are passed as part of the 
struct run_hooks_opt accessible via the void *pp_cb arg like this: 

struct hook_cb_data *hook_cb = pp_cb;
struct receive_hook_feed_state *feed_state = 
hook_cb->options->feed_pipe_cb_data;

Re: [PATCH v2 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API

[Adrian Ratiu]

On Tue, 21 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Fri, Oct 17, 2025 at 05:15:37PM +0300, Adrian Ratiu wrote: 
>> diff --git a/sequencer.c b/sequencer.c index 
>> 5476d39ba9..71ed31c774 100644 --- a/sequencer.c +++ 
>> b/sequencer.c @@ -1292,32 +1292,40 @@ int 
>> update_head_with_reflog(const struct commit *old_head, 
>>  	return ret; }  
>> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, 
>> void *pp_task_cb UNUSED) +{ +	struct hook_cb_data 
>> *hook_cb = pp_cb; +	struct strbuf *to_pipe = 
>> hook_cb->options->feed_pipe_ctx; 
> 
> Not sure, but shouldn't it be possible to set 
> `opt.feed_pipe_cb_data` instead and then access the context via 
> `pp_task_cb` here instead of having to reach into the struct? 

No, because they are different structures. You can look inside the 
existing run_hooks_opt(), it does:

struct hook_cb_data cb_data = {
    ...
    .options = options,
};

They're different nested options structures. :)

Maybe we could simplify these, however refactoring this options 
API is outside the scope of this series, just like refactoring the 
1/0/-1 fd values used by run-command.c accross the entire source 
tree we discussed in the previous version (I tried, then I gave up 
because it's too big & risky for this series).

I'd like to fix all these APIs btw, however it's a big independent 
effort. Who knows, maybe some day in the future... :)

Re: [PATCH v2 04/10] transport: convert pre-push to hook API

[Adrian Ratiu]

On Tue, 21 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Fri, Oct 17, 2025 at 05:15:38PM +0300, Adrian Ratiu wrote: 
>> diff --git a/transport.c b/transport.c index 
>> c7f06a7382..67368754bf 100644 --- a/transport.c +++ 
>> b/transport.c @@ -1316,65 +1316,56 @@ static void 
>> die_with_unpushed_submodules(struct string_list *needs_pushing) 
>>  	die(_("Aborting.")); }  
>> -static int run_pre_push_hook(struct transport *transport, - 
>> struct ref *remote_refs) +static int 
>> pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void 
>> *pp_task_cb UNUSED) 
>>  { 
>> -	int ret = 0, x; -	struct ref *r; -	struct 
>> child_process proc = CHILD_PROCESS_INIT; -	struct strbuf buf; 
>> -	const char *hook_path = find_hook(the_repository, 
>> "pre-push"); +	struct hook_cb_data *hook_cb = pp_cb; + 
>> struct ref *r = hook_cb->options->feed_pipe_ctx; +	struct 
>> strbuf *buf = hook_cb->options->feed_pipe_cb_data; 
> 
> Same question here, isn't `feed_pipe_cb_data` accessible via 
> `pp_task_cb`? May very well be that I misunderstand the two 
> callback context and data, I found that part to be a bit hard to 
> follow. 

I hope I ansewered this in the other replies, so I won't repeat 
here. :)
 
>> +	int ret = 0; 
>>   
>> -	if (!hook_path) -		return 0; +	if (!r) + 
>> return 1; /* no more refs */ 
>>   
>> -	strvec_push(&proc.args, hook_path); - 
>> strvec_push(&proc.args, transport->remote->name); - 
>> strvec_push(&proc.args, transport->url); +	if (!buf) + 
>> BUG("pipe_task_cb must contain a valid strbuf"); 
>>   
>> -	proc.in = -1; -	proc.trace2_hook_name = "pre-push"; + 
>> hook_cb->options->feed_pipe_ctx = r->next; 
> 
> I think that the lines between the "task data" and "task 
> context" are being blurred here. I understood it so that the 
> task data is what is specific to the callback, and that data may 
> be changed to keep track of the state. Subsequent commits do it 
> that way, so shouldn't we also treat the context as immutable 
> here and instead handle iteration via the data? 

Excellent observation! I will do exactly that in v3. Thanks!
 
>> +	strbuf_reset(buf); 
> 
> Nit: it would make sense to move this reset down a bit close to 
> the first call that writes to it.

Yes, will do.
 
> 
>> -	if (start_command(&proc)) { - 
>> finish_command(&proc); -		return -1; -	} - - 
>> sigchain_push(SIGPIPE, SIG_IGN); +	if (!r->peer_ref) return 
>> 0; +	if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 
>> 0; +	if (r->status == REF_STATUS_REJECT_STALE) return 0; +	if 
>> (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0; +	if 
>> (r->status == REF_STATUS_UPTODATE) return 0; 
> 
> Nit, feel free to ignore: this might read a tiny bit nicer with 
> a switch statement. 

Ack, will fix in v3.

Re: [PATCH v2 07/10] run-command: allow capturing of collated output

[Adrian Ratiu]

On Tue, 21 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Fri, Oct 17, 2025 at 05:15:41PM +0300, Adrian Ratiu wrote: 
>> diff --git a/run-command.h b/run-command.h index 
>> e536ed7544..2c2484478b 100644 --- a/run-command.h +++ 
>> b/run-command.h @@ -436,6 +436,20 @@ typedef int 
>> (*feed_pipe_fn)(int child_in, 
>>  				void *pp_cb, void *pp_task_cb);  
>> +/** + * If this callback is provided, instead of collating 
>> process output to stderr, + * they will be collated into a new 
>> pipe. consume_sideband_fn will be called + * repeatedly. When 
>> output is available on that pipe, it will be contained in + * 
>> 'output'. But it will be called with an empty 'output' too, to 
>> allow for + * keepalives or similar operations if necessary.  + 
>> * + * pp_cb is the callback cookie as passed into 
>> run_processes_parallel.  + * + * Since this callback is 
>> provided with the collated output, no task cookie is + * 
>> provided.  + */ +typedef void (*consume_sideband_fn)(struct 
>> strbuf *output, void *pp_cb); 
> 
> I think the interface overall makes sense, but isn't the wording 
> we use here very specific for hooks? 
> 
> Taking a step back, what this thing seems to do is to take the 
> hook's output and customize how exactly we handle this. That 
> isn't really specific to any kind of "sideband", even though our 
> use case in the hook code does use this for sidebands. 
> 
> So maybe we should call this `consume_output_fn` and adapt the 
> rest of the code accordingly? Because that's what we ultimately 
> do here, IIUC. 

Yes, that is a good idea. run-command is supposed to be a lower 
level, more generic layer than the hooks, so it should not be 
concerned with concepts such as "sideband". 

Will do in v3. Thanks!

Re: [PATCH v2 00/10] Convert remaining hooks to hook.h

[Adrian Ratiu]

On Tue, 21 Oct 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Fri, Oct 17, 2025 at 05:15:34PM +0300, Adrian Ratiu wrote: 
>> Hello everyone,  This is v2 of the series which converts the 
>> remaining hooks to the new API.   I addressed all the feedback 
>> received in v1, with two small exceptions (the ones starting 
>> with "Opted not to" in the below Changes list).   I had a minor 
>> conflict with an upstream change [1] which was trivial to fix. 
>> I added 1 new commit and squashed together two commits (the 
>> simplified update hooks), so in total it's still 10 patches. 
>> The plan is to follow this up with another series which enables 
>> config-based hooks and parallel hook execution, where possible. 
>> As always this is based on the latest master branch, I've 
>> pushed it to GitHub and ran the CI pipeline [3]. The Win+Meson 
>> "missing libgitcore.a" and doc "invalid escape sequence" 
>> failures seem to be unrelated, since I get them without these 
>> patches.   1: 
>> https://github.com/git/git/commit/22e7bc801cd9c5e5b5c4489b631be28e506fec42 
>> 2: 
>> https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v2 
>> 3: https://github.com/10ne1/git/actions/runs/18593709082 
>> Changes between v1 -> v2: * Added a new commit with a mechanism 
>> to override ungroup options (Junio) * Addded a BUG if hook 
>> path_to_stdin and feed_pipe are both provided (Junio) * The 
>> feed_pipe cb can be set independently from path_to_stdin 
>> (Junio) * Simplified the post-rewrite callback (Patrick, 
>> Phillip and Junio) * Document that hook caller owns the 
>> feed_pipe_ctx (Junio) * Removed unnecessary "child" -> 
>> "notes_cp" renames (Phillip) * Reuse strbuf inside pre-push cb 
>> to avoid multiple alloc (Phillip) * Simplified pre-push hook cb 
>> logic (Phillip) * Rewrote reference-transaction cb logic to 
>> mirror pre-push (Patrick) * Simplified the update hook cb by 
>> removing the keepalive logic (Emily) * Squashed the simplified 
>> update and post-update conversions * Iterator types, if 
>> conditions and other small fixes (Patrick) * Fixed a conflict 
>> in refs.c with an upstream for loop sign compare check * Opted 
>> not to use -1 to signify no fd value instead of 0, because I'd 
>> have to 
>>   significantly rework the run-command.h .in/.out/.err API 
>>   (Patrick) 
>> * Opted not to move sigchain_push(SIGPIPE, SIG_IGN); into 
>> pp_buffer_stdin()) 
>>   because it will called too many times inside the process loop 
>>   (Patrick) 
>> * Added Helped-by: Emily Shaffer tag to the 
>> reference-transaction coversion * Comments, typos, stray lines, 
>> commit rewordings (Ben, Patrick, Emily, Junio) 
> 
> By the way, it would have helped to have a range-diff here so 
> that it's easy to see what exactly changed between the two 
> different versions.  Could you maybe include that in subsequent 
> iterations? 
 
Sure, I will include a range-diff going forward.

Thank you again for your very awesome feedback,
Adrian

Re: [PATCH v2 09/10] receive-pack: convert update hooks to new API

[Kristoffer Haugsbakk]

On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote:
> From: Emily Shaffer <emilyshaffer@google.com>
>
> Use the new hook sideband API introduced in the previous commit.
>
> The hook API avoids creating a custom struct child_process and other
> internal hook plumbing (e.g. calling find_hook()) and prepares for
> the specification of hooks via configs or running parallel hooks.
>
> Execution is still sequential through the current hook.[ch] via the
> run_proces_parallel_opts.processes=1 arg.

s/run_proces_parallel_opts/run_process_parallel_opts/

>
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
> ---

Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API

[Kristoffer Haugsbakk]

On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote:
> From: Emily Shaffer <emilyshaffer@google.com>
>
> This converts the last remaining hooks to the new hook API, for
> the same benefits as the previous conversions (no need to toggle
> signals, manage custom struct child_process, call find_hook(),
> prepares for specifyinig hooks via configs, etc.).
>
> I noticed a performance degradation when processing large amounts
> of hook input with just 1 line per callback, due to run-command's
> ppoll loop, therefore I batched 500 lines per callback, to ensure

I don’t see `ppoll` in `run-command.c`.

> similar pipe throughput as before and to avoid hook child waiting
> on stdin.
>
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
> ---

Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

On Tue, 28 Oct 2025, "Kristoffer Haugsbakk" 
<kristofferhaugsbakk@fastmail.com> wrote:
> On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote: 
>> From: Emily Shaffer <emilyshaffer@google.com> 
>> 
>> This converts the last remaining hooks to the new hook API, for 
>> the same benefits as the previous conversions (no need to 
>> toggle signals, manage custom struct child_process, call 
>> find_hook(), prepares for specifyinig hooks via configs, etc.). 
>> 
>> I noticed a performance degradation when processing large 
>> amounts of hook input with just 1 line per callback, due to 
>> run-command's ppoll loop, therefore I batched 500 lines per 
>> callback, to ensure 
> 
> I don’t see `ppoll` in `run-command.c`. 

Good point, it's poll not ppoll. :)

(The while(1) loop in run_processes_parallel() from run-command.c 
calls pp_buffer_stderr() then poll with an output_timeout of 100).

I will fix this together with the other typos you pointed out in 
v3.

Thank you,
Adrian

Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API

[Kristoffer Haugsbakk]

On Wed, Oct 29, 2025, at 14:46, Adrian Ratiu wrote:
> On Tue, 28 Oct 2025, "Kristoffer Haugsbakk"
> <kristofferhaugsbakk@fastmail.com> wrote:
>> On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote:
>>>[snip]
>> I don’t see `ppoll` in `run-command.c`.
>
> Good point, it's poll not ppoll. :)
>
> (The while(1) loop in run_processes_parallel() from run-command.c
> calls pp_buffer_stderr() then poll with an output_timeout of 100).
>
> I will fix this together with the other typos you pointed out in
> v3.

Thank you.  I considered marking them all as `nit` but this series has
spanned over such a long time that they seemed worth fixing.

Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> +static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)

Overly long line and cannot read.  Can you stick to 80-column lines?

In any case, the reason I am responding to this message is not about
coding styles, but it seems to be the one whose leak is holding the
CI job from passing at the tip of 'seen'.

>  {
> +	struct command *cmd = state->cmd;
>  
> +	strbuf_reset(&state->buf);
>  
> +	/* batch lines to avoid going through run-command's ppoll for each line */
> +	for (int i = 0; i < lines_batch_size; i++) {
> +		while (cmd &&
> +		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
> +			cmd = cmd->next;
>  
> +		if (!cmd)
> +			break;  /* no more commands left */
>  
> +		if (!state->report)
> +			state->report = cmd->report;
>  
> +		if (state->report) {
> +			struct object_id *old_oid;
> +			struct object_id *new_oid;
> +			const char *ref_name;
>  
> +			old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
> +			new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
> +			ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
>  
> +			strbuf_addf(&state->buf, "%s %s %s\n",
> +				    oid_to_hex(old_oid), oid_to_hex(new_oid),
> +				    ref_name);
>  
> +			state->report = state->report->next;
> +			if (!state->report)
> +				cmd = cmd->next;
> +		} else {
> +			strbuf_addf(&state->buf, "%s %s %s\n",
> +				    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
> +				    cmd->ref_name);
> +			cmd = cmd->next;
> +		}
>  	}
>  
> +	state->cmd = cmd;
>  
> +	if (state->buf.len > 0) {
> +		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
> +		if (ret < 0) {
> +			if (errno == EPIPE)
> +				return 1; /* child closed pipe */
> +			return ret;
> +		}
> +	}
> +
> +	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
>  }

> +static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
>  {
> -	struct receive_hook_feed_state *state = state_;
> -	struct command *cmd = state->cmd;
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;
> ... 
> +	/* first-time setup */
> +	if (!hook_cb->options->feed_pipe_cb_data) {
> +		struct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;
> +		if (!ctx)
> +			BUG("run_hooks_opt.feed_pipe_ctx required for receive hook");
> +
> +		hook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));

The allocation done here seems to be causing one (smaller) leak.

  https://github.com/git/git/actions/runs/19381820975/job/55461902226#step:10:3994

> +		feed_state = hook_cb->options->feed_pipe_cb_data;
> +		strbuf_init(&feed_state->buf, 0);
> +		feed_state->cmd = ctx->cmd;
> +		feed_state->skip_broken = ctx->skip_broken;
> +		feed_state->report = NULL;
>  	}
> +
> +	/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */
> +	if (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)
> +		return 0; /* still have more data to feed */

It appears to me that the larger leak the leak checker finds

  https://github.com/git/git/actions/runs/19381820975/job/55461902226#step:10:4017

is in find_receive_hook() called from here, where state->buf has
accumulates a lot.

builtin/receive-pack.c:847 its stacktrace #5 talks about is probably
pointing at the struf_addf() call there.  The particular test is
about deliberately causing EPIPE by the making other side refuse to
read, so some error handling on this side is missing the necessary
deallocation, perhaps?

> +	strbuf_release(&feed_state->buf);
> +
> +	if (hook_cb->options->feed_pipe_cb_data)
> +		FREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);
> +
> +	return 1; /* done feeding, run-command can close pipe */
> +}

Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

On Sat, 15 Nov 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Adrian Ratiu <adrian.ratiu@collabora.com> writes: 
> 
>> +static int feed_receive_hook(int hook_stdin_fd, struct 
>> receive_hook_feed_state *state, int lines_batch_size) 
> 
> Overly long line and cannot read.  Can you stick to 80-column 
> lines? 
> 
> In any case, the reason I am responding to this message is not 
> about coding styles, but it seems to be the one whose leak is 
> holding the CI job from passing at the tip of 'seen'. 
>

Hi Junio and thanks for these; will address and send a refresh 
with the rest of feedback in v3 very soon.

[PATCH v3 00/10] Convert remaining hooks to hook.h

[Adrian Ratiu]

Hello everyone,

This series finishes hook.[ch] conversion for remaining hooks in preparation for
adding config-based hooks and enabling hook parallelization where possible.

v3 addresses review feedback received in v2 plus some minor code simplifications
mentioned below. A v2->v3 range diff is also provided for reviewer convenience.

This is based on the latest master branch and I've checked for conflicts with
next and seen branches. The code is also available on GitHub [1] and a successful
CI run [2] is also available.

1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v3
2: https://github.com/10ne1/git/actions/runs/19639495095

Changes between v2 -> v3:
* Fixed receive hook memory leaks by moving allocations to callers who own the mem (Junio)
* Simplified receive hook stdin callback by removing the redundant context addition (Adrian)
* Moved ref pointer from callback context to cb_data in the pre-push hook (Patrick)
* Renamed the run-command output API to remove the hook-specific sideband wording (Patrick)
* Added all REF_STATUS_REJECT_* statuses to pre_push hook (Adrian)
* Improved documentation for hook.h and run-command.h (Patrick)
* Small code cleanups for data types, conditionals, BUGs, commit msgs (Patrick, Kristoffer, Junio)

Range-diff between v2 -> v3:
 1:  0c280cf782 !  1:  42aef8fea9 run-command: add stdin callback for parallelization
    @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,
     +			    const struct run_process_parallel_opts *opts)
     +{
     +	/* Buffer stdin for each pipe. */
    -+	for (ssize_t i = 0; i < opts->processes; i++) {
    ++	for (size_t i = 0; i < opts->processes; i++) {
     +		struct child_process *proc = &pp->children[i].process;
     +		int ret;
     +
    @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp UNUSED,
      		strbuf_addstr(err, "asking for a quick stop\n");
      	else
      		fprintf(stderr, "asking for a quick stop\n");
    -+	if (pp_task_cb)
    -+		FREE_AND_NULL(pp_task_cb);
    ++
    ++	FREE_AND_NULL(pp_task_cb);
    ++
      	return 1;
      }
      
    @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp UNUSED,
     +			       void *pp_cb UNUSED,
     +			       void *pp_task_cb)
     +{
    -+	if (pp_task_cb)
    -+		FREE_AND_NULL(pp_task_cb);
    ++	FREE_AND_NULL(pp_task_cb);
     +	return 0;
     +}
     +
 2:  94657d9279 !  2:  00c5b45987 hook: provide stdin via callback
    @@ hook.c: int run_hooks_opt(struct repository *r, const char *hook_name,
      		BUG("a struct run_hooks_opt must be provided to run_hooks");
      
     +	if (options->path_to_stdin && options->feed_pipe)
    -+		BUG("choose only one method to populate hook stdin");
    ++		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
     +
      	if (options->invoked_hook)
      		*options->invoked_hook = 0;
    @@ hook.h: struct run_hooks_opt
      	const char *path_to_stdin;
     +
     +	/**
    -+	 * Callback to ask for more content to pipe to each hook stdin.
    ++	 * Callback used to incrementally feed a child hook stdin pipe.
     +	 *
    -+	 * If a hook needs to consume large quantities of data (e.g. a
    -+	 * list of all refs received in a client push), feeding data via
    -+	 * in-memory strings or slurping to/from files via path_to_stdin
    -+	 * is inefficient, so this callback allows for piecemeal writes.
    ++	 * Useful especially if a hook consumes large quantities of data
    ++	 * (e.g. a list of all refs in a client push), so feeding it via
    ++	 * in-memory strings or slurping to/from files is inefficient.
    ++	 * While the callback allows piecemeal writing, it can also be
    ++	 * used for smaller inputs, where it gets called only once.
     +	 *
    -+	 * Add initalization context to hook.feed_pipe_ctx.
    ++	 * Add hook callback initalization context to `feed_pipe_ctx`.
    ++	 * Add Hook callback internal state to `feed_pipe_cb_data`.
     +	 *
    -+	 * The caller owns hook.feed_pipe_ctx and has to release any
    -+	 * resources after hooks finish execution.
     +	 */
     +	feed_pipe_fn feed_pipe;
    ++
    ++	/**
    ++	 * Opaque data pointer used to pass context to `feed_pipe_fn`.
    ++	 *
    ++	 * It can be accessed via the second callback arg:
    ++	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;
    ++	 *
    ++	 * The caller is responsible for managing the memory for this data.
    ++	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
    ++	 */
     +	void *feed_pipe_ctx;
     +
     +	/**
    -+	 * Use this to keep internal state for your feed_pipe_fn callback.
    -+	 * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.
    ++	 * Opaque data pointer used to keep internal state across callback calls.
    ++	 *
    ++	 * It can be accessed via the second callback arg:
    ++	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_cb_data;
    ++	 *
    ++	 * The caller is responsible for managing the memory for this data.
    ++	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
     +	 */
     +	void *feed_pipe_cb_data;
      };
 3:  5ddbf1d1d9 =  3:  08792569df hook: convert 'post-rewrite' hook in sequencer.c to hook API
 4:  948eb95587 !  4:  8a76eb11cf transport: convert pre-push to hook API
    @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_
      
     -static int run_pre_push_hook(struct transport *transport,
     -			     struct ref *remote_refs)
    -+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
    - {
    +-{
     -	int ret = 0, x;
     -	struct ref *r;
     -	struct child_process proc = CHILD_PROCESS_INIT;
    --	struct strbuf buf;
    ++struct feed_pre_push_hook_data {
    + 	struct strbuf buf;
     -	const char *hook_path = find_hook(the_repository, "pre-push");
    -+	struct hook_cb_data *hook_cb = pp_cb;
    -+	struct ref *r = hook_cb->options->feed_pipe_ctx;
    -+	struct strbuf *buf = hook_cb->options->feed_pipe_cb_data;
    -+	int ret = 0;
    ++	const struct ref *refs;
    ++};
      
     -	if (!hook_path)
     -		return 0;
    -+	if (!r)
    -+		return 1; /* no more refs */
    ++static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
    ++{
    ++	struct hook_cb_data *hook_cb = pp_cb;
    ++	struct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;
    ++	const struct ref *r = data->refs;
    ++	int ret = 0;
      
     -	strvec_push(&proc.args, hook_path);
     -	strvec_push(&proc.args, transport->remote->name);
     -	strvec_push(&proc.args, transport->url);
    -+	if (!buf)
    -+		BUG("pipe_task_cb must contain a valid strbuf");
    ++	if (!r)
    ++		return 1; /* no more refs */
      
     -	proc.in = -1;
     -	proc.trace2_hook_name = "pre-push";
    -+	hook_cb->options->feed_pipe_ctx = r->next;
    -+	strbuf_reset(buf);
    ++	data->refs = r->next;
      
     -	if (start_command(&proc)) {
     -		finish_command(&proc);
     -		return -1;
    --	}
    --
    ++	switch (r->status) {
    ++	case REF_STATUS_REJECT_ALREADY_EXISTS:
    ++	case REF_STATUS_REJECT_FETCH_FIRST:
    ++	case REF_STATUS_REJECT_NEEDS_FORCE:
    ++	case REF_STATUS_REJECT_NODELETE:
    ++	case REF_STATUS_REJECT_NONFASTFORWARD:
    ++	case REF_STATUS_REJECT_REMOTE_UPDATED:
    ++	case REF_STATUS_REJECT_SHALLOW:
    ++	case REF_STATUS_REJECT_STALE:
    ++	case REF_STATUS_UPTODATE:
    ++		return 0; /* skip refs which won't be pushed */
    ++	default:
    ++		break;
    + 	}
    + 
     -	sigchain_push(SIGPIPE, SIG_IGN);
    -+	if (!r->peer_ref) return 0;
    -+	if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;
    -+	if (r->status == REF_STATUS_REJECT_STALE) return 0;
    -+	if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;
    -+	if (r->status == REF_STATUS_UPTODATE) return 0;
    ++	if (!r->peer_ref)
    ++		return 0;
      
     -	strbuf_init(&buf, 256);
    -+	strbuf_addf(buf, "%s %s %s %s\n",
    ++	strbuf_reset(&data->buf);
    ++	strbuf_addf(&data->buf, "%s %s %s %s\n",
     +		    r->peer_ref->name, oid_to_hex(&r->new_oid),
     +		    r->name, oid_to_hex(&r->old_oid));
      
    @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_
     -		if (r->status == REF_STATUS_REJECT_STALE) continue;
     -		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
     -		if (r->status == REF_STATUS_UPTODATE) continue;
    -+	ret = write_in_full(hook_stdin_fd, buf->buf, buf->len);
    ++	ret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);
     +	if (ret < 0 && errno != EPIPE)
     +		return ret; /* We do not mind if a hook does not read all refs. */
      
    @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_
     +			     struct ref *remote_refs)
     +{
     +	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
    -+	struct strbuf buf = STRBUF_INIT;
    ++	struct feed_pre_push_hook_data data;
     +	int ret = 0;
    - 
    --	strbuf_release(&buf);
    ++
     +	strvec_push(&opt.args, transport->remote->name);
     +	strvec_push(&opt.args, transport->url);
      
    +-	strbuf_release(&buf);
    ++	strbuf_init(&data.buf, 0);
    ++	data.refs = remote_refs;
    + 
     -	x = close(proc.in);
     -	if (!ret)
     -		ret = x;
     +	opt.feed_pipe = pre_push_hook_feed_stdin;
    -+	opt.feed_pipe_ctx = remote_refs;
    -+	opt.feed_pipe_cb_data = &buf;
    ++	opt.feed_pipe_cb_data = &data;
      
     -	sigchain_pop(SIGPIPE);
     +	ret = run_hooks_opt(the_repository, "pre-push", &opt);
    @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_
     -	x = finish_command(&proc);
     -	if (!ret)
     -		ret = x;
    -+	strbuf_release(&buf);
    ++	strbuf_release(&data.buf);
      
      	return ret;
      }
 5:  64322e2659 =  5:  92452c6707 reference-transaction: use hook API instead of run-command
 6:  62306f1058 =  6:  6ec922ccef hook: allow overriding the ungroup option
 7:  eb284028a1 !  7:  3098e8f31a run-command: allow capturing of collated output
    @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp,
      	 * iteration, the buffered output is non empty.
      	 */
     -	strbuf_write(&pp->buffered_output, stderr);
    -+	if (opts->consume_sideband)
    -+		opts->consume_sideband(&pp->buffered_output, opts->data);
    ++	if (opts->consume_output)
    ++		opts->consume_output(&pp->buffered_output, opts->data);
     +	else
     +		strbuf_write(&pp->buffered_output, stderr);
      	strbuf_release(&pp->buffered_output);
    @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp,
      	if (pp->children[i].state == GIT_CP_WORKING &&
      	    pp->children[i].err.len) {
     -		strbuf_write(&pp->children[i].err, stderr);
    -+		if (opts->consume_sideband)
    -+			opts->consume_sideband(&pp->children[i].err, opts->data);
    ++		if (opts->consume_output)
    ++			opts->consume_output(&pp->children[i].err, opts->data);
     +		else
     +			strbuf_write(&pp->children[i].err, stderr);
      		strbuf_reset(&pp->children[i].err);
    @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,
      
     -			strbuf_write(&pp->children[i].err, stderr);
     +			/* Output errors, then all other finished child processes */
    -+			if (opts->consume_sideband) {
    -+				opts->consume_sideband(&pp->children[i].err, opts->data);
    -+				opts->consume_sideband(&pp->buffered_output, opts->data);
    ++			if (opts->consume_output) {
    ++				opts->consume_output(&pp->children[i].err, opts->data);
    ++				opts->consume_output(&pp->buffered_output, opts->data);
     +			} else {
     +				strbuf_write(&pp->children[i].err, stderr);
     +				strbuf_write(&pp->buffered_output, stderr);
    @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt
      					   "max:%"PRIuMAX,
      					   (uintmax_t)opts->processes);
      
    -+	if (opts->ungroup && opts->consume_sideband)
    -+		BUG("ungroup and reading sideband are mutualy exclusive");
    ++	if (opts->ungroup && opts->consume_output)
    ++		BUG("ungroup and reading output are mutualy exclusive");
     +
      	/*
      	 * Child tasks might receive input via stdin, terminating early (or not), so
    @@ run-command.h: typedef int (*feed_pipe_fn)(int child_in,
      				void *pp_task_cb);
      
     +/**
    -+ * If this callback is provided, instead of collating process output to stderr,
    -+ * they will be collated into a new pipe. consume_sideband_fn will be called
    -+ * repeatedly. When output is available on that pipe, it will be contained in
    -+ * 'output'. But it will be called with an empty 'output' too, to allow for
    -+ * keepalives or similar operations if necessary.
    ++ * If this callback is provided, output is collated into a new pipe instead
    ++ * of the process stderr. Then `consume_output_fn` will be called repeatedly
    ++ * with output contained in the `output` arg. It will also be called with an
    ++ * empty `output` to allow for keepalives or similar operations if necessary.
     + *
     + * pp_cb is the callback cookie as passed into run_processes_parallel.
    -+ *
    -+ * Since this callback is provided with the collated output, no task cookie is
    -+ * provided.
    ++ * No task cookie is provided because the callback receives collated output.
     + */
    -+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);
    ++typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);
     +
      /**
       * This callback is called on every child process that finished processing.
    @@ run-command.h: struct run_process_parallel_opts
      	feed_pipe_fn feed_pipe;
      
     +	/*
    -+	 * consume_sideband: see consume_sideband_fn() above. This can be NULL
    ++	 * consume_output: see consume_output_fn() above. This can be NULL
     +	 * to omit any special handling.
     +	 */
    -+	consume_sideband_fn consume_sideband;
    ++	consume_output_fn consume_output;
     +
      	/**
      	 * task_finished: See task_finished_fn() above. This can be
    @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp UNUSED,
      	return 0;
      }
      
    -+static void test_consume_sideband(struct strbuf *output, void *cb UNUSED)
    ++static void test_divert_output(struct strbuf *output, void *cb UNUSED)
     +{
    -+	FILE *sideband;
    ++	FILE *output_file;
     +
    -+	sideband = fopen("./sideband", "a");
    ++	output_file = fopen("./output_file", "a");
     +
    -+	strbuf_write(output, sideband);
    -+	fclose(sideband);
    ++	strbuf_write(output, output_file);
    ++	fclose(output_file);
     +}
     +
      static int task_finished(int result UNUSED,
    @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
      		.get_next_task = next_test,
      		.start_failure = test_failed,
      		.feed_pipe = test_stdin_pipe_feed,
    -+		.consume_sideband = test_consume_sideband,
    ++		.consume_output = test_divert_output,
      		.task_finished = test_finished,
      		.data = &suite,
      	};
    @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)
      		opts.get_next_task = parallel_next;
      		opts.task_finished = task_finished_quiet;
      		opts.feed_pipe = test_stdin_pipe_feed;
    -+	} else if (!strcmp(argv[1], "run-command-sideband")) {
    ++	} else if (!strcmp(argv[1], "run-command-divert-output")) {
     +		opts.get_next_task = parallel_next;
    -+		opts.consume_sideband = test_consume_sideband;
    ++		opts.consume_output = test_divert_output;
     +		opts.task_finished = task_finished_quiet;
      	} else {
      		ret = 1;
    @@ t/t0061-run-command.sh: test_expect_success 'run_command runs ungrouped in paral
      '
      
     +test_expect_success 'run_command can divert output' '
    -+	test_when_finished rm sideband &&
    -+	test-tool run-command run-command-sideband 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
    ++	test_when_finished rm output_file &&
    ++	test-tool run-command run-command-divert-output 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
     +	test_must_be_empty actual &&
    -+	test_cmp expect sideband
    ++	test_cmp expect output_file
     +'
     +
      test_expect_success 'run_command listens to stdin' '
 8:  dc2de535a5 !  8:  1a21310174 hooks: allow callers to capture output
    @@ hook.c: int run_hooks_opt(struct repository *r, const char *hook_name,
      		.get_next_task = pick_next_hook,
      		.start_failure = notify_start_failure,
      		.feed_pipe = options->feed_pipe,
    -+		.consume_sideband = options->consume_sideband,
    ++		.consume_output = options->consume_output,
      		.task_finished = notify_hook_finished,
      
      		.data = &cb_data,
     
      ## hook.h ##
     @@ hook.h: struct run_hooks_opt
    - 	 * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.
    + 	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
      	 */
      	void *feed_pipe_cb_data;
     +
    @@ hook.h: struct run_hooks_opt
     +	 * run_command:run_parallel_processes(). See t/helper/test-run-command.c
     +	 * for an example.
     +	 */
    -+	consume_sideband_fn consume_sideband;
    ++	consume_output_fn consume_output;
      };
      
      #define RUN_HOOKS_OPT_INIT { \
 9:  2f85907a2a !  9:  6a2895750a receive-pack: convert update hooks to new API
    @@ Commit message
         the specification of hooks via configs or running parallel hooks.
     
         Execution is still sequential through the current hook.[ch] via the
    -    run_proces_parallel_opts.processes=1 arg.
    +    run_process_parallel_opts.processes=1 arg.
     
         Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
      	if (use_sideband)
     -		copy_to_sideband(proc.err, -1, NULL);
     -	return finish_command(&proc);
    -+		opt.consume_sideband = hook_output_to_sideband;
    ++		opt.consume_output = hook_output_to_sideband;
     +
     +	return run_hooks_opt(the_repository, "update", &opt);
      }
    @@ builtin/receive-pack.c: static const char *update(struct command *cmd, struct sh
     -	proc.err = use_sideband ? -1 : 0;
     -	proc.trace2_hook_name = "post-update";
     +	if (use_sideband)
    -+		opt.consume_sideband = hook_output_to_sideband;
    ++		opt.consume_output = hook_output_to_sideband;
      
     -	if (!start_command(&proc)) {
     -		if (use_sideband)
10:  5dff0f4980 ! 10:  254e7dd351 receive-pack: convert receive hooks to hook API
    @@ Commit message
     
         I noticed a performance degradation when processing large amounts
         of hook input with just 1 line per callback, due to run-command's
    -    ppoll loop, therefore I batched 500 lines per callback, to ensure
    +    poll loop, therefore I batched 500 lines per callback, to ensure
         similar pipe throughput as before and to avoid hook child waiting
         on stdin.
     
    @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process
      					     "GIT_PUSH_CERT_NONCE_SLOP=%ld",
      					     nonce_stamp_slop);
      		}
    - 	}
    - }
    - 
    -+struct receive_hook_feed_context {
    -+	struct command *cmd;
    -+	int skip_broken;
    -+};
    -+
    - struct receive_hook_feed_state {
    - 	struct command *cmd;
    +@@ builtin/receive-pack.c: struct receive_hook_feed_state {
      	struct ref_push_report *report;
      	int skip_broken;
      	struct strbuf buf;
    @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process
     -typedef int (*feed_fn)(void *, const char **, size_t *);
     -static int run_and_feed_hook(const char *hook_name, feed_fn feed,
     -			     struct receive_hook_feed_state *feed_state)
    -+static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)
    ++static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
      {
     -	struct child_process proc = CHILD_PROCESS_INIT;
     -	struct async muxer;
     -	int code;
     -	const char *hook_path = find_hook(the_repository, hook_name);
    ++	struct hook_cb_data *hook_cb = pp_cb;
    ++	struct receive_hook_feed_state *state = hook_cb->options->feed_pipe_cb_data;
     +	struct command *cmd = state->cmd;
    ++	unsigned int lines_batch_size = 500;
      
     -	if (!hook_path)
     -		return 0;
    @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process
     -			     (uintmax_t)feed_state->push_options->nr);
     -	} else
     -		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
    -+	/* batch lines to avoid going through run-command's ppoll for each line */
    -+	for (int i = 0; i < lines_batch_size; i++) {
    ++	/* batch lines to avoid going through run-command's poll loop for each line */
    ++	for (unsigned int i = 0; i < lines_batch_size; i++) {
     +		while (cmd &&
     +		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
     +			cmd = cmd->next;
    @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process
      }
      
     -static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
    -+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
    ++static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
      {
     -	struct receive_hook_feed_state *state = state_;
     -	struct command *cmd = state->cmd;
    -+	struct hook_cb_data *hook_cb = pp_cb;
    -+	struct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;
    - 
    +-
     -	while (cmd &&
     -	       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
     -		cmd = cmd->next;
    @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process
     -	if (bufp) {
     -		*bufp = state->buf.buf;
     -		*sizep = state->buf.len;
    -+	/* first-time setup */
    -+	if (!hook_cb->options->feed_pipe_cb_data) {
    -+		struct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;
    -+		if (!ctx)
    -+			BUG("run_hooks_opt.feed_pipe_ctx required for receive hook");
    -+
    -+		hook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));
    -+		feed_state = hook_cb->options->feed_pipe_cb_data;
    -+		strbuf_init(&feed_state->buf, 0);
    -+		feed_state->cmd = ctx->cmd;
    -+		feed_state->skip_broken = ctx->skip_broken;
    -+		feed_state->report = NULL;
    - 	}
    +-	}
     -	return 0;
    -+
    -+	/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */
    -+	if (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)
    -+		return 0; /* still have more data to feed */
    -+
    -+	strbuf_release(&feed_state->buf);
    -+
    -+	if (hook_cb->options->feed_pipe_cb_data)
    -+		FREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);
    -+
    -+	return 1; /* done feeding, run-command can close pipe */
    -+}
    -+
    -+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
    -+{
     +	if (output && output->len)
     +		send_sideband(1, 2, output->buf, output->len, use_sideband);
      }
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
      {
     -	struct receive_hook_feed_state state;
     -	int status;
    -+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
    -+	struct receive_hook_feed_context ctx;
    -+	struct command *iter = commands;
    - 
    +-
     -	strbuf_init(&state.buf, 0);
     -	state.cmd = commands;
     -	state.skip_broken = skip_broken;
     -	state.report = NULL;
     -	if (feed_receive_hook(&state, NULL, NULL))
    ++	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
    ++	struct command *iter = commands;
    ++	struct receive_hook_feed_state *feed_state;
    ++	int ret;
    ++
     +	/* if there are no valid commands, don't invoke the hook at all. */
     +	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
     +		iter = iter->next;
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
     +
     +	/* set up sideband printer */
     +	if (use_sideband)
    -+		opt.consume_sideband = hook_output_to_sideband;
    ++		opt.consume_output = hook_output_to_sideband;
     +
     +	/* set up stdin callback */
    -+	ctx.cmd = commands;
    -+	ctx.skip_broken = skip_broken;
    ++	feed_state = xmalloc(sizeof(struct receive_hook_feed_state));
    ++	feed_state->cmd = commands;
    ++	feed_state->skip_broken = skip_broken;
    ++	feed_state->report = NULL;
    ++	strbuf_init(&feed_state->buf, 0);
    ++	opt.feed_pipe_cb_data = feed_state;
     +	opt.feed_pipe = feed_receive_hook_cb;
    -+	opt.feed_pipe_ctx = &ctx;
     +
    -+	return run_hooks_opt(the_repository, hook_name, &opt);
    ++	ret = run_hooks_opt(the_repository, hook_name, &opt);
    ++
    ++	strbuf_release(&feed_state->buf);
    ++	FREE_AND_NULL(opt.feed_pipe_cb_data);
    ++
    ++	return ret;
      }
      
      static int run_update_hook(struct command *cmd)


Adrian Ratiu (2):
  reference-transaction: use hook API instead of run-command
  hook: allow overriding the ungroup option

Emily Shaffer (8):
  run-command: add stdin callback for parallelization
  hook: provide stdin via callback
  hook: convert 'post-rewrite' hook in sequencer.c to hook API
  transport: convert pre-push to hook API
  run-command: allow capturing of collated output
  hooks: allow callers to capture output
  receive-pack: convert update hooks to new API
  receive-pack: convert receive hooks to hook API

 builtin/hook.c              |   6 +
 builtin/receive-pack.c      | 271 +++++++++++++++---------------------
 commit.c                    |   3 +
 hook.c                      |  21 ++-
 hook.h                      |  51 +++++++
 refs.c                      | 101 +++++++-------
 run-command.c               | 110 +++++++++++++--
 run-command.h               |  39 ++++++
 sequencer.c                 |  42 +++---
 t/helper/test-run-command.c |  67 ++++++++-
 t/t0061-run-command.sh      |  38 +++++
 transport.c                 |  95 +++++++------
 12 files changed, 562 insertions(+), 282 deletions(-)

-- 
2.51.0

[PATCH v3 01/10] run-command: add stdin callback for parallelization

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

If a user of the run_processes_parallel() API wants to pipe a large
amount of information to the stdin of each parallel command, that
data could exceed the pipe buffer of the process's stdin and can be
too big to store in-memory via strbuf & friends or to slurp to a file.

Generally this is solved by repeatedly writing to child_process.in
between calls to start_command() and finish_command(). For a specific
pre-existing example of this, see transport.c:run_pre_push_hook().

This adds a generic callback API to run_processes_parallel() to do
exactly that in a unified manner, similar to the existing callback APIs,
which can then be used by hooks.h to convert the remaining hooks to the
new, simpler parallel interface.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 82 +++++++++++++++++++++++++++++++++----
 run-command.h               | 22 ++++++++++
 t/helper/test-run-command.c | 52 ++++++++++++++++++++++-
 t/t0061-run-command.sh      | 31 ++++++++++++++
 4 files changed, 178 insertions(+), 9 deletions(-)

diff --git a/run-command.c b/run-command.c
index ed9575bd6a..9bf5911727 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
+static void pp_buffer_stdin(struct parallel_processes *pp,
+			    const struct run_process_parallel_opts *opts)
+{
+	/* Buffer stdin for each pipe. */
+	for (size_t i = 0; i < opts->processes; i++) {
+		struct child_process *proc = &pp->children[i].process;
+		int ret;
+
+		if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)
+			continue;
+
+		/*
+		 * child input is provided via path_to_stdin when the feed_pipe cb is
+		 * missing, so we just signal an EOF.
+		 */
+		if (!opts->feed_pipe) {
+			close(proc->in);
+			proc->in = 0;
+			continue;
+		}
+
+		/**
+		 * Feed the pipe:
+		 *   ret < 0 means error
+		 *   ret == 0 means there is more data to be fed
+		 *   ret > 0 means feeding finished
+		 */
+		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
+		if (ret < 0)
+			die_errno("feed_pipe");
+
+		if (ret) {
+			close(proc->in);
+			proc->in = 0;
+		}
+	}
+}
+
 static void pp_buffer_stderr(struct parallel_processes *pp,
 			     const struct run_process_parallel_opts *opts,
 			     int output_timeout)
@@ -1722,6 +1760,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		pp->children[i].state = GIT_CP_FREE;
 		if (pp->pfd)
 			pp->pfd[i].fd = -1;
+		pp->children[i].process.in = 0;
 		child_process_init(&pp->children[i].process);
 
 		if (opts->ungroup) {
@@ -1756,6 +1795,32 @@ static int pp_collect_finished(struct parallel_processes *pp,
 	return result;
 }
 
+static void pp_handle_child_IO(struct parallel_processes *pp,
+				const struct run_process_parallel_opts *opts,
+				int output_timeout)
+{
+	/*
+	 * First push input, if any (it might no-op), to child tasks to avoid them blocking
+	 * after input. This also prevents deadlocks when ungrouping below, if a child blocks
+	 * while the parent also waits for them to finish.
+	 */
+	pp_buffer_stdin(pp, opts);
+
+	if (opts->ungroup) {
+		for (size_t i = 0; i < opts->processes; i++) {
+			int child_ready_for_cleanup =
+				pp->children[i].state == GIT_CP_WORKING &&
+				pp->children[i].process.in == 0;
+
+			if (child_ready_for_cleanup)
+				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
+		}
+	} else {
+		pp_buffer_stderr(pp, opts, output_timeout);
+		pp_output(pp);
+	}
+}
+
 void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
@@ -1775,6 +1840,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	/*
+	 * Child tasks might receive input via stdin, terminating early (or not), so
+	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
+	 * actually writes the data to children stdin fds.
+	 */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
 	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (i = 0;
@@ -1792,13 +1864,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts->ungroup) {
-			for (size_t i = 0; i < opts->processes; i++)
-				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
-		} else {
-			pp_buffer_stderr(&pp, opts, output_timeout);
-			pp_output(&pp);
-		}
+		pp_handle_child_IO(&pp, opts, output_timeout);
 		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
@@ -1809,6 +1875,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 
 	pp_cleanup(&pp, opts);
 
+	sigchain_pop(SIGPIPE);
+
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
diff --git a/run-command.h b/run-command.h
index 0df25e445f..e536ed7544 100644
--- a/run-command.h
+++ b/run-command.h
@@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * This callback is repeatedly called on every child process who requests
+ * start_command() to create a pipe by setting child_process.in < 0.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel, and
+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.
+ * The contents of 'send' will be read into the pipe and passed to the pipe.
+ *
+ * Returns < 0 for error
+ * Returns == 0 when there is more data to be fed (will be called again)
+ * Returns > 0 when finished (child closed fd or no more data to be fed)
+ */
+typedef int (*feed_pipe_fn)(int child_in,
+				void *pp_cb,
+				void *pp_task_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -473,6 +489,12 @@ struct run_process_parallel_opts
 	 */
 	start_failure_fn start_failure;
 
+	/*
+	 * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any
+	 * special handling.
+	 */
+	feed_pipe_fn feed_pipe;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 3719f23cc2..4a56456894 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -23,19 +23,26 @@ static int number_callbacks;
 static int parallel_next(struct child_process *cp,
 			 struct strbuf *err,
 			 void *cb,
-			 void **task_cb UNUSED)
+			 void **task_cb)
 {
 	struct child_process *d = cb;
 	if (number_callbacks >= 4)
 		return 0;
 
 	strvec_pushv(&cp->args, d->args.v);
+	cp->in = d->in;
+	cp->no_stdin = d->no_stdin;
 	if (err)
 		strbuf_addstr(err, "preloaded output of a child\n");
 	else
 		fprintf(stderr, "preloaded output of a child\n");
 
 	number_callbacks++;
+
+	/* test_stdin callback will use this to count remaining lines */
+	*task_cb = xmalloc(sizeof(int));
+	*(int*)(*task_cb) = 2;
+
 	return 1;
 }
 
@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
-			 void *pp_task_cb UNUSED)
+			 void *pp_task_cb)
 {
 	if (err)
 		strbuf_addstr(err, "asking for a quick stop\n");
 	else
 		fprintf(stderr, "asking for a quick stop\n");
+
+	FREE_AND_NULL(pp_task_cb);
+
 	return 1;
 }
 
+static int task_finished_quiet(int result UNUSED,
+			       struct strbuf *err UNUSED,
+			       void *pp_cb UNUSED,
+			       void *pp_task_cb)
+{
+	FREE_AND_NULL(pp_task_cb);
+	return 0;
+}
+
+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)
+{
+	int *lines_remaining = task_cb;
+
+	if (*lines_remaining) {
+		struct strbuf buf = STRBUF_INIT;
+		strbuf_addf(&buf, "sample stdin %d\n", --(*lines_remaining));
+		if (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {
+			if (errno == EPIPE) {
+				/* child closed stdin, nothing more to do */
+				strbuf_release(&buf);
+				return 1;
+			}
+			die_errno("write");
+		}
+		strbuf_release(&buf);
+	}
+
+	return !(*lines_remaining);
+}
+
 struct testsuite {
 	struct string_list tests, failed;
 	int next;
@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)
 	struct run_process_parallel_opts opts = {
 		.get_next_task = next_test,
 		.start_failure = test_failed,
+		.feed_pipe = test_stdin_pipe_feed,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
 		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		opts.get_next_task = no_job;
 		opts.task_finished = task_finished;
+	} else if (!strcmp(argv[1], "run-command-stdin")) {
+		proc.in = -1;
+		proc.no_stdin = 0;
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
+		opts.feed_pipe = test_stdin_pipe_feed;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 76d4936a87..2f77fde0d9 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command listens to stdin' '
+	cat >expect <<-\EOF &&
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	EOF
+
+	write_script stdin-script <<-\EOF &&
+	echo "listening for stdin:"
+	while read line
+	do
+		echo "$line"
+	done
+	EOF
+	test-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&
+	test_cmp expect actual
+'
+
 cat >expect <<-EOF
 preloaded output of a child
 asking for a quick stop
-- 
2.51.0

[PATCH v3 02/10] hook: provide stdin via callback

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This adds a callback mechanism for feeding stdin to hooks alongside
the existing path_to_stdin (which slurps a file's content to stdin).

The advantage of this new callback is that it can feed stdin without
going through the FS layer. This helps when feeding large amount of
data and uses the run-command parallel stdin callback introduced in
the preceding commit.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 hook.c | 15 +++++++++++++++
 hook.h | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)

diff --git a/hook.c b/hook.c
index b3de1048bf..cd2bb7418a 100644
--- a/hook.c
+++ b/hook.c
@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,
 
 	cp->no_stdin = 1;
 	strvec_pushv(&cp->env, hook_cb->options->env.v);
+
+	if (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	/* reopen the file for stdin; run_command closes it. */
 	if (hook_cb->options->path_to_stdin) {
 		cp->no_stdin = 0;
 		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
 	}
+
+	if (hook_cb->options->feed_pipe) {
+		cp->no_stdin = 0;
+		/* start_command() will allocate a pipe / stdin fd for us */
+		cp->in = -1;
+	}
+
 	cp->stdout_to_stderr = 1;
 	cp->trace2_hook_name = hook_cb->hook_name;
 	cp->dir = hook_cb->options->dir;
@@ -140,6 +151,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
+		.feed_pipe = options->feed_pipe,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
@@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
 
+	if (options->path_to_stdin && options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	if (options->invoked_hook)
 		*options->invoked_hook = 0;
 
diff --git a/hook.h b/hook.h
index 11863fa734..dd87326a5a 100644
--- a/hook.h
+++ b/hook.h
@@ -1,6 +1,7 @@
 #ifndef HOOK_H
 #define HOOK_H
 #include "strvec.h"
+#include "run-command.h"
 
 struct repository;
 
@@ -37,6 +38,43 @@ struct run_hooks_opt
 	 * Path to file which should be piped to stdin for each hook.
 	 */
 	const char *path_to_stdin;
+
+	/**
+	 * Callback used to incrementally feed a child hook stdin pipe.
+	 *
+	 * Useful especially if a hook consumes large quantities of data
+	 * (e.g. a list of all refs in a client push), so feeding it via
+	 * in-memory strings or slurping to/from files is inefficient.
+	 * While the callback allows piecemeal writing, it can also be
+	 * used for smaller inputs, where it gets called only once.
+	 *
+	 * Add hook callback initalization context to `feed_pipe_ctx`.
+	 * Add Hook callback internal state to `feed_pipe_cb_data`.
+	 *
+	 */
+	feed_pipe_fn feed_pipe;
+
+	/**
+	 * Opaque data pointer used to pass context to `feed_pipe_fn`.
+	 *
+	 * It can be accessed via the second callback arg:
+	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_ctx;
+
+	/**
+	 * Opaque data pointer used to keep internal state across callback calls.
+	 *
+	 * It can be accessed via the second callback arg:
+	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_cb_data;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_cb_data;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.0

[PATCH v3 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Replace the custom run-command calls used by post-rewrite with
the newer and simpler hook_run_opt(), which does not need to
create a custom 'struct child_process' or call find_hook().

Another benefit of using the hook API is that hook_run_opt()
handles the SIGPIPE toggle logic.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 sequencer.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 5476d39ba9..71ed31c774 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,
 	return ret;
 }
 
+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
+{
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
+	int ret;
+
+	if (!to_pipe)
+		BUG("pipe_from_strbuf called without feed_pipe_ctx");
+
+	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
+
+	return 1; /* done writing */
+}
+
 static int run_rewrite_hook(const struct object_id *oldoid,
 			    const struct object_id *newoid)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 	int code;
 	struct strbuf sb = STRBUF_INIT;
-	const char *hook_path = find_hook(the_repository, "post-rewrite");
 
-	if (!hook_path)
-		return 0;
+	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
 
-	strvec_pushl(&proc.args, hook_path, "amend", NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "post-rewrite";
+	opt.feed_pipe_ctx = &sb;
+	opt.feed_pipe = pipe_from_strbuf;
+
+	strvec_push(&opt.args, "amend");
+
+	code = run_hooks_opt(the_repository, "post-rewrite", &opt);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
-	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
-	sigchain_push(SIGPIPE, SIG_IGN);
-	write_in_full(proc.in, sb.buf, sb.len);
-	close(proc.in);
 	strbuf_release(&sb);
-	sigchain_pop(SIGPIPE);
-	return finish_command(&proc);
+	return code;
 }
 
 void commit_post_rewrite(struct repository *r,
-- 
2.51.0

[PATCH v3 04/10] transport: convert pre-push to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Move the pre-push hook from custom run-command invocations to
the new hook API which doesn't require a custom child_process
structure and signal toggling.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 transport.c | 95 ++++++++++++++++++++++++++++-------------------------
 1 file changed, 51 insertions(+), 44 deletions(-)

diff --git a/transport.c b/transport.c
index c7f06a7382..b7a6d21d22 100644
--- a/transport.c
+++ b/transport.c
@@ -1316,65 +1316,72 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
 	die(_("Aborting."));
 }
 
-static int run_pre_push_hook(struct transport *transport,
-			     struct ref *remote_refs)
-{
-	int ret = 0, x;
-	struct ref *r;
-	struct child_process proc = CHILD_PROCESS_INIT;
+struct feed_pre_push_hook_data {
 	struct strbuf buf;
-	const char *hook_path = find_hook(the_repository, "pre-push");
+	const struct ref *refs;
+};
 
-	if (!hook_path)
-		return 0;
+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
+{
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;
+	const struct ref *r = data->refs;
+	int ret = 0;
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, transport->remote->name);
-	strvec_push(&proc.args, transport->url);
+	if (!r)
+		return 1; /* no more refs */
 
-	proc.in = -1;
-	proc.trace2_hook_name = "pre-push";
+	data->refs = r->next;
 
-	if (start_command(&proc)) {
-		finish_command(&proc);
-		return -1;
+	switch (r->status) {
+	case REF_STATUS_REJECT_ALREADY_EXISTS:
+	case REF_STATUS_REJECT_FETCH_FIRST:
+	case REF_STATUS_REJECT_NEEDS_FORCE:
+	case REF_STATUS_REJECT_NODELETE:
+	case REF_STATUS_REJECT_NONFASTFORWARD:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+	case REF_STATUS_REJECT_SHALLOW:
+	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_UPTODATE:
+		return 0; /* skip refs which won't be pushed */
+	default:
+		break;
 	}
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	if (!r->peer_ref)
+		return 0;
 
-	strbuf_init(&buf, 256);
+	strbuf_reset(&data->buf);
+	strbuf_addf(&data->buf, "%s %s %s %s\n",
+		    r->peer_ref->name, oid_to_hex(&r->new_oid),
+		    r->name, oid_to_hex(&r->old_oid));
 
-	for (r = remote_refs; r; r = r->next) {
-		if (!r->peer_ref) continue;
-		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
-		if (r->status == REF_STATUS_REJECT_STALE) continue;
-		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
-		if (r->status == REF_STATUS_UPTODATE) continue;
+	ret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);
+	if (ret < 0 && errno != EPIPE)
+		return ret; /* We do not mind if a hook does not read all refs. */
 
-		strbuf_reset(&buf);
-		strbuf_addf( &buf, "%s %s %s %s\n",
-			 r->peer_ref->name, oid_to_hex(&r->new_oid),
-			 r->name, oid_to_hex(&r->old_oid));
+	return 0;
+}
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			/* We do not mind if a hook does not read all refs. */
-			if (errno != EPIPE)
-				ret = -1;
-			break;
-		}
-	}
+static int run_pre_push_hook(struct transport *transport,
+			     struct ref *remote_refs)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct feed_pre_push_hook_data data;
+	int ret = 0;
+
+	strvec_push(&opt.args, transport->remote->name);
+	strvec_push(&opt.args, transport->url);
 
-	strbuf_release(&buf);
+	strbuf_init(&data.buf, 0);
+	data.refs = remote_refs;
 
-	x = close(proc.in);
-	if (!ret)
-		ret = x;
+	opt.feed_pipe = pre_push_hook_feed_stdin;
+	opt.feed_pipe_cb_data = &data;
 
-	sigchain_pop(SIGPIPE);
+	ret = run_hooks_opt(the_repository, "pre-push", &opt);
 
-	x = finish_command(&proc);
-	if (!ret)
-		ret = x;
+	strbuf_release(&data.buf);
 
 	return ret;
 }
-- 
2.51.0

[PATCH v3 05/10] reference-transaction: use hook API instead of run-command

[Adrian Ratiu]

Convert the reference-transaction hook to the new hook API,
so it doesn't need to set up a struct child_process, call
find_hook or toggle the pipe signals.

The stdin feed callback is processing one ref update per
call. I haven't noticed any performance degradation due
to this, however we can batch as many we want in each call,
to ensure a good pipe throughtput (i.e. the child does not
wait after stdin).

Helped-by: Emily Shaffer <nasamuffin@google.com>
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 refs.c | 101 ++++++++++++++++++++++++++++++---------------------------
 1 file changed, 53 insertions(+), 48 deletions(-)

diff --git a/refs.c b/refs.c
index 5583f6e09d..2b699cb44e 100644
--- a/refs.c
+++ b/refs.c
@@ -2422,68 +2422,73 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-static int run_transaction_hook(struct ref_transaction *transaction,
-				const char *state)
+struct transaction_feed_cb_data {
+	size_t index;
+	struct strbuf buf;
+};
+
+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct strbuf buf = STRBUF_INIT;
-	const char *hook;
-	int ret = 0;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct run_hooks_opt *opt = hook_cb->options;
+	struct ref_transaction *transaction = opt->feed_pipe_ctx;
+	struct transaction_feed_cb_data *feed_cb_data = opt->feed_pipe_cb_data;
+	struct strbuf *buf = &feed_cb_data->buf;
+	struct ref_update *update;
+	size_t i = feed_cb_data->index++;
+	int ret;
 
-	hook = find_hook(transaction->ref_store->repo, "reference-transaction");
-	if (!hook)
-		return ret;
+	if (i >= transaction->nr)
+		return 1; /* No more refs to process */
 
-	strvec_pushl(&proc.args, hook, state, NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "reference-transaction";
+	update = transaction->updates[i];
 
-	ret = start_command(&proc);
-	if (ret)
-		return ret;
+	if (update->flags & REF_LOG_ONLY)
+		return 0;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	strbuf_reset(buf);
 
-	for (size_t i = 0; i < transaction->nr; i++) {
-		struct ref_update *update = transaction->updates[i];
+	if (!(update->flags & REF_HAVE_OLD))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->old_target)
+		strbuf_addf(buf, "ref:%s ", update->old_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->old_oid));
 
-		if (update->flags & REF_LOG_ONLY)
-			continue;
+	if (!(update->flags & REF_HAVE_NEW))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->new_target)
+		strbuf_addf(buf, "ref:%s ", update->new_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->new_oid));
 
-		strbuf_reset(&buf);
+	strbuf_addf(buf, "%s\n", update->refname);
 
-		if (!(update->flags & REF_HAVE_OLD))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->old_target)
-			strbuf_addf(&buf, "ref:%s ", update->old_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->old_oid));
+	ret = write_in_full(hook_stdin_fd, buf->buf, buf->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
 
-		if (!(update->flags & REF_HAVE_NEW))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->new_target)
-			strbuf_addf(&buf, "ref:%s ", update->new_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->new_oid));
+	return 0; /* no more input to feed */
+}
+
+static int run_transaction_hook(struct ref_transaction *transaction,
+				const char *state)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct transaction_feed_cb_data feed_ctx = { 0 };
+	int ret = 0;
 
-		strbuf_addf(&buf, "%s\n", update->refname);
+	strvec_push(&opt.args, state);
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			if (errno != EPIPE) {
-				/* Don't leak errno outside this API */
-				errno = 0;
-				ret = -1;
-			}
-			break;
-		}
-	}
+	opt.feed_pipe = transaction_hook_feed_stdin;
+	opt.feed_pipe_ctx = transaction;
+	opt.feed_pipe_cb_data = &feed_ctx;
 
-	close(proc.in);
-	sigchain_pop(SIGPIPE);
-	strbuf_release(&buf);
+	strbuf_init(&feed_ctx.buf, 0);
+
+	ret = run_hooks_opt(transaction->ref_store->repo, "reference-transaction", &opt);
 
-	ret |= finish_command(&proc);
+	strbuf_release(&feed_ctx.buf);
 	return ret;
 }
 
-- 
2.51.0

[PATCH v3 06/10] hook: allow overriding the ungroup option

[Adrian Ratiu]

When calling run_process_parallel() in run_hooks_opt(), the
ungroup option is currently hardcoded to .ungroup = 1.

This causes problems when ungrouping should be disabled, for
example when sideband-reading collated output from child hooks,
because sideband-reading and ungrouping are mutually exclusive.

Thus a new hook.h option is added to allow overriding.

The existing ungroup=1 behavior is preserved in the run_hooks()
API and the "hook run" command. We could modify these to take
an option if necessary, so I added two code comments there.

Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/hook.c | 6 ++++++
 commit.c       | 3 +++
 hook.c         | 5 ++++-
 hook.h         | 5 +++++
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/builtin/hook.c b/builtin/hook.c
index 7afec380d2..73e7b8c2e8 100644
--- a/builtin/hook.c
+++ b/builtin/hook.c
@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,
 	if (!argc)
 		goto usage;
 
+	/*
+	 * All current "hook run" use-cases require ungrouped child output.
+	 * If this changes, a hook run argument can be added to toggle it.
+	 */
+	opt.ungroup = 1;
+
 	/*
 	 * Having a -- for "run" when providing <hook-args> is
 	 * mandatory.
diff --git a/commit.c b/commit.c
index 16d91b2bfc..7da33dde86 100644
--- a/commit.c
+++ b/commit.c
@@ -1965,6 +1965,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,
 		strvec_push(&opt.args, arg);
 	va_end(args);
 
+	/* All commit hook use-cases require ungrouping child output. */
+	opt.ungroup = 1;
+
 	opt.invoked_hook = invoked_hook;
 	return run_hooks_opt(the_repository, name, &opt);
 }
diff --git a/hook.c b/hook.c
index cd2bb7418a..89c7a7a9cb 100644
--- a/hook.c
+++ b/hook.c
@@ -147,7 +147,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.tr2_label = hook_name,
 
 		.processes = 1,
-		.ungroup = 1,
+		.ungroup = options->ungroup,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
@@ -192,6 +192,9 @@ int run_hooks(struct repository *r, const char *hook_name)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
+	/* All use-cases of this API require ungrouping. */
+	opt.ungroup = 1;
+
 	return run_hooks_opt(r, hook_name, &opt);
 }
 
diff --git a/hook.h b/hook.h
index dd87326a5a..dc9eff1e57 100644
--- a/hook.h
+++ b/hook.h
@@ -34,6 +34,11 @@ struct run_hooks_opt
 	 */
 	int *invoked_hook;
 
+	/**
+	 * Allow hooks to set run_processes_parallel() 'ungroup' behavior.
+	 */
+	unsigned int ungroup:1;
+
 	/**
 	 * Path to file which should be piped to stdin for each hook.
 	 */
-- 
2.51.0

[PATCH v3 07/10] run-command: allow capturing of collated output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some callers, for example server-side hooks which wish to relay hook
output to clients across a transport, want to capture what would
normally print to stderr and do something else with it. Allow that via a
callback.

By calling the callback regardless of whether there's output available,
we allow clients to send e.g. a keepalive if necessary.

Because we expose a strbuf, not a fd or FILE*, there's no need to create
a temporary pipe or similar - we can just skip the print to stderr and
instead hand it to the caller.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 30 ++++++++++++++++++++++--------
 run-command.h               | 17 +++++++++++++++++
 t/helper/test-run-command.c | 15 +++++++++++++++
 t/t0061-run-command.sh      |  7 +++++++
 4 files changed, 61 insertions(+), 8 deletions(-)

diff --git a/run-command.c b/run-command.c
index 9bf5911727..a7bf642647 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1578,7 +1578,10 @@ static void pp_cleanup(struct parallel_processes *pp,
 	 * When get_next_task added messages to the buffer in its last
 	 * iteration, the buffered output is non empty.
 	 */
-	strbuf_write(&pp->buffered_output, stderr);
+	if (opts->consume_output)
+		opts->consume_output(&pp->buffered_output, opts->data);
+	else
+		strbuf_write(&pp->buffered_output, stderr);
 	strbuf_release(&pp->buffered_output);
 
 	sigchain_pop_common();
@@ -1717,13 +1720,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 	}
 }
 
-static void pp_output(const struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp,
+		      const struct run_process_parallel_opts *opts)
 {
 	size_t i = pp->output_owner;
 
 	if (pp->children[i].state == GIT_CP_WORKING &&
 	    pp->children[i].err.len) {
-		strbuf_write(&pp->children[i].err, stderr);
+		if (opts->consume_output)
+			opts->consume_output(&pp->children[i].err, opts->data);
+		else
+			strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
 	}
 }
@@ -1771,11 +1778,15 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		} else {
 			const size_t n = opts->processes;
 
-			strbuf_write(&pp->children[i].err, stderr);
+			/* Output errors, then all other finished child processes */
+			if (opts->consume_output) {
+				opts->consume_output(&pp->children[i].err, opts->data);
+				opts->consume_output(&pp->buffered_output, opts->data);
+			} else {
+				strbuf_write(&pp->children[i].err, stderr);
+				strbuf_write(&pp->buffered_output, stderr);
+			}
 			strbuf_reset(&pp->children[i].err);
-
-			/* Output all other finished child processes */
-			strbuf_write(&pp->buffered_output, stderr);
 			strbuf_reset(&pp->buffered_output);
 
 			/*
@@ -1817,7 +1828,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,
 		}
 	} else {
 		pp_buffer_stderr(pp, opts, output_timeout);
-		pp_output(pp);
+		pp_output(pp, opts);
 	}
 }
 
@@ -1840,6 +1851,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	if (opts->ungroup && opts->consume_output)
+		BUG("ungroup and reading output are mutualy exclusive");
+
 	/*
 	 * Child tasks might receive input via stdin, terminating early (or not), so
 	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
diff --git a/run-command.h b/run-command.h
index e536ed7544..67df2f005e 100644
--- a/run-command.h
+++ b/run-command.h
@@ -436,6 +436,17 @@ typedef int (*feed_pipe_fn)(int child_in,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * If this callback is provided, output is collated into a new pipe instead
+ * of the process stderr. Then `consume_output_fn` will be called repeatedly
+ * with output contained in the `output` arg. It will also be called with an
+ * empty `output` to allow for keepalives or similar operations if necessary.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel.
+ * No task cookie is provided because the callback receives collated output.
+ */
+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -495,6 +506,12 @@ struct run_process_parallel_opts
 	 */
 	feed_pipe_fn feed_pipe;
 
+	/*
+	 * consume_output: see consume_output_fn() above. This can be NULL
+	 * to omit any special handling.
+	 */
+	consume_output_fn consume_output;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 4a56456894..49eace8dce 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,
 	return 0;
 }
 
+static void test_divert_output(struct strbuf *output, void *cb UNUSED)
+{
+	FILE *output_file;
+
+	output_file = fopen("./output_file", "a");
+
+	strbuf_write(output, output_file);
+	fclose(output_file);
+}
+
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)
 		.get_next_task = next_test,
 		.start_failure = test_failed,
 		.feed_pipe = test_stdin_pipe_feed,
+		.consume_output = test_divert_output,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished_quiet;
 		opts.feed_pipe = test_stdin_pipe_feed;
+	} else if (!strcmp(argv[1], "run-command-divert-output")) {
+		opts.get_next_task = parallel_next;
+		opts.consume_output = test_divert_output;
+		opts.task_finished = task_finished_quiet;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 2f77fde0d9..74529e219e 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command can divert output' '
+	test_when_finished rm output_file &&
+	test-tool run-command run-command-divert-output 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
+	test_must_be_empty actual &&
+	test_cmp expect output_file
+'
+
 test_expect_success 'run_command listens to stdin' '
 	cat >expect <<-\EOF &&
 	preloaded output of a child
-- 
2.51.0

[PATCH v3 08/10] hooks: allow callers to capture output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some server-side hooks will require capturing output to send over
sideband instead of printing directly to stderr. Expose that capability.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c | 1 +
 hook.h | 8 ++++++++
 2 files changed, 9 insertions(+)

diff --git a/hook.c b/hook.c
index 89c7a7a9cb..4ca6ea36b2 100644
--- a/hook.c
+++ b/hook.c
@@ -152,6 +152,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
 		.feed_pipe = options->feed_pipe,
+		.consume_output = options->consume_output,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
diff --git a/hook.h b/hook.h
index dc9eff1e57..1df604f0e0 100644
--- a/hook.h
+++ b/hook.h
@@ -80,6 +80,14 @@ struct run_hooks_opt
 	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
 	 */
 	void *feed_pipe_cb_data;
+
+	/*
+	 * Populate this to capture output and prevent it from being printed to
+	 * stderr. This will be passed directly through to
+	 * run_command:run_parallel_processes(). See t/helper/test-run-command.c
+	 * for an example.
+	 */
+	consume_output_fn consume_output;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.0

[PATCH v3 09/10] receive-pack: convert update hooks to new API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Use the new hook sideband API introduced in the previous commit.

The hook API avoids creating a custom struct child_process and other
internal hook plumbing (e.g. calling find_hook()) and prepares for
the specification of hooks via configs or running parallel hooks.

Execution is still sequential through the current hook.[ch] via the
run_process_parallel_opts.processes=1 arg.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 60 +++++++++++++++---------------------------
 1 file changed, 21 insertions(+), 39 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index e8ee0e7321..d95df748cd 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,
 	return status;
 }
 
-static int run_update_hook(struct command *cmd)
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	int code;
-	const char *hook_path = find_hook(the_repository, "update");
-
-	if (!hook_path)
-		return 0;
+	if (output && output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
+}
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, cmd->ref_name);
-	strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
-	strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
+static int run_update_hook(struct command *cmd)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "update";
+	strvec_pushl(&opt.args,
+		     cmd->ref_name,
+		     oid_to_hex(&cmd->old_oid),
+		     oid_to_hex(&cmd->new_oid),
+		     NULL);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
 	if (use_sideband)
-		copy_to_sideband(proc.err, -1, NULL);
-	return finish_command(&proc);
+		opt.consume_output = hook_output_to_sideband;
+
+	return run_hooks_opt(the_repository, "update", &opt);
 }
 
 static struct command *find_command_by_refname(struct command *list,
@@ -1639,33 +1634,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 static void run_update_post_hook(struct command *commands)
 {
 	struct command *cmd;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	const char *hook;
-
-	hook = find_hook(the_repository, "post-update");
-	if (!hook)
-		return;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		if (cmd->error_string || cmd->did_not_exist)
 			continue;
-		if (!proc.args.nr)
-			strvec_push(&proc.args, hook);
-		strvec_push(&proc.args, cmd->ref_name);
+		strvec_push(&opt.args, cmd->ref_name);
 	}
-	if (!proc.args.nr)
+	if (!opt.args.nr)
 		return;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "post-update";
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
 
-	if (!start_command(&proc)) {
-		if (use_sideband)
-			copy_to_sideband(proc.err, -1, NULL);
-		finish_command(&proc);
-	}
+	run_hooks_opt(the_repository, "post-update", &opt);
 }
 
 static void check_aliased_update_internal(struct command *cmd,
-- 
2.51.0

[PATCH v3 10/10] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This converts the last remaining hooks to the new hook API, for
the same benefits as the previous conversions (no need to toggle
signals, manage custom struct child_process, call find_hook(),
prepares for specifyinig hooks via configs, etc.).

I noticed a performance degradation when processing large amounts
of hook input with just 1 line per callback, due to run-command's
poll loop, therefore I batched 500 lines per callback, to ensure
similar pipe throughput as before and to avoid hook child waiting
on stdin.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 221 ++++++++++++++++++-----------------------
 1 file changed, 99 insertions(+), 122 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index d95df748cd..be8a270a43 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -748,7 +748,7 @@ static int check_cert_push_options(const struct string_list *push_options)
 	return retval;
 }
 
-static void prepare_push_cert_sha1(struct child_process *proc)
+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)
 {
 	static int already_done;
 
@@ -774,23 +774,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)
 		nonce_status = check_nonce(sigcheck.payload);
 	}
 	if (!is_null_oid(&push_cert_oid)) {
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT=%s",
 			     oid_to_hex(&push_cert_oid));
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_SIGNER=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_SIGNER=%s",
 			     sigcheck.signer ? sigcheck.signer : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_KEY=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_KEY=%s",
 			     sigcheck.key ? sigcheck.key : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_STATUS=%c",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_STATUS=%c",
 			     sigcheck.result);
 		if (push_cert_nonce) {
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE=%s",
 				     push_cert_nonce);
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE_STATUS=%s",
 				     nonce_status);
 			if (nonce_status == NONCE_SLOP)
-				strvec_pushf(&proc->env,
+				strvec_pushf(&opt->env,
 					     "GIT_PUSH_CERT_NONCE_SLOP=%ld",
 					     nonce_stamp_slop);
 		}
@@ -802,119 +802,71 @@ struct receive_hook_feed_state {
 	struct ref_push_report *report;
 	int skip_broken;
 	struct strbuf buf;
-	const struct string_list *push_options;
 };
 
-typedef int (*feed_fn)(void *, const char **, size_t *);
-static int run_and_feed_hook(const char *hook_name, feed_fn feed,
-			     struct receive_hook_feed_state *feed_state)
+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct async muxer;
-	int code;
-	const char *hook_path = find_hook(the_repository, hook_name);
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct receive_hook_feed_state *state = hook_cb->options->feed_pipe_cb_data;
+	struct command *cmd = state->cmd;
+	unsigned int lines_batch_size = 500;
 
-	if (!hook_path)
-		return 0;
+	strbuf_reset(&state->buf);
 
-	strvec_push(&proc.args, hook_path);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = hook_name;
-
-	if (feed_state->push_options) {
-		size_t i;
-		for (i = 0; i < feed_state->push_options->nr; i++)
-			strvec_pushf(&proc.env,
-				     "GIT_PUSH_OPTION_%"PRIuMAX"=%s",
-				     (uintmax_t)i,
-				     feed_state->push_options->items[i].string);
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
-			     (uintmax_t)feed_state->push_options->nr);
-	} else
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
+	/* batch lines to avoid going through run-command's poll loop for each line */
+	for (unsigned int i = 0; i < lines_batch_size; i++) {
+		while (cmd &&
+		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
+			cmd = cmd->next;
 
-	if (tmp_objdir)
-		strvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));
+		if (!cmd)
+			break;  /* no more commands left */
 
-	if (use_sideband) {
-		memset(&muxer, 0, sizeof(muxer));
-		muxer.proc = copy_to_sideband;
-		muxer.in = -1;
-		code = start_async(&muxer);
-		if (code)
-			return code;
-		proc.err = muxer.in;
-	}
+		if (!state->report)
+			state->report = cmd->report;
 
-	prepare_push_cert_sha1(&proc);
+		if (state->report) {
+			struct object_id *old_oid;
+			struct object_id *new_oid;
+			const char *ref_name;
 
-	code = start_command(&proc);
-	if (code) {
-		if (use_sideband)
-			finish_async(&muxer);
-		return code;
-	}
+			old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
+			new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
+			ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(old_oid), oid_to_hex(new_oid),
+				    ref_name);
 
-	while (1) {
-		const char *buf;
-		size_t n;
-		if (feed(feed_state, &buf, &n))
-			break;
-		if (write_in_full(proc.in, buf, n) < 0)
-			break;
+			state->report = state->report->next;
+			if (!state->report)
+				cmd = cmd->next;
+		} else {
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
+				    cmd->ref_name);
+			cmd = cmd->next;
+		}
 	}
-	close(proc.in);
-	if (use_sideband)
-		finish_async(&muxer);
 
-	sigchain_pop(SIGPIPE);
+	state->cmd = cmd;
 
-	return finish_command(&proc);
+	if (state->buf.len > 0) {
+		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
+		if (ret < 0) {
+			if (errno == EPIPE)
+				return 1; /* child closed pipe */
+			return ret;
+		}
+	}
+
+	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
 }
 
-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
 {
-	struct receive_hook_feed_state *state = state_;
-	struct command *cmd = state->cmd;
-
-	while (cmd &&
-	       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
-		cmd = cmd->next;
-	if (!cmd)
-		return -1; /* EOF */
-	if (!bufp)
-		return 0; /* OK, can feed something. */
-	strbuf_reset(&state->buf);
-	if (!state->report)
-		state->report = cmd->report;
-	if (state->report) {
-		struct object_id *old_oid;
-		struct object_id *new_oid;
-		const char *ref_name;
-
-		old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
-		new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
-		ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(old_oid), oid_to_hex(new_oid),
-			    ref_name);
-		state->report = state->report->next;
-		if (!state->report)
-			state->cmd = cmd->next;
-	} else {
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
-			    cmd->ref_name);
-		state->cmd = cmd->next;
-	}
-	if (bufp) {
-		*bufp = state->buf.buf;
-		*sizep = state->buf.len;
-	}
-	return 0;
+	if (output && output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
 }
 
 static int run_receive_hook(struct command *commands,
@@ -922,26 +874,51 @@ static int run_receive_hook(struct command *commands,
 			    int skip_broken,
 			    const struct string_list *push_options)
 {
-	struct receive_hook_feed_state state;
-	int status;
-
-	strbuf_init(&state.buf, 0);
-	state.cmd = commands;
-	state.skip_broken = skip_broken;
-	state.report = NULL;
-	if (feed_receive_hook(&state, NULL, NULL))
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct command *iter = commands;
+	struct receive_hook_feed_state *feed_state;
+	int ret;
+
+	/* if there are no valid commands, don't invoke the hook at all. */
+	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
+		iter = iter->next;
+	if (!iter)
 		return 0;
-	state.cmd = commands;
-	state.push_options = push_options;
-	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
-	strbuf_release(&state.buf);
-	return status;
-}
 
-static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
-{
-	if (output && output->len)
-		send_sideband(1, 2, output->buf, output->len, use_sideband);
+	if (push_options) {
+		int i;
+		for (i = 0; i < push_options->nr; i++)
+			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
+				     push_options->items[i].string);
+		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
+					     (uintmax_t)push_options->nr);
+	} else
+		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+
+	if (tmp_objdir)
+		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+
+	prepare_push_cert_sha1(&opt);
+
+	/* set up sideband printer */
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
+
+	/* set up stdin callback */
+	feed_state = xmalloc(sizeof(struct receive_hook_feed_state));
+	feed_state->cmd = commands;
+	feed_state->skip_broken = skip_broken;
+	feed_state->report = NULL;
+	strbuf_init(&feed_state->buf, 0);
+	opt.feed_pipe_cb_data = feed_state;
+	opt.feed_pipe = feed_receive_hook_cb;
+
+	ret = run_hooks_opt(the_repository, hook_name, &opt);
+
+	strbuf_release(&feed_state->buf);
+	FREE_AND_NULL(opt.feed_pipe_cb_data);
+
+	return ret;
 }
 
 static int run_update_hook(struct command *cmd)
-- 
2.51.0

Re: [PATCH v3 04/10] transport: convert pre-push to hook API

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> From: Emily Shaffer <emilyshaffer@google.com>
>
> Move the pre-push hook from custom run-command invocations to
> the new hook API which doesn't require a custom child_process
> structure and signal toggling.
>
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
> ---
>  transport.c | 95 ++++++++++++++++++++++++++++-------------------------
>  1 file changed, 51 insertions(+), 44 deletions(-)

So, this completes what 01/10 hinted at when it created a
generalized interface modelled after how pre-push hook was run.  We
used to spawn the pre-push hook and fed its standard input by
calling write_in_full().  Now that is largely encapsulated in
run_hooks_opt(), but the application specific processing (namely,
what we write to the pre-push hook, i.e. the list of ref update
status) is given in pre_push_hook_feed_stdin() callback defined here
and given to the run_hooks_opt() call.

In other words, the mechanisms are very cleanly separated between
generic machinery and the client specific processing.  Nice.

How and where does the pipe we are writing into (i.e. hook_stdin_fd)
gets closed when we are done with the child?

> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
> +{
> +	struct hook_cb_data *hook_cb = pp_cb;
> +	struct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;
> +	const struct ref *r = data->refs;
> +	int ret = 0;
>  
> -	strvec_push(&proc.args, hook_path);
> -	strvec_push(&proc.args, transport->remote->name);
> -	strvec_push(&proc.args, transport->url);
> +	if (!r)
> +		return 1; /* no more refs */
>  
> -	proc.in = -1;
> -	proc.trace2_hook_name = "pre-push";
> +	data->refs = r->next;
>  
> -	if (start_command(&proc)) {
> -		finish_command(&proc);
> -		return -1;
> +	switch (r->status) {
> +	case REF_STATUS_REJECT_ALREADY_EXISTS:
> +	case REF_STATUS_REJECT_FETCH_FIRST:
> +	case REF_STATUS_REJECT_NEEDS_FORCE:
> +	case REF_STATUS_REJECT_NODELETE:
> +	case REF_STATUS_REJECT_NONFASTFORWARD:
> +	case REF_STATUS_REJECT_REMOTE_UPDATED:
> +	case REF_STATUS_REJECT_SHALLOW:
> +	case REF_STATUS_REJECT_STALE:
> +	case REF_STATUS_UPTODATE:
> +		return 0; /* skip refs which won't be pushed */
> +	default:
> +		break;
>  	}
>  
> -	sigchain_push(SIGPIPE, SIG_IGN);
> +	if (!r->peer_ref)
> +		return 0;
>  
> -	strbuf_init(&buf, 256);
> +	strbuf_reset(&data->buf);
> +	strbuf_addf(&data->buf, "%s %s %s %s\n",
> +		    r->peer_ref->name, oid_to_hex(&r->new_oid),
> +		    r->name, oid_to_hex(&r->old_oid));
>  
> -	for (r = remote_refs; r; r = r->next) {
> -		if (!r->peer_ref) continue;
> -		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
> -		if (r->status == REF_STATUS_REJECT_STALE) continue;
> -		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
> -		if (r->status == REF_STATUS_UPTODATE) continue;
> +	ret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);
> +	if (ret < 0 && errno != EPIPE)
> +		return ret; /* We do not mind if a hook does not read all refs. */
>  
> -		strbuf_reset(&buf);
> -		strbuf_addf( &buf, "%s %s %s %s\n",
> -			 r->peer_ref->name, oid_to_hex(&r->new_oid),
> -			 r->name, oid_to_hex(&r->old_oid));
> +	return 0;
> +}
>  
> -		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
> -			/* We do not mind if a hook does not read all refs. */
> -			if (errno != EPIPE)
> -				ret = -1;
> -			break;
> -		}
> -	}
> +static int run_pre_push_hook(struct transport *transport,
> +			     struct ref *remote_refs)
> +{
> +	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
> +	struct feed_pre_push_hook_data data;
> +	int ret = 0;
> +
> +	strvec_push(&opt.args, transport->remote->name);
> +	strvec_push(&opt.args, transport->url);
>  
> -	strbuf_release(&buf);
> +	strbuf_init(&data.buf, 0);
> +	data.refs = remote_refs;
>  
> -	x = close(proc.in);
> -	if (!ret)
> -		ret = x;
> +	opt.feed_pipe = pre_push_hook_feed_stdin;
> +	opt.feed_pipe_cb_data = &data;
>  
> -	sigchain_pop(SIGPIPE);
> +	ret = run_hooks_opt(the_repository, "pre-push", &opt);
>  
> -	x = finish_command(&proc);
> -	if (!ret)
> -		ret = x;
> +	strbuf_release(&data.buf);
>  
>  	return ret;
>  }

Re: [PATCH v3 01/10] run-command: add stdin callback for parallelization

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> +static void pp_buffer_stdin(struct parallel_processes *pp,
> +			    const struct run_process_parallel_opts *opts)
> +{
> +	/* Buffer stdin for each pipe. */
> +	for (size_t i = 0; i < opts->processes; i++) {
> +		struct child_process *proc = &pp->children[i].process;
> +		int ret;
> +
> +		if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)
> +			continue;

This combination of two conditions is a recurring theme in this
series (e.g., something close to the reverse of this is called
child-ready-for-cleanup later).

I wonder if it makes the code easier to follow if we add a set of
helpers that take a pointer to &pp->children[i] (and we probably
should give the struct that describes each child managed in the
"struct parallel_processes" some name) and asks "is this thing
done?" etc.

> +	/*
> +	 * Child tasks might receive input via stdin, terminating early (or not), so
> +	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
> +	 * actually writes the data to children stdin fds.
> +	 */
> +	sigchain_push(SIGPIPE, SIG_IGN);
> +
>  	pp_init(&pp, opts, &pp_sig);
>  	while (1) {
>  		for (i = 0;
> @@ -1792,13 +1864,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  		}
>  		if (!pp.nr_processes)
>  			break;
> -		if (opts->ungroup) {
> -			for (size_t i = 0; i < opts->processes; i++)
> -				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
> -		} else {
> -			pp_buffer_stderr(&pp, opts, output_timeout);
> -			pp_output(&pp);
> -		}
> +		pp_handle_child_IO(&pp, opts, output_timeout);

OK, this helper roughly does the same thing as the removed if/else
(and a bit more).

> @@ -1809,6 +1875,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  
>  	pp_cleanup(&pp, opts);
>  
> +	sigchain_pop(SIGPIPE);

OK.

> diff --git a/run-command.h b/run-command.h
> index 0df25e445f..e536ed7544 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,
>  				void *pp_cb,
>  				void *pp_task_cb);
>  
> +/**
> + * This callback is repeatedly called on every child process who requests
> + * start_command() to create a pipe by setting child_process.in < 0.
> + *
> + * pp_cb is the callback cookie as passed into run_processes_parallel, and
> + * pp_task_cb is the callback cookie as passed into get_next_task_fn.
> + * The contents of 'send' will be read into the pipe and passed to the pipe.

There is no 'send' seen around here.  Does this refer to something
a later step adds?

Re: [PATCH v3 01/10] run-command: add stdin callback for parallelization

[Adrian Ratiu]

On Tue, 25 Nov 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Adrian Ratiu <adrian.ratiu@collabora.com> writes: 
> 
>> +static void pp_buffer_stdin(struct parallel_processes *pp, + 
>> const struct run_process_parallel_opts *opts) +{ +	/* Buffer 
>> stdin for each pipe. */ +	for (size_t i = 0; i < 
>> opts->processes; i++) { +		struct child_process *proc 
>> = &pp->children[i].process; +		int ret; + + 
>> if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0) + 
>> continue; 
> 
> This combination of two conditions is a recurring theme in this 
> series (e.g., something close to the reverse of this is called 
> child-ready-for-cleanup later). 
> 
> I wonder if it makes the code easier to follow if we add a set 
> of helpers that take a pointer to &pp->children[i] (and we 
> probably should give the struct that describes each child 
> managed in the "struct parallel_processes" some name) and asks 
> "is this thing done?" etc.

I think this is a good idea and worth exploring.

Will see what I can come up and if this makes code patterns 
clearer. Thanks!
 
>> diff --git a/run-command.h b/run-command.h index 
>> 0df25e445f..e536ed7544 100644 --- a/run-command.h +++ 
>> b/run-command.h @@ -420,6 +420,22 @@ typedef int 
>> (*start_failure_fn)(struct strbuf *out, 
>>  				void *pp_cb, void *pp_task_cb);  
>> +/** + * This callback is repeatedly called on every child 
>> process who requests + * start_command() to create a pipe by 
>> setting child_process.in < 0.  + * + * pp_cb is the callback 
>> cookie as passed into run_processes_parallel, and + * 
>> pp_task_cb is the callback cookie as passed into 
>> get_next_task_fn.  + * The contents of 'send' will be read into 
>> the pipe and passed to the pipe. 
> 
> There is no 'send' seen around here.  Does this refer to 
> something a later step adds? 

Good catch. This is a leftover comment from one of the previous patch
iterations. Will remove.

Re: [PATCH v3 04/10] transport: convert pre-push to hook API

[Adrian Ratiu]

On Mon, 24 Nov 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Adrian Ratiu <adrian.ratiu@collabora.com> writes: 
> 
>> From: Emily Shaffer <emilyshaffer@google.com> 
>> 
>> Move the pre-push hook from custom run-command invocations to 
>> the new hook API which doesn't require a custom child_process 
>> structure and signal toggling. 
>> 
>> Signed-off-by: Emily Shaffer <emilyshaffer@google.com> 
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> 
>> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com> --- 
>>  transport.c | 95 
>>  ++++++++++++++++++++++++++++------------------------- 1 file 
>>  changed, 51 insertions(+), 44 deletions(-) 
> 
> So, this completes what 01/10 hinted at when it created a 
> generalized interface modelled after how pre-push hook was run. 
> We used to spawn the pre-push hook and fed its standard input by 
> calling write_in_full().  Now that is largely encapsulated in 
> run_hooks_opt(), but the application specific processing 
> (namely, what we write to the pre-push hook, i.e. the list of 
> ref update status) is given in pre_push_hook_feed_stdin() 
> callback defined here and given to the run_hooks_opt() call. 
> 
> In other words, the mechanisms are very cleanly separated 
> between generic machinery and the client specific processing. 
> Nice. 
> 
> How and where does the pipe we are writing into 
> (i.e. hook_stdin_fd) gets closed when we are done with the 
> child? 
 
That happens in run-command.c:pp_buffer_stdin() which calls the 
feed pipe callback then closes the fd when feeding is finished:

/** 
 * Feed the pipe: *   ret < 0 means error *   ret == 0 means there 
 is more data to be fed *   ret > 0 means feeding finished */
ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
...

Re: [PATCH v3 02/10] hook: provide stdin via callback

[Adrian Ratiu]

On Mon, 24 Nov 2025, Adrian Ratiu <adrian.ratiu@collabora.com> 
wrote:
> From: Emily Shaffer <emilyshaffer@google.com> 
> 
> This adds a callback mechanism for feeding stdin to hooks 
> alongside the existing path_to_stdin (which slurps a file's 
> content to stdin). 
> 
> The advantage of this new callback is that it can feed stdin 
> without going through the FS layer. This helps when feeding 
> large amount of data and uses the run-command parallel stdin 
> callback introduced in the preceding commit. 
> 
> Signed-off-by: Emily Shaffer <emilyshaffer@google.com> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> 
> Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com> --- 
>  hook.c | 15 +++++++++++++++ hook.h | 38 
>  ++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 
>  insertions(+) 
> 
> diff --git a/hook.c b/hook.c index b3de1048bf..cd2bb7418a 100644 
> --- a/hook.c +++ b/hook.c @@ -65,11 +65,22 @@ static int 
> pick_next_hook(struct child_process *cp, 
>   cp->no_stdin = 1; strvec_pushv(&cp->env, 
>  hook_cb->options->env.v); 
> + +	if (hook_cb->options->path_to_stdin && 
> hook_cb->options->feed_pipe) +		BUG("options 
> path_to_stdin and feed_pipe are mutually exclusive"); + 
>  	/* reopen the file for stdin; run_command closes it. */ if 
>  (hook_cb->options->path_to_stdin) { cp->no_stdin = 0; cp->in = 
>  xopen(hook_cb->options->path_to_stdin, O_RDONLY); } 
> + +	if (hook_cb->options->feed_pipe) { + 
> cp->no_stdin = 0; +		/* start_command() will allocate a 
> pipe / stdin fd for us */ +		cp->in = -1; +	} + 
>  	cp->stdout_to_stderr = 1; cp->trace2_hook_name = 
>  hook_cb->hook_name; cp->dir = hook_cb->options->dir; 
> @@ -140,6 +151,7 @@ int run_hooks_opt(struct repository *r, 
> const char *hook_name, 
>   .get_next_task = pick_next_hook, .start_failure = 
>  notify_start_failure, 
> +		.feed_pipe = options->feed_pipe, 
>  		.task_finished = notify_hook_finished,  .data = 
>  &cb_data, 
> @@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, 
> const char *hook_name, 
>  	if (!options) BUG("a struct run_hooks_opt must be provided 
>  to run_hooks");  
> +	if (options->path_to_stdin && options->feed_pipe) + 
> BUG("options path_to_stdin and feed_pipe are mutually 
> exclusive"); + 
>  	if (options->invoked_hook) *options->invoked_hook = 0;  
> diff --git a/hook.h b/hook.h index 11863fa734..dd87326a5a 100644 
> --- a/hook.h +++ b/hook.h @@ -1,6 +1,7 @@ 
>  #ifndef HOOK_H #define HOOK_H #include "strvec.h" 
> +#include "run-command.h" 
>   struct repository;  
> @@ -37,6 +38,43 @@ struct run_hooks_opt 
>  	 * Path to file which should be piped to stdin for each 
>  hook.  */ const char *path_to_stdin; 
> + +	/** +	 * Callback used to incrementally feed a child 
> hook stdin pipe.  +	 * +	 * Useful especially if a hook 
> consumes large quantities of data +	 * (e.g. a list of all 
> refs in a client push), so feeding it via +	 * in-memory 
> strings or slurping to/from files is inefficient.  +	 * While 
> the callback allows piecemeal writing, it can also be +	 * 
> used for smaller inputs, where it gets called only once.  +	 * 
> +	 * Add hook callback initalization context to 
> `feed_pipe_ctx`.  +	 * Add Hook callback internal state to 
> `feed_pipe_cb_data`.  +	 * +	 */ +	feed_pipe_fn 
> feed_pipe; + +	/** +	 * Opaque data pointer used to 
> pass context to `feed_pipe_fn`.  +	 * +	 * It can be 
> accessed via the second callback arg: +	 * ((struct 
> hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx; +	 * 
> +	 * The caller is responsible for managing the memory for 
> this data.  +	 * Only useful when using 
> `run_hooks_opt.feed_pipe`, otherwise ignore it.  +	 */ + 
> void *feed_pipe_ctx; + +	/** +	 * Opaque data pointer 
> used to keep internal state across callback calls.  +	 * +	 * 
> It can be accessed via the second callback arg: +	 * 
> ((struct hook_cb_data *) 
> pp_cb)->hook_cb->options->feed_pipe_cb_data; 

I just noticed the small inconsistency in this comment (second cb 
arg vs the actual code example). Will fix in v4.

I also have an idea how to further simplify this API based on the 
parallel hook execution work I'm doing (that is a separate patch 
series built upon this one), so I'll see if I can make this 
simpler, to avoid going through hook_cb->options entirely, similar 
to what Patrick suggested in v2.

Re: [PATCH v3 02/10] hook: provide stdin via callback

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> On Mon, 24 Nov 2025, Adrian Ratiu <adrian.ratiu@collabora.com> 
> wrote:
>> From: Emily Shaffer <emilyshaffer@google.com> 
>> 
>> This adds a callback mechanism for feeding stdin to hooks 
>> ...
>> this data.  +	 * Only useful when using 
>> `run_hooks_opt.feed_pipe`, otherwise ignore it.  +	 */ + 
>> void *feed_pipe_ctx; + +	/** +	 * Opaque data pointer 
>> used to keep internal state across callback calls.  +	 * +	 * 
>> It can be accessed via the second callback arg: +	 * 
>> ((struct hook_cb_data *) 
>> pp_cb)->hook_cb->options->feed_pipe_cb_data; 
>
> I just noticed the small inconsistency in this comment (second cb 
> arg vs the actual code example). Will fix in v4.
>
> I also have an idea how to further simplify this API based on the 
> parallel hook execution work I'm doing (that is a separate patch 
> series built upon this one), so I'll see if I can make this 
> simpler, to avoid going through hook_cb->options entirely, similar 
> to what Patrick suggested in v2.

Not about the contents of your message, but I often see in your
replies unintelligible randomly line-wrapped text, and this message
was an example of such.  You can see how it appears to others by
visiting

  https://lore.kernel.org/git/87iketf0sn.fsf@gentoo.mail-host-address-is-not-set/

I _think_ it has to do with your e-mail client, and possibly use of
the flowed format in

    Content-Type: text/plain; charset=utf-8; format=flowed

Please make sure your e-mail client is set up correctly.

Thanks.

Re: [PATCH v3 02/10] hook: provide stdin via callback

[Adrian Ratiu]

On Sat, 29 Nov 2025, Junio C Hamano <gitster@pobox.com> wrote:
> Not about the contents of your message, but I often see in your 
> replies unintelligible randomly line-wrapped text, and this 
> message was an example of such.  You can see how it appears to 
> others by visiting 
> 
>   https://lore.kernel.org/git/87iketf0sn.fsf@gentoo.mail-host-address-is-not-set/ 
> 
> I _think_ it has to do with your e-mail client, and possibly use 
> of the flowed format in 
> 
>     Content-Type: text/plain; charset=utf-8; format=flowed 
> 
> Please make sure your e-mail client is set up correctly. 
 
Ack, it's either my client or the smtp server. Will fix.

Thanks for raising it up!

[PATCH v4 00/11] Convert remaining hooks to hook.h

[Adrian Ratiu]

Hello everyone,

This series finishes the hook.[ch] conversion for the remaining hooks in
preparation for adding config-based hooks and enabling parallel hook
execution where possible (that will be a separate series from this one).

v4 is a minor cleanup refresh: details + range-diff below.

It is based on the latest master branch. There are no conflicts with
next and seen branches, the code is available on GitHub [1] and a
successful CI run [2] is also provided.

1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v4
2: https://github.com/10ne1/git/actions/runs/19929313184

Changes in v4:
* Simplified hook callbacks by passing 'feed_pipe_cb_data' directly via pp_task_cb (Patrick & Adrian)
* Removed stale comment from run-command.h (Junio)
* Introduced new helpers for pp child states to improve readability (Junio)

v3 -> v4 range-diff:
 -:  ---------- >  1:  b252d447f5 run-command: add first helper for pp child states
 1:  42aef8fea9 !  2:  f6fd3b9b0f run-command: add stdin callback for parallelization
    @@ Commit message
         Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
     
      ## run-command.c ##
    +@@ run-command.c: static int child_is_working(const struct parallel_child *pp_child)
    + 	return pp_child->state == GIT_CP_WORKING;
    + }
    + 
    ++static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)
    ++{
    ++	return child_is_working(pp_child) && !pp_child->process.in;
    ++}
    ++
    ++static int child_is_receiving_input(const struct parallel_child *pp_child)
    ++{
    ++	return child_is_working(pp_child) && pp_child->process.in > 0;
    ++}
    ++
    + struct parallel_processes {
    + 	size_t nr_processes;
    + 
     @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,
      	return 0;
      }
    @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,
     +		struct child_process *proc = &pp->children[i].process;
     +		int ret;
     +
    -+		if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)
    ++		if (!child_is_receiving_input(&pp->children[i]))
     +			continue;
     +
     +		/*
    @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,
     +	pp_buffer_stdin(pp, opts);
     +
     +	if (opts->ungroup) {
    -+		for (size_t i = 0; i < opts->processes; i++) {
    -+			int child_ready_for_cleanup =
    -+				pp->children[i].state == GIT_CP_WORKING &&
    -+				pp->children[i].process.in == 0;
    -+
    -+			if (child_ready_for_cleanup)
    ++		for (size_t i = 0; i < opts->processes; i++)
    ++			if (child_is_ready_for_cleanup(&pp->children[i]))
     +				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
    -+		}
     +	} else {
     +		pp_buffer_stderr(pp, opts, output_timeout);
     +		pp_output(pp);
    @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,
     + *
     + * pp_cb is the callback cookie as passed into run_processes_parallel, and
     + * pp_task_cb is the callback cookie as passed into get_next_task_fn.
    -+ * The contents of 'send' will be read into the pipe and passed to the pipe.
     + *
     + * Returns < 0 for error
     + * Returns == 0 when there is more data to be fed (will be called again)
 2:  00c5b45987 !  3:  59c07b618e hook: provide stdin via callback
    @@ Commit message
         Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
     
      ## hook.c ##
    +@@ hook.c: int hook_exists(struct repository *r, const char *name)
    + static int pick_next_hook(struct child_process *cp,
    + 			  struct strbuf *out UNUSED,
    + 			  void *pp_cb,
    +-			  void **pp_task_cb UNUSED)
    ++			  void **pp_task_cb)
    + {
    + 	struct hook_cb_data *hook_cb = pp_cb;
    + 	const char *hook_path = hook_cb->hook_path;
     @@ hook.c: static int pick_next_hook(struct child_process *cp,
      
      	cp->no_stdin = 1;
    @@ hook.c: static int pick_next_hook(struct child_process *cp,
      	cp->stdout_to_stderr = 1;
      	cp->trace2_hook_name = hook_cb->hook_name;
      	cp->dir = hook_cb->options->dir;
    +@@ hook.c: static int pick_next_hook(struct child_process *cp,
    + 	strvec_push(&cp->args, hook_path);
    + 	strvec_pushv(&cp->args, hook_cb->options->args.v);
    + 
    ++	/*
    ++	 * Provide per-hook internal state via task_cb for easy access, so
    ++	 * hook callbacks don't have to go through hook_cb->options.
    ++	 */
    ++	*pp_task_cb = hook_cb->options->feed_pipe_cb_data;
    ++
    + 	/*
    + 	 * This pick_next_hook() will be called again, we're only
    + 	 * running one hook, so indicate that no more work will be
     @@ hook.c: int run_hooks_opt(struct repository *r, const char *hook_name,
      
      		.get_next_task = pick_next_hook,
    @@ hook.h: struct run_hooks_opt
     +	/**
     +	 * Opaque data pointer used to pass context to `feed_pipe_fn`.
     +	 *
    -+	 * It can be accessed via the second callback arg:
    ++	 * It can be accessed via the second callback arg 'pp_cb':
     +	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;
     +	 *
     +	 * The caller is responsible for managing the memory for this data.
    @@ hook.h: struct run_hooks_opt
     +	/**
     +	 * Opaque data pointer used to keep internal state across callback calls.
     +	 *
    -+	 * It can be accessed via the second callback arg:
    -+	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_cb_data;
    ++	 * It can be accessed directly via the third callback arg 'pp_task_cb':
    ++	 * struct ... *state = pp_task_cb;
     +	 *
     +	 * The caller is responsible for managing the memory for this data.
     +	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
 3:  08792569df =  4:  8f591319a4 hook: convert 'post-rewrite' hook in sequencer.c to hook API
 4:  8a76eb11cf !  5:  2427717a26 transport: convert pre-push to hook API
    @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_
      
     -	if (!hook_path)
     -		return 0;
    -+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
    ++static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
     +{
    -+	struct hook_cb_data *hook_cb = pp_cb;
    -+	struct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;
    ++	struct feed_pre_push_hook_data *data = pp_task_cb;
     +	const struct ref *r = data->refs;
     +	int ret = 0;
      
 5:  92452c6707 !  6:  22467bb074 reference-transaction: use hook API instead of run-command
    @@ refs.c: static int ref_update_reject_duplicates(struct string_list *refnames,
     +	struct strbuf buf;
     +};
     +
    -+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
    ++static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)
      {
     -	struct child_process proc = CHILD_PROCESS_INIT;
     -	struct strbuf buf = STRBUF_INIT;
     -	const char *hook;
     -	int ret = 0;
     +	struct hook_cb_data *hook_cb = pp_cb;
    -+	struct run_hooks_opt *opt = hook_cb->options;
    -+	struct ref_transaction *transaction = opt->feed_pipe_ctx;
    -+	struct transaction_feed_cb_data *feed_cb_data = opt->feed_pipe_cb_data;
    ++	struct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;
    ++	struct transaction_feed_cb_data *feed_cb_data = pp_task_cb;
     +	struct strbuf *buf = &feed_cb_data->buf;
     +	struct ref_update *update;
     +	size_t i = feed_cb_data->index++;
 6:  6ec922ccef =  7:  9b0b13379c hook: allow overriding the ungroup option
 7:  3098e8f31a !  8:  cae3c984e2 run-command: allow capturing of collated output
    @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp,
      {
      	size_t i = pp->output_owner;
      
    - 	if (pp->children[i].state == GIT_CP_WORKING &&
    + 	if (child_is_working(&pp->children[i]) &&
      	    pp->children[i].err.len) {
     -		strbuf_write(&pp->children[i].err, stderr);
     +		if (opts->consume_output)
    @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,
      
      			/*
     @@ run-command.c: static void pp_handle_child_IO(struct parallel_processes *pp,
    - 		}
    + 				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
      	} else {
      		pp_buffer_stderr(pp, opts, output_timeout);
     -		pp_output(pp);
 8:  1a21310174 =  9:  0383be2ebd hooks: allow callers to capture output
 9:  6a2895750a = 10:  d8e234c453 receive-pack: convert update hooks to new API
10:  254e7dd351 ! 11:  6bad9aed4d receive-pack: convert receive hooks to hook API
    @@ builtin/receive-pack.c: struct receive_hook_feed_state {
     -typedef int (*feed_fn)(void *, const char **, size_t *);
     -static int run_and_feed_hook(const char *hook_name, feed_fn feed,
     -			     struct receive_hook_feed_state *feed_state)
    -+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
    ++static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
      {
     -	struct child_process proc = CHILD_PROCESS_INIT;
     -	struct async muxer;
     -	int code;
     -	const char *hook_path = find_hook(the_repository, hook_name);
    -+	struct hook_cb_data *hook_cb = pp_cb;
    -+	struct receive_hook_feed_state *state = hook_cb->options->feed_pipe_cb_data;
    ++	struct receive_hook_feed_state *state = pp_task_cb;
     +	struct command *cmd = state->cmd;
     +	unsigned int lines_batch_size = 500;
      

Adrian Ratiu (3):
  run-command: add first helper for pp child states
  reference-transaction: use hook API instead of run-command
  hook: allow overriding the ungroup option

Emily Shaffer (8):
  run-command: add stdin callback for parallelization
  hook: provide stdin via callback
  hook: convert 'post-rewrite' hook in sequencer.c to hook API
  transport: convert pre-push to hook API
  run-command: allow capturing of collated output
  hooks: allow callers to capture output
  receive-pack: convert update hooks to new API
  receive-pack: convert receive hooks to hook API

 builtin/hook.c              |   6 +
 builtin/receive-pack.c      | 270 +++++++++++++++---------------------
 commit.c                    |   3 +
 hook.c                      |  29 +++-
 hook.h                      |  51 +++++++
 refs.c                      | 100 ++++++-------
 run-command.c               | 142 +++++++++++++++----
 run-command.h               |  38 +++++
 sequencer.c                 |  42 +++---
 t/helper/test-run-command.c |  67 ++++++++-
 t/t0061-run-command.sh      |  38 +++++
 transport.c                 |  94 +++++++------
 12 files changed, 587 insertions(+), 293 deletions(-)

-- 
2.51.0

[PATCH v4 01/11] run-command: add first helper for pp child states

[Adrian Ratiu]

There is a recurring pattern of testing parallel process child states
and file descriptors to determine if a child is running, receiving any
input or if it's ready for cleanup.

Name the pp_child structure and introduce a first helper to make these
checks more readable. Next commits will add more helpers and checks.

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/run-command.c b/run-command.c
index ed9575bd6a..82eeac38bf 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1478,15 +1478,22 @@ enum child_state {
 	GIT_CP_WAIT_CLEANUP,
 };
 
+struct parallel_child {
+	enum child_state state;
+	struct child_process process;
+	struct strbuf err;
+	void *data;
+};
+
+static int child_is_working(const struct parallel_child *pp_child)
+{
+	return pp_child->state == GIT_CP_WORKING;
+}
+
 struct parallel_processes {
 	size_t nr_processes;
 
-	struct {
-		enum child_state state;
-		struct child_process process;
-		struct strbuf err;
-		void *data;
-	} *children;
+	struct parallel_child *children;
 	/*
 	 * The struct pollfd is logically part of *children,
 	 * but the system call expects it as its own array.
@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,
 			  int signo)
 {
 	for (size_t i = 0; i < opts->processes; i++)
-		if (pp->children[i].state == GIT_CP_WORKING)
+		if (child_is_working(&pp->children[i]))
 			kill(pp->children[i].process.pid, signo);
 }
 
@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 
 	/* Buffer output from all pipes. */
 	for (size_t i = 0; i < opts->processes; i++) {
-		if (pp->children[i].state == GIT_CP_WORKING &&
+		if (child_is_working(&pp->children[i]) &&
 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
 			int n = strbuf_read_once(&pp->children[i].err,
 						 pp->children[i].process.err, 0);
@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)
 {
 	size_t i = pp->output_owner;
 
-	if (pp->children[i].state == GIT_CP_WORKING &&
+	if (child_is_working(&pp->children[i]) &&
 	    pp->children[i].err.len) {
 		strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			 * running process time.
 			 */
 			for (i = 0; i < n; i++)
-				if (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)
+				if (child_is_working(&pp->children[(pp->output_owner + i) % n]))
 					break;
 			pp->output_owner = (pp->output_owner + i) % n;
 		}
-- 
2.51.0

[PATCH v4 02/11] run-command: add stdin callback for parallelization

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

If a user of the run_processes_parallel() API wants to pipe a large
amount of information to the stdin of each parallel command, that
data could exceed the pipe buffer of the process's stdin and can be
too big to store in-memory via strbuf & friends or to slurp to a file.

Generally this is solved by repeatedly writing to child_process.in
between calls to start_command() and finish_command(). For a specific
pre-existing example of this, see transport.c:run_pre_push_hook().

This adds a generic callback API to run_processes_parallel() to do
exactly that in a unified manner, similar to the existing callback APIs,
which can then be used by hooks.h to convert the remaining hooks to the
new, simpler parallel interface.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 87 ++++++++++++++++++++++++++++++++++---
 run-command.h               | 21 +++++++++
 t/helper/test-run-command.c | 52 +++++++++++++++++++++-
 t/t0061-run-command.sh      | 31 +++++++++++++
 4 files changed, 182 insertions(+), 9 deletions(-)

diff --git a/run-command.c b/run-command.c
index 82eeac38bf..a608d37fb2 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)
 	return pp_child->state == GIT_CP_WORKING;
 }
 
+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)
+{
+	return child_is_working(pp_child) && !pp_child->process.in;
+}
+
+static int child_is_receiving_input(const struct parallel_child *pp_child)
+{
+	return child_is_working(pp_child) && pp_child->process.in > 0;
+}
+
 struct parallel_processes {
 	size_t nr_processes;
 
@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
+static void pp_buffer_stdin(struct parallel_processes *pp,
+			    const struct run_process_parallel_opts *opts)
+{
+	/* Buffer stdin for each pipe. */
+	for (size_t i = 0; i < opts->processes; i++) {
+		struct child_process *proc = &pp->children[i].process;
+		int ret;
+
+		if (!child_is_receiving_input(&pp->children[i]))
+			continue;
+
+		/*
+		 * child input is provided via path_to_stdin when the feed_pipe cb is
+		 * missing, so we just signal an EOF.
+		 */
+		if (!opts->feed_pipe) {
+			close(proc->in);
+			proc->in = 0;
+			continue;
+		}
+
+		/**
+		 * Feed the pipe:
+		 *   ret < 0 means error
+		 *   ret == 0 means there is more data to be fed
+		 *   ret > 0 means feeding finished
+		 */
+		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
+		if (ret < 0)
+			die_errno("feed_pipe");
+
+		if (ret) {
+			close(proc->in);
+			proc->in = 0;
+		}
+	}
+}
+
 static void pp_buffer_stderr(struct parallel_processes *pp,
 			     const struct run_process_parallel_opts *opts,
 			     int output_timeout)
@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		pp->children[i].state = GIT_CP_FREE;
 		if (pp->pfd)
 			pp->pfd[i].fd = -1;
+		pp->children[i].process.in = 0;
 		child_process_init(&pp->children[i].process);
 
 		if (opts->ungroup) {
@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,
 	return result;
 }
 
+static void pp_handle_child_IO(struct parallel_processes *pp,
+				const struct run_process_parallel_opts *opts,
+				int output_timeout)
+{
+	/*
+	 * First push input, if any (it might no-op), to child tasks to avoid them blocking
+	 * after input. This also prevents deadlocks when ungrouping below, if a child blocks
+	 * while the parent also waits for them to finish.
+	 */
+	pp_buffer_stdin(pp, opts);
+
+	if (opts->ungroup) {
+		for (size_t i = 0; i < opts->processes; i++)
+			if (child_is_ready_for_cleanup(&pp->children[i]))
+				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
+	} else {
+		pp_buffer_stderr(pp, opts, output_timeout);
+		pp_output(pp);
+	}
+}
+
 void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	/*
+	 * Child tasks might receive input via stdin, terminating early (or not), so
+	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
+	 * actually writes the data to children stdin fds.
+	 */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
 	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (i = 0;
@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts->ungroup) {
-			for (size_t i = 0; i < opts->processes; i++)
-				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
-		} else {
-			pp_buffer_stderr(&pp, opts, output_timeout);
-			pp_output(&pp);
-		}
+		pp_handle_child_IO(&pp, opts, output_timeout);
 		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 
 	pp_cleanup(&pp, opts);
 
+	sigchain_pop(SIGPIPE);
+
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
diff --git a/run-command.h b/run-command.h
index 0df25e445f..e1ca965b5b 100644
--- a/run-command.h
+++ b/run-command.h
@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * This callback is repeatedly called on every child process who requests
+ * start_command() to create a pipe by setting child_process.in < 0.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel, and
+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.
+ *
+ * Returns < 0 for error
+ * Returns == 0 when there is more data to be fed (will be called again)
+ * Returns > 0 when finished (child closed fd or no more data to be fed)
+ */
+typedef int (*feed_pipe_fn)(int child_in,
+				void *pp_cb,
+				void *pp_task_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -473,6 +488,12 @@ struct run_process_parallel_opts
 	 */
 	start_failure_fn start_failure;
 
+	/*
+	 * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any
+	 * special handling.
+	 */
+	feed_pipe_fn feed_pipe;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 3719f23cc2..4a56456894 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -23,19 +23,26 @@ static int number_callbacks;
 static int parallel_next(struct child_process *cp,
 			 struct strbuf *err,
 			 void *cb,
-			 void **task_cb UNUSED)
+			 void **task_cb)
 {
 	struct child_process *d = cb;
 	if (number_callbacks >= 4)
 		return 0;
 
 	strvec_pushv(&cp->args, d->args.v);
+	cp->in = d->in;
+	cp->no_stdin = d->no_stdin;
 	if (err)
 		strbuf_addstr(err, "preloaded output of a child\n");
 	else
 		fprintf(stderr, "preloaded output of a child\n");
 
 	number_callbacks++;
+
+	/* test_stdin callback will use this to count remaining lines */
+	*task_cb = xmalloc(sizeof(int));
+	*(int*)(*task_cb) = 2;
+
 	return 1;
 }
 
@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
-			 void *pp_task_cb UNUSED)
+			 void *pp_task_cb)
 {
 	if (err)
 		strbuf_addstr(err, "asking for a quick stop\n");
 	else
 		fprintf(stderr, "asking for a quick stop\n");
+
+	FREE_AND_NULL(pp_task_cb);
+
 	return 1;
 }
 
+static int task_finished_quiet(int result UNUSED,
+			       struct strbuf *err UNUSED,
+			       void *pp_cb UNUSED,
+			       void *pp_task_cb)
+{
+	FREE_AND_NULL(pp_task_cb);
+	return 0;
+}
+
+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)
+{
+	int *lines_remaining = task_cb;
+
+	if (*lines_remaining) {
+		struct strbuf buf = STRBUF_INIT;
+		strbuf_addf(&buf, "sample stdin %d\n", --(*lines_remaining));
+		if (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {
+			if (errno == EPIPE) {
+				/* child closed stdin, nothing more to do */
+				strbuf_release(&buf);
+				return 1;
+			}
+			die_errno("write");
+		}
+		strbuf_release(&buf);
+	}
+
+	return !(*lines_remaining);
+}
+
 struct testsuite {
 	struct string_list tests, failed;
 	int next;
@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)
 	struct run_process_parallel_opts opts = {
 		.get_next_task = next_test,
 		.start_failure = test_failed,
+		.feed_pipe = test_stdin_pipe_feed,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
 		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		opts.get_next_task = no_job;
 		opts.task_finished = task_finished;
+	} else if (!strcmp(argv[1], "run-command-stdin")) {
+		proc.in = -1;
+		proc.no_stdin = 0;
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
+		opts.feed_pipe = test_stdin_pipe_feed;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 76d4936a87..2f77fde0d9 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command listens to stdin' '
+	cat >expect <<-\EOF &&
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	EOF
+
+	write_script stdin-script <<-\EOF &&
+	echo "listening for stdin:"
+	while read line
+	do
+		echo "$line"
+	done
+	EOF
+	test-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&
+	test_cmp expect actual
+'
+
 cat >expect <<-EOF
 preloaded output of a child
 asking for a quick stop
-- 
2.51.0

[PATCH v4 03/11] hook: provide stdin via callback

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This adds a callback mechanism for feeding stdin to hooks alongside
the existing path_to_stdin (which slurps a file's content to stdin).

The advantage of this new callback is that it can feed stdin without
going through the FS layer. This helps when feeding large amount of
data and uses the run-command parallel stdin callback introduced in
the preceding commit.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 hook.c | 23 ++++++++++++++++++++++-
 hook.h | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/hook.c b/hook.c
index b3de1048bf..5ddd7678d1 100644
--- a/hook.c
+++ b/hook.c
@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)
 static int pick_next_hook(struct child_process *cp,
 			  struct strbuf *out UNUSED,
 			  void *pp_cb,
-			  void **pp_task_cb UNUSED)
+			  void **pp_task_cb)
 {
 	struct hook_cb_data *hook_cb = pp_cb;
 	const char *hook_path = hook_cb->hook_path;
@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,
 
 	cp->no_stdin = 1;
 	strvec_pushv(&cp->env, hook_cb->options->env.v);
+
+	if (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	/* reopen the file for stdin; run_command closes it. */
 	if (hook_cb->options->path_to_stdin) {
 		cp->no_stdin = 0;
 		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
 	}
+
+	if (hook_cb->options->feed_pipe) {
+		cp->no_stdin = 0;
+		/* start_command() will allocate a pipe / stdin fd for us */
+		cp->in = -1;
+	}
+
 	cp->stdout_to_stderr = 1;
 	cp->trace2_hook_name = hook_cb->hook_name;
 	cp->dir = hook_cb->options->dir;
@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,
 	strvec_push(&cp->args, hook_path);
 	strvec_pushv(&cp->args, hook_cb->options->args.v);
 
+	/*
+	 * Provide per-hook internal state via task_cb for easy access, so
+	 * hook callbacks don't have to go through hook_cb->options.
+	 */
+	*pp_task_cb = hook_cb->options->feed_pipe_cb_data;
+
 	/*
 	 * This pick_next_hook() will be called again, we're only
 	 * running one hook, so indicate that no more work will be
@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
+		.feed_pipe = options->feed_pipe,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
 
+	if (options->path_to_stdin && options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	if (options->invoked_hook)
 		*options->invoked_hook = 0;
 
diff --git a/hook.h b/hook.h
index 11863fa734..51cab785ea 100644
--- a/hook.h
+++ b/hook.h
@@ -1,6 +1,7 @@
 #ifndef HOOK_H
 #define HOOK_H
 #include "strvec.h"
+#include "run-command.h"
 
 struct repository;
 
@@ -37,6 +38,43 @@ struct run_hooks_opt
 	 * Path to file which should be piped to stdin for each hook.
 	 */
 	const char *path_to_stdin;
+
+	/**
+	 * Callback used to incrementally feed a child hook stdin pipe.
+	 *
+	 * Useful especially if a hook consumes large quantities of data
+	 * (e.g. a list of all refs in a client push), so feeding it via
+	 * in-memory strings or slurping to/from files is inefficient.
+	 * While the callback allows piecemeal writing, it can also be
+	 * used for smaller inputs, where it gets called only once.
+	 *
+	 * Add hook callback initalization context to `feed_pipe_ctx`.
+	 * Add Hook callback internal state to `feed_pipe_cb_data`.
+	 *
+	 */
+	feed_pipe_fn feed_pipe;
+
+	/**
+	 * Opaque data pointer used to pass context to `feed_pipe_fn`.
+	 *
+	 * It can be accessed via the second callback arg 'pp_cb':
+	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_ctx;
+
+	/**
+	 * Opaque data pointer used to keep internal state across callback calls.
+	 *
+	 * It can be accessed directly via the third callback arg 'pp_task_cb':
+	 * struct ... *state = pp_task_cb;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_cb_data;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.0

[PATCH v4 04/11] hook: convert 'post-rewrite' hook in sequencer.c to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Replace the custom run-command calls used by post-rewrite with
the newer and simpler hook_run_opt(), which does not need to
create a custom 'struct child_process' or call find_hook().

Another benefit of using the hook API is that hook_run_opt()
handles the SIGPIPE toggle logic.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 sequencer.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 5476d39ba9..71ed31c774 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,
 	return ret;
 }
 
+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
+{
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
+	int ret;
+
+	if (!to_pipe)
+		BUG("pipe_from_strbuf called without feed_pipe_ctx");
+
+	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
+
+	return 1; /* done writing */
+}
+
 static int run_rewrite_hook(const struct object_id *oldoid,
 			    const struct object_id *newoid)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 	int code;
 	struct strbuf sb = STRBUF_INIT;
-	const char *hook_path = find_hook(the_repository, "post-rewrite");
 
-	if (!hook_path)
-		return 0;
+	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
 
-	strvec_pushl(&proc.args, hook_path, "amend", NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "post-rewrite";
+	opt.feed_pipe_ctx = &sb;
+	opt.feed_pipe = pipe_from_strbuf;
+
+	strvec_push(&opt.args, "amend");
+
+	code = run_hooks_opt(the_repository, "post-rewrite", &opt);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
-	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
-	sigchain_push(SIGPIPE, SIG_IGN);
-	write_in_full(proc.in, sb.buf, sb.len);
-	close(proc.in);
 	strbuf_release(&sb);
-	sigchain_pop(SIGPIPE);
-	return finish_command(&proc);
+	return code;
 }
 
 void commit_post_rewrite(struct repository *r,
-- 
2.51.0

[PATCH v4 05/11] transport: convert pre-push to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Move the pre-push hook from custom run-command invocations to
the new hook API which doesn't require a custom child_process
structure and signal toggling.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 transport.c | 94 ++++++++++++++++++++++++++++-------------------------
 1 file changed, 50 insertions(+), 44 deletions(-)

diff --git a/transport.c b/transport.c
index c7f06a7382..047f2cefba 100644
--- a/transport.c
+++ b/transport.c
@@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
 	die(_("Aborting."));
 }
 
-static int run_pre_push_hook(struct transport *transport,
-			     struct ref *remote_refs)
-{
-	int ret = 0, x;
-	struct ref *r;
-	struct child_process proc = CHILD_PROCESS_INIT;
+struct feed_pre_push_hook_data {
 	struct strbuf buf;
-	const char *hook_path = find_hook(the_repository, "pre-push");
+	const struct ref *refs;
+};
 
-	if (!hook_path)
-		return 0;
+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
+{
+	struct feed_pre_push_hook_data *data = pp_task_cb;
+	const struct ref *r = data->refs;
+	int ret = 0;
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, transport->remote->name);
-	strvec_push(&proc.args, transport->url);
+	if (!r)
+		return 1; /* no more refs */
 
-	proc.in = -1;
-	proc.trace2_hook_name = "pre-push";
+	data->refs = r->next;
 
-	if (start_command(&proc)) {
-		finish_command(&proc);
-		return -1;
+	switch (r->status) {
+	case REF_STATUS_REJECT_ALREADY_EXISTS:
+	case REF_STATUS_REJECT_FETCH_FIRST:
+	case REF_STATUS_REJECT_NEEDS_FORCE:
+	case REF_STATUS_REJECT_NODELETE:
+	case REF_STATUS_REJECT_NONFASTFORWARD:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+	case REF_STATUS_REJECT_SHALLOW:
+	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_UPTODATE:
+		return 0; /* skip refs which won't be pushed */
+	default:
+		break;
 	}
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	if (!r->peer_ref)
+		return 0;
 
-	strbuf_init(&buf, 256);
+	strbuf_reset(&data->buf);
+	strbuf_addf(&data->buf, "%s %s %s %s\n",
+		    r->peer_ref->name, oid_to_hex(&r->new_oid),
+		    r->name, oid_to_hex(&r->old_oid));
 
-	for (r = remote_refs; r; r = r->next) {
-		if (!r->peer_ref) continue;
-		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
-		if (r->status == REF_STATUS_REJECT_STALE) continue;
-		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
-		if (r->status == REF_STATUS_UPTODATE) continue;
+	ret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);
+	if (ret < 0 && errno != EPIPE)
+		return ret; /* We do not mind if a hook does not read all refs. */
 
-		strbuf_reset(&buf);
-		strbuf_addf( &buf, "%s %s %s %s\n",
-			 r->peer_ref->name, oid_to_hex(&r->new_oid),
-			 r->name, oid_to_hex(&r->old_oid));
+	return 0;
+}
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			/* We do not mind if a hook does not read all refs. */
-			if (errno != EPIPE)
-				ret = -1;
-			break;
-		}
-	}
+static int run_pre_push_hook(struct transport *transport,
+			     struct ref *remote_refs)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct feed_pre_push_hook_data data;
+	int ret = 0;
+
+	strvec_push(&opt.args, transport->remote->name);
+	strvec_push(&opt.args, transport->url);
 
-	strbuf_release(&buf);
+	strbuf_init(&data.buf, 0);
+	data.refs = remote_refs;
 
-	x = close(proc.in);
-	if (!ret)
-		ret = x;
+	opt.feed_pipe = pre_push_hook_feed_stdin;
+	opt.feed_pipe_cb_data = &data;
 
-	sigchain_pop(SIGPIPE);
+	ret = run_hooks_opt(the_repository, "pre-push", &opt);
 
-	x = finish_command(&proc);
-	if (!ret)
-		ret = x;
+	strbuf_release(&data.buf);
 
 	return ret;
 }
-- 
2.51.0

[PATCH v4 06/11] reference-transaction: use hook API instead of run-command

[Adrian Ratiu]

Convert the reference-transaction hook to the new hook API,
so it doesn't need to set up a struct child_process, call
find_hook or toggle the pipe signals.

The stdin feed callback is processing one ref update per
call. I haven't noticed any performance degradation due
to this, however we can batch as many we want in each call,
to ensure a good pipe throughtput (i.e. the child does not
wait after stdin).

Helped-by: Emily Shaffer <nasamuffin@google.com>
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 refs.c | 100 ++++++++++++++++++++++++++++++---------------------------
 1 file changed, 52 insertions(+), 48 deletions(-)

diff --git a/refs.c b/refs.c
index 5583f6e09d..a708f0352e 100644
--- a/refs.c
+++ b/refs.c
@@ -2422,68 +2422,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-static int run_transaction_hook(struct ref_transaction *transaction,
-				const char *state)
+struct transaction_feed_cb_data {
+	size_t index;
+	struct strbuf buf;
+};
+
+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct strbuf buf = STRBUF_INIT;
-	const char *hook;
-	int ret = 0;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;
+	struct transaction_feed_cb_data *feed_cb_data = pp_task_cb;
+	struct strbuf *buf = &feed_cb_data->buf;
+	struct ref_update *update;
+	size_t i = feed_cb_data->index++;
+	int ret;
 
-	hook = find_hook(transaction->ref_store->repo, "reference-transaction");
-	if (!hook)
-		return ret;
+	if (i >= transaction->nr)
+		return 1; /* No more refs to process */
 
-	strvec_pushl(&proc.args, hook, state, NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "reference-transaction";
+	update = transaction->updates[i];
 
-	ret = start_command(&proc);
-	if (ret)
-		return ret;
+	if (update->flags & REF_LOG_ONLY)
+		return 0;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	strbuf_reset(buf);
 
-	for (size_t i = 0; i < transaction->nr; i++) {
-		struct ref_update *update = transaction->updates[i];
+	if (!(update->flags & REF_HAVE_OLD))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->old_target)
+		strbuf_addf(buf, "ref:%s ", update->old_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->old_oid));
 
-		if (update->flags & REF_LOG_ONLY)
-			continue;
+	if (!(update->flags & REF_HAVE_NEW))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->new_target)
+		strbuf_addf(buf, "ref:%s ", update->new_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->new_oid));
 
-		strbuf_reset(&buf);
+	strbuf_addf(buf, "%s\n", update->refname);
 
-		if (!(update->flags & REF_HAVE_OLD))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->old_target)
-			strbuf_addf(&buf, "ref:%s ", update->old_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->old_oid));
+	ret = write_in_full(hook_stdin_fd, buf->buf, buf->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
 
-		if (!(update->flags & REF_HAVE_NEW))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->new_target)
-			strbuf_addf(&buf, "ref:%s ", update->new_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->new_oid));
+	return 0; /* no more input to feed */
+}
+
+static int run_transaction_hook(struct ref_transaction *transaction,
+				const char *state)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct transaction_feed_cb_data feed_ctx = { 0 };
+	int ret = 0;
 
-		strbuf_addf(&buf, "%s\n", update->refname);
+	strvec_push(&opt.args, state);
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			if (errno != EPIPE) {
-				/* Don't leak errno outside this API */
-				errno = 0;
-				ret = -1;
-			}
-			break;
-		}
-	}
+	opt.feed_pipe = transaction_hook_feed_stdin;
+	opt.feed_pipe_ctx = transaction;
+	opt.feed_pipe_cb_data = &feed_ctx;
 
-	close(proc.in);
-	sigchain_pop(SIGPIPE);
-	strbuf_release(&buf);
+	strbuf_init(&feed_ctx.buf, 0);
+
+	ret = run_hooks_opt(transaction->ref_store->repo, "reference-transaction", &opt);
 
-	ret |= finish_command(&proc);
+	strbuf_release(&feed_ctx.buf);
 	return ret;
 }
 
-- 
2.51.0

[PATCH v4 07/11] hook: allow overriding the ungroup option

[Adrian Ratiu]

When calling run_process_parallel() in run_hooks_opt(), the
ungroup option is currently hardcoded to .ungroup = 1.

This causes problems when ungrouping should be disabled, for
example when sideband-reading collated output from child hooks,
because sideband-reading and ungrouping are mutually exclusive.

Thus a new hook.h option is added to allow overriding.

The existing ungroup=1 behavior is preserved in the run_hooks()
API and the "hook run" command. We could modify these to take
an option if necessary, so I added two code comments there.

Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/hook.c | 6 ++++++
 commit.c       | 3 +++
 hook.c         | 5 ++++-
 hook.h         | 5 +++++
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/builtin/hook.c b/builtin/hook.c
index 7afec380d2..73e7b8c2e8 100644
--- a/builtin/hook.c
+++ b/builtin/hook.c
@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,
 	if (!argc)
 		goto usage;
 
+	/*
+	 * All current "hook run" use-cases require ungrouped child output.
+	 * If this changes, a hook run argument can be added to toggle it.
+	 */
+	opt.ungroup = 1;
+
 	/*
 	 * Having a -- for "run" when providing <hook-args> is
 	 * mandatory.
diff --git a/commit.c b/commit.c
index 16d91b2bfc..7da33dde86 100644
--- a/commit.c
+++ b/commit.c
@@ -1965,6 +1965,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,
 		strvec_push(&opt.args, arg);
 	va_end(args);
 
+	/* All commit hook use-cases require ungrouping child output. */
+	opt.ungroup = 1;
+
 	opt.invoked_hook = invoked_hook;
 	return run_hooks_opt(the_repository, name, &opt);
 }
diff --git a/hook.c b/hook.c
index 5ddd7678d1..00a1e2ad22 100644
--- a/hook.c
+++ b/hook.c
@@ -153,7 +153,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.tr2_label = hook_name,
 
 		.processes = 1,
-		.ungroup = 1,
+		.ungroup = options->ungroup,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
@@ -198,6 +198,9 @@ int run_hooks(struct repository *r, const char *hook_name)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
+	/* All use-cases of this API require ungrouping. */
+	opt.ungroup = 1;
+
 	return run_hooks_opt(r, hook_name, &opt);
 }
 
diff --git a/hook.h b/hook.h
index 51cab785ea..01f91feab8 100644
--- a/hook.h
+++ b/hook.h
@@ -34,6 +34,11 @@ struct run_hooks_opt
 	 */
 	int *invoked_hook;
 
+	/**
+	 * Allow hooks to set run_processes_parallel() 'ungroup' behavior.
+	 */
+	unsigned int ungroup:1;
+
 	/**
 	 * Path to file which should be piped to stdin for each hook.
 	 */
-- 
2.51.0

[PATCH v4 08/11] run-command: allow capturing of collated output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some callers, for example server-side hooks which wish to relay hook
output to clients across a transport, want to capture what would
normally print to stderr and do something else with it. Allow that via a
callback.

By calling the callback regardless of whether there's output available,
we allow clients to send e.g. a keepalive if necessary.

Because we expose a strbuf, not a fd or FILE*, there's no need to create
a temporary pipe or similar - we can just skip the print to stderr and
instead hand it to the caller.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 30 ++++++++++++++++++++++--------
 run-command.h               | 17 +++++++++++++++++
 t/helper/test-run-command.c | 15 +++++++++++++++
 t/t0061-run-command.sh      |  7 +++++++
 4 files changed, 61 insertions(+), 8 deletions(-)

diff --git a/run-command.c b/run-command.c
index a608d37fb2..6b1e4a3453 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1595,7 +1595,10 @@ static void pp_cleanup(struct parallel_processes *pp,
 	 * When get_next_task added messages to the buffer in its last
 	 * iteration, the buffered output is non empty.
 	 */
-	strbuf_write(&pp->buffered_output, stderr);
+	if (opts->consume_output)
+		opts->consume_output(&pp->buffered_output, opts->data);
+	else
+		strbuf_write(&pp->buffered_output, stderr);
 	strbuf_release(&pp->buffered_output);
 
 	sigchain_pop_common();
@@ -1734,13 +1737,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 	}
 }
 
-static void pp_output(const struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp,
+		      const struct run_process_parallel_opts *opts)
 {
 	size_t i = pp->output_owner;
 
 	if (child_is_working(&pp->children[i]) &&
 	    pp->children[i].err.len) {
-		strbuf_write(&pp->children[i].err, stderr);
+		if (opts->consume_output)
+			opts->consume_output(&pp->children[i].err, opts->data);
+		else
+			strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
 	}
 }
@@ -1788,11 +1795,15 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		} else {
 			const size_t n = opts->processes;
 
-			strbuf_write(&pp->children[i].err, stderr);
+			/* Output errors, then all other finished child processes */
+			if (opts->consume_output) {
+				opts->consume_output(&pp->children[i].err, opts->data);
+				opts->consume_output(&pp->buffered_output, opts->data);
+			} else {
+				strbuf_write(&pp->children[i].err, stderr);
+				strbuf_write(&pp->buffered_output, stderr);
+			}
 			strbuf_reset(&pp->children[i].err);
-
-			/* Output all other finished child processes */
-			strbuf_write(&pp->buffered_output, stderr);
 			strbuf_reset(&pp->buffered_output);
 
 			/*
@@ -1829,7 +1840,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,
 				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
 	} else {
 		pp_buffer_stderr(pp, opts, output_timeout);
-		pp_output(pp);
+		pp_output(pp, opts);
 	}
 }
 
@@ -1852,6 +1863,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	if (opts->ungroup && opts->consume_output)
+		BUG("ungroup and reading output are mutualy exclusive");
+
 	/*
 	 * Child tasks might receive input via stdin, terminating early (or not), so
 	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
diff --git a/run-command.h b/run-command.h
index e1ca965b5b..7093252863 100644
--- a/run-command.h
+++ b/run-command.h
@@ -435,6 +435,17 @@ typedef int (*feed_pipe_fn)(int child_in,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * If this callback is provided, output is collated into a new pipe instead
+ * of the process stderr. Then `consume_output_fn` will be called repeatedly
+ * with output contained in the `output` arg. It will also be called with an
+ * empty `output` to allow for keepalives or similar operations if necessary.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel.
+ * No task cookie is provided because the callback receives collated output.
+ */
+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -494,6 +505,12 @@ struct run_process_parallel_opts
 	 */
 	feed_pipe_fn feed_pipe;
 
+	/*
+	 * consume_output: see consume_output_fn() above. This can be NULL
+	 * to omit any special handling.
+	 */
+	consume_output_fn consume_output;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 4a56456894..49eace8dce 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,
 	return 0;
 }
 
+static void test_divert_output(struct strbuf *output, void *cb UNUSED)
+{
+	FILE *output_file;
+
+	output_file = fopen("./output_file", "a");
+
+	strbuf_write(output, output_file);
+	fclose(output_file);
+}
+
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)
 		.get_next_task = next_test,
 		.start_failure = test_failed,
 		.feed_pipe = test_stdin_pipe_feed,
+		.consume_output = test_divert_output,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished_quiet;
 		opts.feed_pipe = test_stdin_pipe_feed;
+	} else if (!strcmp(argv[1], "run-command-divert-output")) {
+		opts.get_next_task = parallel_next;
+		opts.consume_output = test_divert_output;
+		opts.task_finished = task_finished_quiet;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 2f77fde0d9..74529e219e 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command can divert output' '
+	test_when_finished rm output_file &&
+	test-tool run-command run-command-divert-output 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
+	test_must_be_empty actual &&
+	test_cmp expect output_file
+'
+
 test_expect_success 'run_command listens to stdin' '
 	cat >expect <<-\EOF &&
 	preloaded output of a child
-- 
2.51.0

[PATCH v4 09/11] hooks: allow callers to capture output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some server-side hooks will require capturing output to send over
sideband instead of printing directly to stderr. Expose that capability.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c | 1 +
 hook.h | 8 ++++++++
 2 files changed, 9 insertions(+)

diff --git a/hook.c b/hook.c
index 00a1e2ad22..35211e5ed7 100644
--- a/hook.c
+++ b/hook.c
@@ -158,6 +158,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
 		.feed_pipe = options->feed_pipe,
+		.consume_output = options->consume_output,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
diff --git a/hook.h b/hook.h
index 01f91feab8..6b0cb69a7c 100644
--- a/hook.h
+++ b/hook.h
@@ -80,6 +80,14 @@ struct run_hooks_opt
 	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
 	 */
 	void *feed_pipe_cb_data;
+
+	/*
+	 * Populate this to capture output and prevent it from being printed to
+	 * stderr. This will be passed directly through to
+	 * run_command:run_parallel_processes(). See t/helper/test-run-command.c
+	 * for an example.
+	 */
+	consume_output_fn consume_output;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.0

[PATCH v4 10/11] receive-pack: convert update hooks to new API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Use the new hook sideband API introduced in the previous commit.

The hook API avoids creating a custom struct child_process and other
internal hook plumbing (e.g. calling find_hook()) and prepares for
the specification of hooks via configs or running parallel hooks.

Execution is still sequential through the current hook.[ch] via the
run_process_parallel_opts.processes=1 arg.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 60 +++++++++++++++---------------------------
 1 file changed, 21 insertions(+), 39 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index e8ee0e7321..d95df748cd 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,
 	return status;
 }
 
-static int run_update_hook(struct command *cmd)
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	int code;
-	const char *hook_path = find_hook(the_repository, "update");
-
-	if (!hook_path)
-		return 0;
+	if (output && output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
+}
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, cmd->ref_name);
-	strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
-	strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
+static int run_update_hook(struct command *cmd)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "update";
+	strvec_pushl(&opt.args,
+		     cmd->ref_name,
+		     oid_to_hex(&cmd->old_oid),
+		     oid_to_hex(&cmd->new_oid),
+		     NULL);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
 	if (use_sideband)
-		copy_to_sideband(proc.err, -1, NULL);
-	return finish_command(&proc);
+		opt.consume_output = hook_output_to_sideband;
+
+	return run_hooks_opt(the_repository, "update", &opt);
 }
 
 static struct command *find_command_by_refname(struct command *list,
@@ -1639,33 +1634,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 static void run_update_post_hook(struct command *commands)
 {
 	struct command *cmd;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	const char *hook;
-
-	hook = find_hook(the_repository, "post-update");
-	if (!hook)
-		return;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		if (cmd->error_string || cmd->did_not_exist)
 			continue;
-		if (!proc.args.nr)
-			strvec_push(&proc.args, hook);
-		strvec_push(&proc.args, cmd->ref_name);
+		strvec_push(&opt.args, cmd->ref_name);
 	}
-	if (!proc.args.nr)
+	if (!opt.args.nr)
 		return;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "post-update";
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
 
-	if (!start_command(&proc)) {
-		if (use_sideband)
-			copy_to_sideband(proc.err, -1, NULL);
-		finish_command(&proc);
-	}
+	run_hooks_opt(the_repository, "post-update", &opt);
 }
 
 static void check_aliased_update_internal(struct command *cmd,
-- 
2.51.0

[PATCH v4 11/11] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This converts the last remaining hooks to the new hook API, for
the same benefits as the previous conversions (no need to toggle
signals, manage custom struct child_process, call find_hook(),
prepares for specifyinig hooks via configs, etc.).

I noticed a performance degradation when processing large amounts
of hook input with just 1 line per callback, due to run-command's
poll loop, therefore I batched 500 lines per callback, to ensure
similar pipe throughput as before and to avoid hook child waiting
on stdin.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 220 ++++++++++++++++++-----------------------
 1 file changed, 98 insertions(+), 122 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index d95df748cd..a8dfde52bf 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -748,7 +748,7 @@ static int check_cert_push_options(const struct string_list *push_options)
 	return retval;
 }
 
-static void prepare_push_cert_sha1(struct child_process *proc)
+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)
 {
 	static int already_done;
 
@@ -774,23 +774,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)
 		nonce_status = check_nonce(sigcheck.payload);
 	}
 	if (!is_null_oid(&push_cert_oid)) {
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT=%s",
 			     oid_to_hex(&push_cert_oid));
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_SIGNER=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_SIGNER=%s",
 			     sigcheck.signer ? sigcheck.signer : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_KEY=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_KEY=%s",
 			     sigcheck.key ? sigcheck.key : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_STATUS=%c",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_STATUS=%c",
 			     sigcheck.result);
 		if (push_cert_nonce) {
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE=%s",
 				     push_cert_nonce);
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE_STATUS=%s",
 				     nonce_status);
 			if (nonce_status == NONCE_SLOP)
-				strvec_pushf(&proc->env,
+				strvec_pushf(&opt->env,
 					     "GIT_PUSH_CERT_NONCE_SLOP=%ld",
 					     nonce_stamp_slop);
 		}
@@ -802,119 +802,70 @@ struct receive_hook_feed_state {
 	struct ref_push_report *report;
 	int skip_broken;
 	struct strbuf buf;
-	const struct string_list *push_options;
 };
 
-typedef int (*feed_fn)(void *, const char **, size_t *);
-static int run_and_feed_hook(const char *hook_name, feed_fn feed,
-			     struct receive_hook_feed_state *feed_state)
+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct async muxer;
-	int code;
-	const char *hook_path = find_hook(the_repository, hook_name);
+	struct receive_hook_feed_state *state = pp_task_cb;
+	struct command *cmd = state->cmd;
+	unsigned int lines_batch_size = 500;
 
-	if (!hook_path)
-		return 0;
+	strbuf_reset(&state->buf);
 
-	strvec_push(&proc.args, hook_path);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = hook_name;
-
-	if (feed_state->push_options) {
-		size_t i;
-		for (i = 0; i < feed_state->push_options->nr; i++)
-			strvec_pushf(&proc.env,
-				     "GIT_PUSH_OPTION_%"PRIuMAX"=%s",
-				     (uintmax_t)i,
-				     feed_state->push_options->items[i].string);
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
-			     (uintmax_t)feed_state->push_options->nr);
-	} else
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
+	/* batch lines to avoid going through run-command's poll loop for each line */
+	for (unsigned int i = 0; i < lines_batch_size; i++) {
+		while (cmd &&
+		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
+			cmd = cmd->next;
 
-	if (tmp_objdir)
-		strvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));
+		if (!cmd)
+			break;  /* no more commands left */
 
-	if (use_sideband) {
-		memset(&muxer, 0, sizeof(muxer));
-		muxer.proc = copy_to_sideband;
-		muxer.in = -1;
-		code = start_async(&muxer);
-		if (code)
-			return code;
-		proc.err = muxer.in;
-	}
+		if (!state->report)
+			state->report = cmd->report;
 
-	prepare_push_cert_sha1(&proc);
+		if (state->report) {
+			struct object_id *old_oid;
+			struct object_id *new_oid;
+			const char *ref_name;
 
-	code = start_command(&proc);
-	if (code) {
-		if (use_sideband)
-			finish_async(&muxer);
-		return code;
-	}
+			old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
+			new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
+			ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(old_oid), oid_to_hex(new_oid),
+				    ref_name);
 
-	while (1) {
-		const char *buf;
-		size_t n;
-		if (feed(feed_state, &buf, &n))
-			break;
-		if (write_in_full(proc.in, buf, n) < 0)
-			break;
+			state->report = state->report->next;
+			if (!state->report)
+				cmd = cmd->next;
+		} else {
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
+				    cmd->ref_name);
+			cmd = cmd->next;
+		}
 	}
-	close(proc.in);
-	if (use_sideband)
-		finish_async(&muxer);
 
-	sigchain_pop(SIGPIPE);
+	state->cmd = cmd;
 
-	return finish_command(&proc);
+	if (state->buf.len > 0) {
+		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
+		if (ret < 0) {
+			if (errno == EPIPE)
+				return 1; /* child closed pipe */
+			return ret;
+		}
+	}
+
+	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
 }
 
-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
 {
-	struct receive_hook_feed_state *state = state_;
-	struct command *cmd = state->cmd;
-
-	while (cmd &&
-	       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
-		cmd = cmd->next;
-	if (!cmd)
-		return -1; /* EOF */
-	if (!bufp)
-		return 0; /* OK, can feed something. */
-	strbuf_reset(&state->buf);
-	if (!state->report)
-		state->report = cmd->report;
-	if (state->report) {
-		struct object_id *old_oid;
-		struct object_id *new_oid;
-		const char *ref_name;
-
-		old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
-		new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
-		ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(old_oid), oid_to_hex(new_oid),
-			    ref_name);
-		state->report = state->report->next;
-		if (!state->report)
-			state->cmd = cmd->next;
-	} else {
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
-			    cmd->ref_name);
-		state->cmd = cmd->next;
-	}
-	if (bufp) {
-		*bufp = state->buf.buf;
-		*sizep = state->buf.len;
-	}
-	return 0;
+	if (output && output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
 }
 
 static int run_receive_hook(struct command *commands,
@@ -922,26 +873,51 @@ static int run_receive_hook(struct command *commands,
 			    int skip_broken,
 			    const struct string_list *push_options)
 {
-	struct receive_hook_feed_state state;
-	int status;
-
-	strbuf_init(&state.buf, 0);
-	state.cmd = commands;
-	state.skip_broken = skip_broken;
-	state.report = NULL;
-	if (feed_receive_hook(&state, NULL, NULL))
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct command *iter = commands;
+	struct receive_hook_feed_state *feed_state;
+	int ret;
+
+	/* if there are no valid commands, don't invoke the hook at all. */
+	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
+		iter = iter->next;
+	if (!iter)
 		return 0;
-	state.cmd = commands;
-	state.push_options = push_options;
-	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
-	strbuf_release(&state.buf);
-	return status;
-}
 
-static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
-{
-	if (output && output->len)
-		send_sideband(1, 2, output->buf, output->len, use_sideband);
+	if (push_options) {
+		int i;
+		for (i = 0; i < push_options->nr; i++)
+			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
+				     push_options->items[i].string);
+		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
+					     (uintmax_t)push_options->nr);
+	} else
+		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+
+	if (tmp_objdir)
+		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+
+	prepare_push_cert_sha1(&opt);
+
+	/* set up sideband printer */
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
+
+	/* set up stdin callback */
+	feed_state = xmalloc(sizeof(struct receive_hook_feed_state));
+	feed_state->cmd = commands;
+	feed_state->skip_broken = skip_broken;
+	feed_state->report = NULL;
+	strbuf_init(&feed_state->buf, 0);
+	opt.feed_pipe_cb_data = feed_state;
+	opt.feed_pipe = feed_receive_hook_cb;
+
+	ret = run_hooks_opt(the_repository, hook_name, &opt);
+
+	strbuf_release(&feed_state->buf);
+	FREE_AND_NULL(opt.feed_pipe_cb_data);
+
+	return ret;
 }
 
 static int run_update_hook(struct command *cmd)
-- 
2.51.0

Re: [PATCH v4 03/11] hook: provide stdin via callback

[Patrick Steinhardt]

On Thu, Dec 04, 2025 at 04:15:27PM +0200, Adrian Ratiu wrote:
> diff --git a/hook.h b/hook.h
> index 11863fa734..51cab785ea 100644
> --- a/hook.h
> +++ b/hook.h
> @@ -37,6 +38,43 @@ struct run_hooks_opt
>  	 * Path to file which should be piped to stdin for each hook.
>  	 */
>  	const char *path_to_stdin;
> +
> +	/**
> +	 * Callback used to incrementally feed a child hook stdin pipe.
> +	 *
> +	 * Useful especially if a hook consumes large quantities of data
> +	 * (e.g. a list of all refs in a client push), so feeding it via
> +	 * in-memory strings or slurping to/from files is inefficient.
> +	 * While the callback allows piecemeal writing, it can also be
> +	 * used for smaller inputs, where it gets called only once.
> +	 *
> +	 * Add hook callback initalization context to `feed_pipe_ctx`.
> +	 * Add Hook callback internal state to `feed_pipe_cb_data`.

Tiniest nit just to prove that I'm somewhat paying attention:
s/Hook/hook/. This definitely doesn't warrant a reroll.

Patrick

Re: [PATCH v4 05/11] transport: convert pre-push to hook API

[Patrick Steinhardt]

On Thu, Dec 04, 2025 at 04:15:29PM +0200, Adrian Ratiu wrote:
> diff --git a/transport.c b/transport.c
> index c7f06a7382..047f2cefba 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
[snip]
> -	if (start_command(&proc)) {
> -		finish_command(&proc);
> -		return -1;
> +	switch (r->status) {
> +	case REF_STATUS_REJECT_ALREADY_EXISTS:
> +	case REF_STATUS_REJECT_FETCH_FIRST:
> +	case REF_STATUS_REJECT_NEEDS_FORCE:
> +	case REF_STATUS_REJECT_NODELETE:
> +	case REF_STATUS_REJECT_NONFASTFORWARD:
> +	case REF_STATUS_REJECT_REMOTE_UPDATED:
> +	case REF_STATUS_REJECT_SHALLOW:
> +	case REF_STATUS_REJECT_STALE:
> +	case REF_STATUS_UPTODATE:
> +		return 0; /* skip refs which won't be pushed */
> +	default:
> +		break;
>  	}
>  
> -	sigchain_push(SIGPIPE, SIG_IGN);
> +	if (!r->peer_ref)
> +		return 0;
>  
> -	strbuf_init(&buf, 256);
> +	strbuf_reset(&data->buf);
> +	strbuf_addf(&data->buf, "%s %s %s %s\n",
> +		    r->peer_ref->name, oid_to_hex(&r->new_oid),
> +		    r->name, oid_to_hex(&r->old_oid));
>  
> -	for (r = remote_refs; r; r = r->next) {
> -		if (!r->peer_ref) continue;
> -		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
> -		if (r->status == REF_STATUS_REJECT_STALE) continue;
> -		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
> -		if (r->status == REF_STATUS_UPTODATE) continue;

The new code looks a lot nicer in my opinion. But one thing I wonder
about is how these statements translate to the above switch statement.
We ignore a lot more `status` values now compared to previously, and
the reason for this is never explained.

Am I missing anything obvious?

Patrick

Re: [PATCH v4 10/11] receive-pack: convert update hooks to new API

[Patrick Steinhardt]

On Thu, Dec 04, 2025 at 04:15:34PM +0200, Adrian Ratiu wrote:
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index e8ee0e7321..d95df748cd 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,
>  	return status;
>  }
>  
> -static int run_update_hook(struct command *cmd)
> +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
>  {
> -	struct child_process proc = CHILD_PROCESS_INIT;
> -	int code;
> -	const char *hook_path = find_hook(the_repository, "update");
> -
> -	if (!hook_path)
> -		return 0;
> +	if (output && output->len)
> +		send_sideband(1, 2, output->buf, output->len, use_sideband);

Nit, not worth a reroll: the buffer shouldn't ever be `NULL`, should it?
Checking for `output->len` does make sense though, as we may receive
empty buffers for keepalives.

Patrick

Re: [PATCH v4 05/11] transport: convert pre-push to hook API

[Adrian Ratiu]

On Tue, 16 Dec 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Thu, Dec 04, 2025 at 04:15:29PM +0200, Adrian Ratiu wrote:
>> diff --git a/transport.c b/transport.c
>> index c7f06a7382..047f2cefba 100644
>> --- a/transport.c
>> +++ b/transport.c
>> @@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
> [snip]
>> -	if (start_command(&proc)) {
>> -		finish_command(&proc);
>> -		return -1;
>> +	switch (r->status) {
>> +	case REF_STATUS_REJECT_ALREADY_EXISTS:
>> +	case REF_STATUS_REJECT_FETCH_FIRST:
>> +	case REF_STATUS_REJECT_NEEDS_FORCE:
>> +	case REF_STATUS_REJECT_NODELETE:
>> +	case REF_STATUS_REJECT_NONFASTFORWARD:
>> +	case REF_STATUS_REJECT_REMOTE_UPDATED:
>> +	case REF_STATUS_REJECT_SHALLOW:
>> +	case REF_STATUS_REJECT_STALE:
>> +	case REF_STATUS_UPTODATE:
>> +		return 0; /* skip refs which won't be pushed */
>> +	default:
>> +		break;
>>  	}
>>  
>> -	sigchain_push(SIGPIPE, SIG_IGN);
>> +	if (!r->peer_ref)
>> +		return 0;
>>  
>> -	strbuf_init(&buf, 256);
>> +	strbuf_reset(&data->buf);
>> +	strbuf_addf(&data->buf, "%s %s %s %s\n",
>> +		    r->peer_ref->name, oid_to_hex(&r->new_oid),
>> +		    r->name, oid_to_hex(&r->old_oid));
>>  
>> -	for (r = remote_refs; r; r = r->next) {
>> -		if (!r->peer_ref) continue;
>> -		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
>> -		if (r->status == REF_STATUS_REJECT_STALE) continue;
>> -		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
>> -		if (r->status == REF_STATUS_UPTODATE) continue;
>
> The new code looks a lot nicer in my opinion. But one thing I wonder
> about is how these statements translate to the above switch statement.
> We ignore a lot more `status` values now compared to previously, and
> the reason for this is never explained.
>
> Am I missing anything obvious?

I thought it was obvious when writing the code :) it might not be.

The reason is the list was not exhaustive: for refs which we know
beforehand will not be pushed (status rejected), there is no need to
feed the pre-push hook stdin. It's saving a few cpu cycles in some
rejection corner cases, which were missed previously.

I could:
1. Split the extra *_REJECT_* case aditions into a separate commit,
   highlighting and explaining this better.
2. Drop the new cases since they are just a minor improvement in this
   series, not very important for the series overall.

Any preference?

Re: [PATCH v4 10/11] receive-pack: convert update hooks to new API

[Adrian Ratiu]

On Tue, 16 Dec 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Thu, Dec 04, 2025 at 04:15:34PM +0200, Adrian Ratiu wrote:
>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
>> index e8ee0e7321..d95df748cd 100644
>> --- a/builtin/receive-pack.c
>> +++ b/builtin/receive-pack.c
>> @@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,
>>  	return status;
>>  }
>>  
>> -static int run_update_hook(struct command *cmd)
>> +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
>>  {
>> -	struct child_process proc = CHILD_PROCESS_INIT;
>> -	int code;
>> -	const char *hook_path = find_hook(the_repository, "update");
>> -
>> -	if (!hook_path)
>> -		return 0;
>> +	if (output && output->len)
>> +		send_sideband(1, 2, output->buf, output->len, use_sideband);
>
> Nit, not worth a reroll: the buffer shouldn't ever be `NULL`, should it?
> Checking for `output->len` does make sense though, as we may receive
> empty buffers for keepalives.

Good point. NULL output should still be checked to avoid unexpected
segfaults but it should rather trigger a BUG() than a no-op.

Still pretty much harmless as is, will fix if I do a re-roll.

Re: [PATCH v4 05/11] transport: convert pre-push to hook API

[Patrick Steinhardt]

On Tue, Dec 16, 2025 at 11:09:52AM +0200, Adrian Ratiu wrote:
> On Tue, 16 Dec 2025, Patrick Steinhardt <ps@pks.im> wrote:
> > On Thu, Dec 04, 2025 at 04:15:29PM +0200, Adrian Ratiu wrote:
> >> diff --git a/transport.c b/transport.c
> >> index c7f06a7382..047f2cefba 100644
> >> --- a/transport.c
> >> +++ b/transport.c
> >> @@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
> > [snip]
> >> -	if (start_command(&proc)) {
> >> -		finish_command(&proc);
> >> -		return -1;
> >> +	switch (r->status) {
> >> +	case REF_STATUS_REJECT_ALREADY_EXISTS:
> >> +	case REF_STATUS_REJECT_FETCH_FIRST:
> >> +	case REF_STATUS_REJECT_NEEDS_FORCE:
> >> +	case REF_STATUS_REJECT_NODELETE:
> >> +	case REF_STATUS_REJECT_NONFASTFORWARD:
> >> +	case REF_STATUS_REJECT_REMOTE_UPDATED:
> >> +	case REF_STATUS_REJECT_SHALLOW:
> >> +	case REF_STATUS_REJECT_STALE:
> >> +	case REF_STATUS_UPTODATE:
> >> +		return 0; /* skip refs which won't be pushed */
> >> +	default:
> >> +		break;
> >>  	}
> >>  
> >> -	sigchain_push(SIGPIPE, SIG_IGN);
> >> +	if (!r->peer_ref)
> >> +		return 0;
> >>  
> >> -	strbuf_init(&buf, 256);
> >> +	strbuf_reset(&data->buf);
> >> +	strbuf_addf(&data->buf, "%s %s %s %s\n",
> >> +		    r->peer_ref->name, oid_to_hex(&r->new_oid),
> >> +		    r->name, oid_to_hex(&r->old_oid));
> >>  
> >> -	for (r = remote_refs; r; r = r->next) {
> >> -		if (!r->peer_ref) continue;
> >> -		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
> >> -		if (r->status == REF_STATUS_REJECT_STALE) continue;
> >> -		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
> >> -		if (r->status == REF_STATUS_UPTODATE) continue;
> >
> > The new code looks a lot nicer in my opinion. But one thing I wonder
> > about is how these statements translate to the above switch statement.
> > We ignore a lot more `status` values now compared to previously, and
> > the reason for this is never explained.
> >
> > Am I missing anything obvious?
> 
> I thought it was obvious when writing the code :) it might not be.
> 
> The reason is the list was not exhaustive: for refs which we know
> beforehand will not be pushed (status rejected), there is no need to
> feed the pre-push hook stdin. It's saving a few cpu cycles in some
> rejection corner cases, which were missed previously.
> 
> I could:
> 1. Split the extra *_REJECT_* case aditions into a separate commit,
>    highlighting and explaining this better.
> 2. Drop the new cases since they are just a minor improvement in this
>    series, not very important for the series overall.
> 
> Any preference?

I'd personally learn towards (2) unless it is fixing an actual bug that
can be demonstrated. In that case it might make sense to do (1) and
explain why this wasn't an issue until now.

Thanks!

Patrick

Re: [PATCH v4 05/11] transport: convert pre-push to hook API

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> On Tue, Dec 16, 2025 at 11:09:52AM +0200, Adrian Ratiu wrote:
> ...
>> The reason is the list was not exhaustive: for refs which we know
>> beforehand will not be pushed (status rejected), there is no need to
>> feed the pre-push hook stdin. It's saving a few cpu cycles in some
>> rejection corner cases, which were missed previously.
>> 
>> I could:
>> 1. Split the extra *_REJECT_* case aditions into a separate commit,
>>    highlighting and explaining this better.
>> 2. Drop the new cases since they are just a minor improvement in this
>>    series, not very important for the series overall.
>> 
>> Any preference?
>
> I'd personally learn towards (2) unless it is fixing an actual bug that
> can be demonstrated. In that case it might make sense to do (1) and
> explain why this wasn't an issue until now.

I tend to agree.  If my hook did something depending on the branches
the pusher is _trying_ to update, not doing (2) would even be a
regression.

If "git push there next seen" attempts to push these two, we know
seen does not fast-forward and locally decide to refrain from
pushing it, it would silently turn into "git push there next", but
the hook may want to behave differently between these two sets of
command line arguments.

Thanks.

[PATCH v5 00/11] Convert remaining hooks to hook.h

[Adrian Ratiu]

Hello everyone,

This series finishes the hook.[ch] conversion for the remaining hooks in
preparation for adding config-based hooks and enabling parallel hook
execution where possible (that will be a separate series from this one).

v5 is just a minor cleanup refresh: details + range-diff below.

It is based on the latest master branch. There are no conflicts with
next and seen branches, the code is available on GitHub [1] and a
successful CI run [2] is also provided.

1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v5
2: https://github.com/10ne1/git/actions/runs/20274558658

Changes in v5:
* Dropped some REF_STATUS_REJECT_* additions which are mostly unrelated (Patrick)
* Made a null output buffer in hook_output_to_sideband() trigger a BUG (Patrick)
* Cleaned up the diff in patch 11 to not unnecessarily move hook_output_to_sideband()
  around, though this looks a bit ugly in the range-diff for patches 10 & 11 (Adrian)
* Two small nitpicks / typos (Patrick)

Range-diff between v4 -> v5:
 1:  b252d447f5 =  1:  8c16f1bcbf run-command: add first helper for pp child states
 2:  f6fd3b9b0f =  2:  5e6e05ba92 run-command: add stdin callback for parallelization
 3:  59c07b618e !  3:  3669acfe6a hook: provide stdin via callback
    @@ hook.h: struct run_hooks_opt
     +	 * used for smaller inputs, where it gets called only once.
     +	 *
     +	 * Add hook callback initalization context to `feed_pipe_ctx`.
    -+	 * Add Hook callback internal state to `feed_pipe_cb_data`.
    ++	 * Add hook callback internal state to `feed_pipe_cb_data`.
     +	 *
     +	 */
     +	feed_pipe_fn feed_pipe;
 4:  8f591319a4 =  4:  bf9d8680e4 hook: convert 'post-rewrite' hook in sequencer.c to hook API
 5:  2427717a26 !  5:  bdcc1cff34 transport: convert pre-push to hook API
    @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_
     -		finish_command(&proc);
     -		return -1;
     +	switch (r->status) {
    -+	case REF_STATUS_REJECT_ALREADY_EXISTS:
    -+	case REF_STATUS_REJECT_FETCH_FIRST:
    -+	case REF_STATUS_REJECT_NEEDS_FORCE:
    -+	case REF_STATUS_REJECT_NODELETE:
     +	case REF_STATUS_REJECT_NONFASTFORWARD:
     +	case REF_STATUS_REJECT_REMOTE_UPDATED:
    -+	case REF_STATUS_REJECT_SHALLOW:
     +	case REF_STATUS_REJECT_STALE:
     +	case REF_STATUS_UPTODATE:
     +		return 0; /* skip refs which won't be pushed */
 6:  22467bb074 =  6:  9c1d5e8726 reference-transaction: use hook API instead of run-command
 7:  9b0b13379c =  7:  0b986bf0fb hook: allow overriding the ungroup option
 8:  cae3c984e2 =  8:  5f07d07acc run-command: allow capturing of collated output
 9:  0383be2ebd =  9:  c4ff1e2270 hooks: allow callers to capture output
10:  d8e234c453 ! 10:  15c831ca15 receive-pack: convert update hooks to new API
    @@ Commit message
         Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
     
      ## builtin/receive-pack.c ##
    -@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
    - 	return status;
    +@@ builtin/receive-pack.c: static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
    + 	return 0;
      }
      
    --static int run_update_hook(struct command *cmd)
     +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
    ++{
    ++	if (!output)
    ++		BUG("output must be non-NULL");
    ++
    ++	/* buffer might be empty for keepalives */
    ++	if (output->len)
    ++		send_sideband(1, 2, output->buf, output->len, use_sideband);
    ++}
    ++
    + static int run_receive_hook(struct command *commands,
    + 			    const char *hook_name,
    + 			    int skip_broken,
    +@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
    + 
    + static int run_update_hook(struct command *cmd)
      {
     -	struct child_process proc = CHILD_PROCESS_INIT;
     -	int code;
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
     -
     -	if (!hook_path)
     -		return 0;
    -+	if (output && output->len)
    -+		send_sideband(1, 2, output->buf, output->len, use_sideband);
    -+}
    - 
    +-
     -	strvec_push(&proc.args, hook_path);
     -	strvec_push(&proc.args, cmd->ref_name);
     -	strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
     -	strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
    -+static int run_update_hook(struct command *cmd)
    -+{
     +	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
      
     -	proc.no_stdin = 1;
11:  6bad9aed4d ! 11:  4bd2c2974a receive-pack: convert receive hooks to hook API
    @@ builtin/receive-pack.c: struct receive_hook_feed_state {
     -		finish_async(&muxer);
      
     -	sigchain_pop(SIGPIPE);
    -+	state->cmd = cmd;
    - 
    +-
     -	return finish_command(&proc);
    -+	if (state->buf.len > 0) {
    -+		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
    -+		if (ret < 0) {
    -+			if (errno == EPIPE)
    -+				return 1; /* child closed pipe */
    -+			return ret;
    -+		}
    -+	}
    -+
    -+	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
    - }
    +-}
    ++	state->cmd = cmd;
      
     -static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
    -+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
    - {
    +-{
     -	struct receive_hook_feed_state *state = state_;
     -	struct command *cmd = state->cmd;
     -
    @@ builtin/receive-pack.c: struct receive_hook_feed_state {
     -	if (bufp) {
     -		*bufp = state->buf.buf;
     -		*sizep = state->buf.len;
    --	}
    ++	if (state->buf.len > 0) {
    ++		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
    ++		if (ret < 0) {
    ++			if (errno == EPIPE)
    ++				return 1; /* child closed pipe */
    ++			return ret;
    ++		}
    + 	}
     -	return 0;
    -+	if (output && output->len)
    -+		send_sideband(1, 2, output->buf, output->len, use_sideband);
    ++
    ++	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
      }
      
    - static int run_receive_hook(struct command *commands,
    + static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
     @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
      			    int skip_broken,
      			    const struct string_list *push_options)
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
     -	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
     -	strbuf_release(&state.buf);
     -	return status;
    --}
    - 
    --static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
    --{
    --	if (output && output->len)
    --		send_sideband(1, 2, output->buf, output->len, use_sideband);
    ++
     +	if (push_options) {
     +		int i;
     +		for (i = 0; i < push_options->nr; i++)

Adrian Ratiu (3):
  run-command: add first helper for pp child states
  reference-transaction: use hook API instead of run-command
  hook: allow overriding the ungroup option

Emily Shaffer (8):
  run-command: add stdin callback for parallelization
  hook: provide stdin via callback
  hook: convert 'post-rewrite' hook in sequencer.c to hook API
  transport: convert pre-push to hook API
  run-command: allow capturing of collated output
  hooks: allow callers to capture output
  receive-pack: convert update hooks to new API
  receive-pack: convert receive hooks to hook API

 builtin/hook.c              |   6 +
 builtin/receive-pack.c      | 272 ++++++++++++++++--------------------
 commit.c                    |   3 +
 hook.c                      |  29 +++-
 hook.h                      |  51 +++++++
 refs.c                      | 100 ++++++-------
 run-command.c               | 142 +++++++++++++++----
 run-command.h               |  38 +++++
 sequencer.c                 |  42 +++---
 t/helper/test-run-command.c |  67 ++++++++-
 t/t0061-run-command.sh      |  38 +++++
 transport.c                 |  89 ++++++------
 12 files changed, 585 insertions(+), 292 deletions(-)

-- 
2.51.2

[PATCH v5 01/11] run-command: add first helper for pp child states

[Adrian Ratiu]

There is a recurring pattern of testing parallel process child states
and file descriptors to determine if a child is running, receiving any
input or if it's ready for cleanup.

Name the pp_child structure and introduce a first helper to make these
checks more readable. Next commits will add more helpers and checks.

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/run-command.c b/run-command.c
index e3e02475cc..3989673569 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1478,15 +1478,22 @@ enum child_state {
 	GIT_CP_WAIT_CLEANUP,
 };
 
+struct parallel_child {
+	enum child_state state;
+	struct child_process process;
+	struct strbuf err;
+	void *data;
+};
+
+static int child_is_working(const struct parallel_child *pp_child)
+{
+	return pp_child->state == GIT_CP_WORKING;
+}
+
 struct parallel_processes {
 	size_t nr_processes;
 
-	struct {
-		enum child_state state;
-		struct child_process process;
-		struct strbuf err;
-		void *data;
-	} *children;
+	struct parallel_child *children;
 	/*
 	 * The struct pollfd is logically part of *children,
 	 * but the system call expects it as its own array.
@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,
 			  int signo)
 {
 	for (size_t i = 0; i < opts->processes; i++)
-		if (pp->children[i].state == GIT_CP_WORKING)
+		if (child_is_working(&pp->children[i]))
 			kill(pp->children[i].process.pid, signo);
 }
 
@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 
 	/* Buffer output from all pipes. */
 	for (size_t i = 0; i < opts->processes; i++) {
-		if (pp->children[i].state == GIT_CP_WORKING &&
+		if (child_is_working(&pp->children[i]) &&
 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
 			int n = strbuf_read_once(&pp->children[i].err,
 						 pp->children[i].process.err, 0);
@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)
 {
 	size_t i = pp->output_owner;
 
-	if (pp->children[i].state == GIT_CP_WORKING &&
+	if (child_is_working(&pp->children[i]) &&
 	    pp->children[i].err.len) {
 		strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			 * running process time.
 			 */
 			for (i = 0; i < n; i++)
-				if (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)
+				if (child_is_working(&pp->children[(pp->output_owner + i) % n]))
 					break;
 			pp->output_owner = (pp->output_owner + i) % n;
 		}
-- 
2.51.2

[PATCH v5 02/11] run-command: add stdin callback for parallelization

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

If a user of the run_processes_parallel() API wants to pipe a large
amount of information to the stdin of each parallel command, that
data could exceed the pipe buffer of the process's stdin and can be
too big to store in-memory via strbuf & friends or to slurp to a file.

Generally this is solved by repeatedly writing to child_process.in
between calls to start_command() and finish_command(). For a specific
pre-existing example of this, see transport.c:run_pre_push_hook().

This adds a generic callback API to run_processes_parallel() to do
exactly that in a unified manner, similar to the existing callback APIs,
which can then be used by hooks.h to convert the remaining hooks to the
new, simpler parallel interface.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 87 ++++++++++++++++++++++++++++++++++---
 run-command.h               | 21 +++++++++
 t/helper/test-run-command.c | 52 +++++++++++++++++++++-
 t/t0061-run-command.sh      | 31 +++++++++++++
 4 files changed, 182 insertions(+), 9 deletions(-)

diff --git a/run-command.c b/run-command.c
index 3989673569..aaf0e4ecee 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)
 	return pp_child->state == GIT_CP_WORKING;
 }
 
+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)
+{
+	return child_is_working(pp_child) && !pp_child->process.in;
+}
+
+static int child_is_receiving_input(const struct parallel_child *pp_child)
+{
+	return child_is_working(pp_child) && pp_child->process.in > 0;
+}
+
 struct parallel_processes {
 	size_t nr_processes;
 
@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
+static void pp_buffer_stdin(struct parallel_processes *pp,
+			    const struct run_process_parallel_opts *opts)
+{
+	/* Buffer stdin for each pipe. */
+	for (size_t i = 0; i < opts->processes; i++) {
+		struct child_process *proc = &pp->children[i].process;
+		int ret;
+
+		if (!child_is_receiving_input(&pp->children[i]))
+			continue;
+
+		/*
+		 * child input is provided via path_to_stdin when the feed_pipe cb is
+		 * missing, so we just signal an EOF.
+		 */
+		if (!opts->feed_pipe) {
+			close(proc->in);
+			proc->in = 0;
+			continue;
+		}
+
+		/**
+		 * Feed the pipe:
+		 *   ret < 0 means error
+		 *   ret == 0 means there is more data to be fed
+		 *   ret > 0 means feeding finished
+		 */
+		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
+		if (ret < 0)
+			die_errno("feed_pipe");
+
+		if (ret) {
+			close(proc->in);
+			proc->in = 0;
+		}
+	}
+}
+
 static void pp_buffer_stderr(struct parallel_processes *pp,
 			     const struct run_process_parallel_opts *opts,
 			     int output_timeout)
@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		pp->children[i].state = GIT_CP_FREE;
 		if (pp->pfd)
 			pp->pfd[i].fd = -1;
+		pp->children[i].process.in = 0;
 		child_process_init(&pp->children[i].process);
 
 		if (opts->ungroup) {
@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,
 	return result;
 }
 
+static void pp_handle_child_IO(struct parallel_processes *pp,
+				const struct run_process_parallel_opts *opts,
+				int output_timeout)
+{
+	/*
+	 * First push input, if any (it might no-op), to child tasks to avoid them blocking
+	 * after input. This also prevents deadlocks when ungrouping below, if a child blocks
+	 * while the parent also waits for them to finish.
+	 */
+	pp_buffer_stdin(pp, opts);
+
+	if (opts->ungroup) {
+		for (size_t i = 0; i < opts->processes; i++)
+			if (child_is_ready_for_cleanup(&pp->children[i]))
+				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
+	} else {
+		pp_buffer_stderr(pp, opts, output_timeout);
+		pp_output(pp);
+	}
+}
+
 void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	/*
+	 * Child tasks might receive input via stdin, terminating early (or not), so
+	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
+	 * actually writes the data to children stdin fds.
+	 */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
 	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (i = 0;
@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts->ungroup) {
-			for (size_t i = 0; i < opts->processes; i++)
-				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
-		} else {
-			pp_buffer_stderr(&pp, opts, output_timeout);
-			pp_output(&pp);
-		}
+		pp_handle_child_IO(&pp, opts, output_timeout);
 		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 
 	pp_cleanup(&pp, opts);
 
+	sigchain_pop(SIGPIPE);
+
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
diff --git a/run-command.h b/run-command.h
index 0df25e445f..e1ca965b5b 100644
--- a/run-command.h
+++ b/run-command.h
@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * This callback is repeatedly called on every child process who requests
+ * start_command() to create a pipe by setting child_process.in < 0.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel, and
+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.
+ *
+ * Returns < 0 for error
+ * Returns == 0 when there is more data to be fed (will be called again)
+ * Returns > 0 when finished (child closed fd or no more data to be fed)
+ */
+typedef int (*feed_pipe_fn)(int child_in,
+				void *pp_cb,
+				void *pp_task_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -473,6 +488,12 @@ struct run_process_parallel_opts
 	 */
 	start_failure_fn start_failure;
 
+	/*
+	 * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any
+	 * special handling.
+	 */
+	feed_pipe_fn feed_pipe;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 3719f23cc2..4a56456894 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -23,19 +23,26 @@ static int number_callbacks;
 static int parallel_next(struct child_process *cp,
 			 struct strbuf *err,
 			 void *cb,
-			 void **task_cb UNUSED)
+			 void **task_cb)
 {
 	struct child_process *d = cb;
 	if (number_callbacks >= 4)
 		return 0;
 
 	strvec_pushv(&cp->args, d->args.v);
+	cp->in = d->in;
+	cp->no_stdin = d->no_stdin;
 	if (err)
 		strbuf_addstr(err, "preloaded output of a child\n");
 	else
 		fprintf(stderr, "preloaded output of a child\n");
 
 	number_callbacks++;
+
+	/* test_stdin callback will use this to count remaining lines */
+	*task_cb = xmalloc(sizeof(int));
+	*(int*)(*task_cb) = 2;
+
 	return 1;
 }
 
@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
-			 void *pp_task_cb UNUSED)
+			 void *pp_task_cb)
 {
 	if (err)
 		strbuf_addstr(err, "asking for a quick stop\n");
 	else
 		fprintf(stderr, "asking for a quick stop\n");
+
+	FREE_AND_NULL(pp_task_cb);
+
 	return 1;
 }
 
+static int task_finished_quiet(int result UNUSED,
+			       struct strbuf *err UNUSED,
+			       void *pp_cb UNUSED,
+			       void *pp_task_cb)
+{
+	FREE_AND_NULL(pp_task_cb);
+	return 0;
+}
+
+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)
+{
+	int *lines_remaining = task_cb;
+
+	if (*lines_remaining) {
+		struct strbuf buf = STRBUF_INIT;
+		strbuf_addf(&buf, "sample stdin %d\n", --(*lines_remaining));
+		if (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {
+			if (errno == EPIPE) {
+				/* child closed stdin, nothing more to do */
+				strbuf_release(&buf);
+				return 1;
+			}
+			die_errno("write");
+		}
+		strbuf_release(&buf);
+	}
+
+	return !(*lines_remaining);
+}
+
 struct testsuite {
 	struct string_list tests, failed;
 	int next;
@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)
 	struct run_process_parallel_opts opts = {
 		.get_next_task = next_test,
 		.start_failure = test_failed,
+		.feed_pipe = test_stdin_pipe_feed,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
 		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		opts.get_next_task = no_job;
 		opts.task_finished = task_finished;
+	} else if (!strcmp(argv[1], "run-command-stdin")) {
+		proc.in = -1;
+		proc.no_stdin = 0;
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
+		opts.feed_pipe = test_stdin_pipe_feed;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 76d4936a87..2f77fde0d9 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command listens to stdin' '
+	cat >expect <<-\EOF &&
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	EOF
+
+	write_script stdin-script <<-\EOF &&
+	echo "listening for stdin:"
+	while read line
+	do
+		echo "$line"
+	done
+	EOF
+	test-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&
+	test_cmp expect actual
+'
+
 cat >expect <<-EOF
 preloaded output of a child
 asking for a quick stop
-- 
2.51.2

[PATCH v5 03/11] hook: provide stdin via callback

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This adds a callback mechanism for feeding stdin to hooks alongside
the existing path_to_stdin (which slurps a file's content to stdin).

The advantage of this new callback is that it can feed stdin without
going through the FS layer. This helps when feeding large amount of
data and uses the run-command parallel stdin callback introduced in
the preceding commit.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 hook.c | 23 ++++++++++++++++++++++-
 hook.h | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/hook.c b/hook.c
index b3de1048bf..5ddd7678d1 100644
--- a/hook.c
+++ b/hook.c
@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)
 static int pick_next_hook(struct child_process *cp,
 			  struct strbuf *out UNUSED,
 			  void *pp_cb,
-			  void **pp_task_cb UNUSED)
+			  void **pp_task_cb)
 {
 	struct hook_cb_data *hook_cb = pp_cb;
 	const char *hook_path = hook_cb->hook_path;
@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,
 
 	cp->no_stdin = 1;
 	strvec_pushv(&cp->env, hook_cb->options->env.v);
+
+	if (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	/* reopen the file for stdin; run_command closes it. */
 	if (hook_cb->options->path_to_stdin) {
 		cp->no_stdin = 0;
 		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
 	}
+
+	if (hook_cb->options->feed_pipe) {
+		cp->no_stdin = 0;
+		/* start_command() will allocate a pipe / stdin fd for us */
+		cp->in = -1;
+	}
+
 	cp->stdout_to_stderr = 1;
 	cp->trace2_hook_name = hook_cb->hook_name;
 	cp->dir = hook_cb->options->dir;
@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,
 	strvec_push(&cp->args, hook_path);
 	strvec_pushv(&cp->args, hook_cb->options->args.v);
 
+	/*
+	 * Provide per-hook internal state via task_cb for easy access, so
+	 * hook callbacks don't have to go through hook_cb->options.
+	 */
+	*pp_task_cb = hook_cb->options->feed_pipe_cb_data;
+
 	/*
 	 * This pick_next_hook() will be called again, we're only
 	 * running one hook, so indicate that no more work will be
@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
+		.feed_pipe = options->feed_pipe,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
 
+	if (options->path_to_stdin && options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	if (options->invoked_hook)
 		*options->invoked_hook = 0;
 
diff --git a/hook.h b/hook.h
index 11863fa734..2169d4a6bd 100644
--- a/hook.h
+++ b/hook.h
@@ -1,6 +1,7 @@
 #ifndef HOOK_H
 #define HOOK_H
 #include "strvec.h"
+#include "run-command.h"
 
 struct repository;
 
@@ -37,6 +38,43 @@ struct run_hooks_opt
 	 * Path to file which should be piped to stdin for each hook.
 	 */
 	const char *path_to_stdin;
+
+	/**
+	 * Callback used to incrementally feed a child hook stdin pipe.
+	 *
+	 * Useful especially if a hook consumes large quantities of data
+	 * (e.g. a list of all refs in a client push), so feeding it via
+	 * in-memory strings or slurping to/from files is inefficient.
+	 * While the callback allows piecemeal writing, it can also be
+	 * used for smaller inputs, where it gets called only once.
+	 *
+	 * Add hook callback initalization context to `feed_pipe_ctx`.
+	 * Add hook callback internal state to `feed_pipe_cb_data`.
+	 *
+	 */
+	feed_pipe_fn feed_pipe;
+
+	/**
+	 * Opaque data pointer used to pass context to `feed_pipe_fn`.
+	 *
+	 * It can be accessed via the second callback arg 'pp_cb':
+	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_ctx;
+
+	/**
+	 * Opaque data pointer used to keep internal state across callback calls.
+	 *
+	 * It can be accessed directly via the third callback arg 'pp_task_cb':
+	 * struct ... *state = pp_task_cb;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_cb_data;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.2

[PATCH v5 04/11] hook: convert 'post-rewrite' hook in sequencer.c to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Replace the custom run-command calls used by post-rewrite with
the newer and simpler hook_run_opt(), which does not need to
create a custom 'struct child_process' or call find_hook().

Another benefit of using the hook API is that hook_run_opt()
handles the SIGPIPE toggle logic.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 sequencer.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 5476d39ba9..71ed31c774 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,
 	return ret;
 }
 
+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
+{
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
+	int ret;
+
+	if (!to_pipe)
+		BUG("pipe_from_strbuf called without feed_pipe_ctx");
+
+	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
+
+	return 1; /* done writing */
+}
+
 static int run_rewrite_hook(const struct object_id *oldoid,
 			    const struct object_id *newoid)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 	int code;
 	struct strbuf sb = STRBUF_INIT;
-	const char *hook_path = find_hook(the_repository, "post-rewrite");
 
-	if (!hook_path)
-		return 0;
+	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
 
-	strvec_pushl(&proc.args, hook_path, "amend", NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "post-rewrite";
+	opt.feed_pipe_ctx = &sb;
+	opt.feed_pipe = pipe_from_strbuf;
+
+	strvec_push(&opt.args, "amend");
+
+	code = run_hooks_opt(the_repository, "post-rewrite", &opt);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
-	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
-	sigchain_push(SIGPIPE, SIG_IGN);
-	write_in_full(proc.in, sb.buf, sb.len);
-	close(proc.in);
 	strbuf_release(&sb);
-	sigchain_pop(SIGPIPE);
-	return finish_command(&proc);
+	return code;
 }
 
 void commit_post_rewrite(struct repository *r,
-- 
2.51.2

[PATCH v5 05/11] transport: convert pre-push to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Move the pre-push hook from custom run-command invocations to
the new hook API which doesn't require a custom child_process
structure and signal toggling.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 transport.c | 89 +++++++++++++++++++++++++++--------------------------
 1 file changed, 45 insertions(+), 44 deletions(-)

diff --git a/transport.c b/transport.c
index c7f06a7382..6d0f02be5d 100644
--- a/transport.c
+++ b/transport.c
@@ -1316,65 +1316,66 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
 	die(_("Aborting."));
 }
 
-static int run_pre_push_hook(struct transport *transport,
-			     struct ref *remote_refs)
-{
-	int ret = 0, x;
-	struct ref *r;
-	struct child_process proc = CHILD_PROCESS_INIT;
+struct feed_pre_push_hook_data {
 	struct strbuf buf;
-	const char *hook_path = find_hook(the_repository, "pre-push");
+	const struct ref *refs;
+};
 
-	if (!hook_path)
-		return 0;
+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
+{
+	struct feed_pre_push_hook_data *data = pp_task_cb;
+	const struct ref *r = data->refs;
+	int ret = 0;
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, transport->remote->name);
-	strvec_push(&proc.args, transport->url);
+	if (!r)
+		return 1; /* no more refs */
 
-	proc.in = -1;
-	proc.trace2_hook_name = "pre-push";
+	data->refs = r->next;
 
-	if (start_command(&proc)) {
-		finish_command(&proc);
-		return -1;
+	switch (r->status) {
+	case REF_STATUS_REJECT_NONFASTFORWARD:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_UPTODATE:
+		return 0; /* skip refs which won't be pushed */
+	default:
+		break;
 	}
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	if (!r->peer_ref)
+		return 0;
 
-	strbuf_init(&buf, 256);
+	strbuf_reset(&data->buf);
+	strbuf_addf(&data->buf, "%s %s %s %s\n",
+		    r->peer_ref->name, oid_to_hex(&r->new_oid),
+		    r->name, oid_to_hex(&r->old_oid));
 
-	for (r = remote_refs; r; r = r->next) {
-		if (!r->peer_ref) continue;
-		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
-		if (r->status == REF_STATUS_REJECT_STALE) continue;
-		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
-		if (r->status == REF_STATUS_UPTODATE) continue;
+	ret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);
+	if (ret < 0 && errno != EPIPE)
+		return ret; /* We do not mind if a hook does not read all refs. */
 
-		strbuf_reset(&buf);
-		strbuf_addf( &buf, "%s %s %s %s\n",
-			 r->peer_ref->name, oid_to_hex(&r->new_oid),
-			 r->name, oid_to_hex(&r->old_oid));
+	return 0;
+}
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			/* We do not mind if a hook does not read all refs. */
-			if (errno != EPIPE)
-				ret = -1;
-			break;
-		}
-	}
+static int run_pre_push_hook(struct transport *transport,
+			     struct ref *remote_refs)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct feed_pre_push_hook_data data;
+	int ret = 0;
+
+	strvec_push(&opt.args, transport->remote->name);
+	strvec_push(&opt.args, transport->url);
 
-	strbuf_release(&buf);
+	strbuf_init(&data.buf, 0);
+	data.refs = remote_refs;
 
-	x = close(proc.in);
-	if (!ret)
-		ret = x;
+	opt.feed_pipe = pre_push_hook_feed_stdin;
+	opt.feed_pipe_cb_data = &data;
 
-	sigchain_pop(SIGPIPE);
+	ret = run_hooks_opt(the_repository, "pre-push", &opt);
 
-	x = finish_command(&proc);
-	if (!ret)
-		ret = x;
+	strbuf_release(&data.buf);
 
 	return ret;
 }
-- 
2.51.2

[PATCH v5 06/11] reference-transaction: use hook API instead of run-command

[Adrian Ratiu]

Convert the reference-transaction hook to the new hook API,
so it doesn't need to set up a struct child_process, call
find_hook or toggle the pipe signals.

The stdin feed callback is processing one ref update per
call. I haven't noticed any performance degradation due
to this, however we can batch as many we want in each call,
to ensure a good pipe throughtput (i.e. the child does not
wait after stdin).

Helped-by: Emily Shaffer <nasamuffin@google.com>
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 refs.c | 100 ++++++++++++++++++++++++++++++---------------------------
 1 file changed, 52 insertions(+), 48 deletions(-)

diff --git a/refs.c b/refs.c
index 046b695bb2..e06e0cb072 100644
--- a/refs.c
+++ b/refs.c
@@ -2422,68 +2422,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-static int run_transaction_hook(struct ref_transaction *transaction,
-				const char *state)
+struct transaction_feed_cb_data {
+	size_t index;
+	struct strbuf buf;
+};
+
+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct strbuf buf = STRBUF_INIT;
-	const char *hook;
-	int ret = 0;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;
+	struct transaction_feed_cb_data *feed_cb_data = pp_task_cb;
+	struct strbuf *buf = &feed_cb_data->buf;
+	struct ref_update *update;
+	size_t i = feed_cb_data->index++;
+	int ret;
 
-	hook = find_hook(transaction->ref_store->repo, "reference-transaction");
-	if (!hook)
-		return ret;
+	if (i >= transaction->nr)
+		return 1; /* No more refs to process */
 
-	strvec_pushl(&proc.args, hook, state, NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "reference-transaction";
+	update = transaction->updates[i];
 
-	ret = start_command(&proc);
-	if (ret)
-		return ret;
+	if (update->flags & REF_LOG_ONLY)
+		return 0;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	strbuf_reset(buf);
 
-	for (size_t i = 0; i < transaction->nr; i++) {
-		struct ref_update *update = transaction->updates[i];
+	if (!(update->flags & REF_HAVE_OLD))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->old_target)
+		strbuf_addf(buf, "ref:%s ", update->old_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->old_oid));
 
-		if (update->flags & REF_LOG_ONLY)
-			continue;
+	if (!(update->flags & REF_HAVE_NEW))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->new_target)
+		strbuf_addf(buf, "ref:%s ", update->new_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->new_oid));
 
-		strbuf_reset(&buf);
+	strbuf_addf(buf, "%s\n", update->refname);
 
-		if (!(update->flags & REF_HAVE_OLD))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->old_target)
-			strbuf_addf(&buf, "ref:%s ", update->old_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->old_oid));
+	ret = write_in_full(hook_stdin_fd, buf->buf, buf->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
 
-		if (!(update->flags & REF_HAVE_NEW))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->new_target)
-			strbuf_addf(&buf, "ref:%s ", update->new_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->new_oid));
+	return 0; /* no more input to feed */
+}
+
+static int run_transaction_hook(struct ref_transaction *transaction,
+				const char *state)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct transaction_feed_cb_data feed_ctx = { 0 };
+	int ret = 0;
 
-		strbuf_addf(&buf, "%s\n", update->refname);
+	strvec_push(&opt.args, state);
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			if (errno != EPIPE) {
-				/* Don't leak errno outside this API */
-				errno = 0;
-				ret = -1;
-			}
-			break;
-		}
-	}
+	opt.feed_pipe = transaction_hook_feed_stdin;
+	opt.feed_pipe_ctx = transaction;
+	opt.feed_pipe_cb_data = &feed_ctx;
 
-	close(proc.in);
-	sigchain_pop(SIGPIPE);
-	strbuf_release(&buf);
+	strbuf_init(&feed_ctx.buf, 0);
+
+	ret = run_hooks_opt(transaction->ref_store->repo, "reference-transaction", &opt);
 
-	ret |= finish_command(&proc);
+	strbuf_release(&feed_ctx.buf);
 	return ret;
 }
 
-- 
2.51.2

[PATCH v5 07/11] hook: allow overriding the ungroup option

[Adrian Ratiu]

When calling run_process_parallel() in run_hooks_opt(), the
ungroup option is currently hardcoded to .ungroup = 1.

This causes problems when ungrouping should be disabled, for
example when sideband-reading collated output from child hooks,
because sideband-reading and ungrouping are mutually exclusive.

Thus a new hook.h option is added to allow overriding.

The existing ungroup=1 behavior is preserved in the run_hooks()
API and the "hook run" command. We could modify these to take
an option if necessary, so I added two code comments there.

Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/hook.c | 6 ++++++
 commit.c       | 3 +++
 hook.c         | 5 ++++-
 hook.h         | 5 +++++
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/builtin/hook.c b/builtin/hook.c
index 7afec380d2..73e7b8c2e8 100644
--- a/builtin/hook.c
+++ b/builtin/hook.c
@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,
 	if (!argc)
 		goto usage;
 
+	/*
+	 * All current "hook run" use-cases require ungrouped child output.
+	 * If this changes, a hook run argument can be added to toggle it.
+	 */
+	opt.ungroup = 1;
+
 	/*
 	 * Having a -- for "run" when providing <hook-args> is
 	 * mandatory.
diff --git a/commit.c b/commit.c
index 709c9eed58..2527f15d9d 100644
--- a/commit.c
+++ b/commit.c
@@ -1978,6 +1978,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,
 		strvec_push(&opt.args, arg);
 	va_end(args);
 
+	/* All commit hook use-cases require ungrouping child output. */
+	opt.ungroup = 1;
+
 	opt.invoked_hook = invoked_hook;
 	return run_hooks_opt(the_repository, name, &opt);
 }
diff --git a/hook.c b/hook.c
index 5ddd7678d1..00a1e2ad22 100644
--- a/hook.c
+++ b/hook.c
@@ -153,7 +153,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.tr2_label = hook_name,
 
 		.processes = 1,
-		.ungroup = 1,
+		.ungroup = options->ungroup,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
@@ -198,6 +198,9 @@ int run_hooks(struct repository *r, const char *hook_name)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
+	/* All use-cases of this API require ungrouping. */
+	opt.ungroup = 1;
+
 	return run_hooks_opt(r, hook_name, &opt);
 }
 
diff --git a/hook.h b/hook.h
index 2169d4a6bd..78a1a44690 100644
--- a/hook.h
+++ b/hook.h
@@ -34,6 +34,11 @@ struct run_hooks_opt
 	 */
 	int *invoked_hook;
 
+	/**
+	 * Allow hooks to set run_processes_parallel() 'ungroup' behavior.
+	 */
+	unsigned int ungroup:1;
+
 	/**
 	 * Path to file which should be piped to stdin for each hook.
 	 */
-- 
2.51.2

[PATCH v5 08/11] run-command: allow capturing of collated output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some callers, for example server-side hooks which wish to relay hook
output to clients across a transport, want to capture what would
normally print to stderr and do something else with it. Allow that via a
callback.

By calling the callback regardless of whether there's output available,
we allow clients to send e.g. a keepalive if necessary.

Because we expose a strbuf, not a fd or FILE*, there's no need to create
a temporary pipe or similar - we can just skip the print to stderr and
instead hand it to the caller.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 30 ++++++++++++++++++++++--------
 run-command.h               | 17 +++++++++++++++++
 t/helper/test-run-command.c | 15 +++++++++++++++
 t/t0061-run-command.sh      |  7 +++++++
 4 files changed, 61 insertions(+), 8 deletions(-)

diff --git a/run-command.c b/run-command.c
index aaf0e4ecee..2d3c2ac55c 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1595,7 +1595,10 @@ static void pp_cleanup(struct parallel_processes *pp,
 	 * When get_next_task added messages to the buffer in its last
 	 * iteration, the buffered output is non empty.
 	 */
-	strbuf_write(&pp->buffered_output, stderr);
+	if (opts->consume_output)
+		opts->consume_output(&pp->buffered_output, opts->data);
+	else
+		strbuf_write(&pp->buffered_output, stderr);
 	strbuf_release(&pp->buffered_output);
 
 	sigchain_pop_common();
@@ -1734,13 +1737,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 	}
 }
 
-static void pp_output(const struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp,
+		      const struct run_process_parallel_opts *opts)
 {
 	size_t i = pp->output_owner;
 
 	if (child_is_working(&pp->children[i]) &&
 	    pp->children[i].err.len) {
-		strbuf_write(&pp->children[i].err, stderr);
+		if (opts->consume_output)
+			opts->consume_output(&pp->children[i].err, opts->data);
+		else
+			strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
 	}
 }
@@ -1788,11 +1795,15 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		} else {
 			const size_t n = opts->processes;
 
-			strbuf_write(&pp->children[i].err, stderr);
+			/* Output errors, then all other finished child processes */
+			if (opts->consume_output) {
+				opts->consume_output(&pp->children[i].err, opts->data);
+				opts->consume_output(&pp->buffered_output, opts->data);
+			} else {
+				strbuf_write(&pp->children[i].err, stderr);
+				strbuf_write(&pp->buffered_output, stderr);
+			}
 			strbuf_reset(&pp->children[i].err);
-
-			/* Output all other finished child processes */
-			strbuf_write(&pp->buffered_output, stderr);
 			strbuf_reset(&pp->buffered_output);
 
 			/*
@@ -1829,7 +1840,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,
 				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
 	} else {
 		pp_buffer_stderr(pp, opts, output_timeout);
-		pp_output(pp);
+		pp_output(pp, opts);
 	}
 }
 
@@ -1852,6 +1863,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	if (opts->ungroup && opts->consume_output)
+		BUG("ungroup and reading output are mutualy exclusive");
+
 	/*
 	 * Child tasks might receive input via stdin, terminating early (or not), so
 	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
diff --git a/run-command.h b/run-command.h
index e1ca965b5b..7093252863 100644
--- a/run-command.h
+++ b/run-command.h
@@ -435,6 +435,17 @@ typedef int (*feed_pipe_fn)(int child_in,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * If this callback is provided, output is collated into a new pipe instead
+ * of the process stderr. Then `consume_output_fn` will be called repeatedly
+ * with output contained in the `output` arg. It will also be called with an
+ * empty `output` to allow for keepalives or similar operations if necessary.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel.
+ * No task cookie is provided because the callback receives collated output.
+ */
+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -494,6 +505,12 @@ struct run_process_parallel_opts
 	 */
 	feed_pipe_fn feed_pipe;
 
+	/*
+	 * consume_output: see consume_output_fn() above. This can be NULL
+	 * to omit any special handling.
+	 */
+	consume_output_fn consume_output;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 4a56456894..49eace8dce 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,
 	return 0;
 }
 
+static void test_divert_output(struct strbuf *output, void *cb UNUSED)
+{
+	FILE *output_file;
+
+	output_file = fopen("./output_file", "a");
+
+	strbuf_write(output, output_file);
+	fclose(output_file);
+}
+
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)
 		.get_next_task = next_test,
 		.start_failure = test_failed,
 		.feed_pipe = test_stdin_pipe_feed,
+		.consume_output = test_divert_output,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished_quiet;
 		opts.feed_pipe = test_stdin_pipe_feed;
+	} else if (!strcmp(argv[1], "run-command-divert-output")) {
+		opts.get_next_task = parallel_next;
+		opts.consume_output = test_divert_output;
+		opts.task_finished = task_finished_quiet;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 2f77fde0d9..74529e219e 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command can divert output' '
+	test_when_finished rm output_file &&
+	test-tool run-command run-command-divert-output 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
+	test_must_be_empty actual &&
+	test_cmp expect output_file
+'
+
 test_expect_success 'run_command listens to stdin' '
 	cat >expect <<-\EOF &&
 	preloaded output of a child
-- 
2.51.2

[PATCH v5 09/11] hooks: allow callers to capture output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some server-side hooks will require capturing output to send over
sideband instead of printing directly to stderr. Expose that capability.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c | 1 +
 hook.h | 8 ++++++++
 2 files changed, 9 insertions(+)

diff --git a/hook.c b/hook.c
index 00a1e2ad22..35211e5ed7 100644
--- a/hook.c
+++ b/hook.c
@@ -158,6 +158,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
 		.feed_pipe = options->feed_pipe,
+		.consume_output = options->consume_output,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
diff --git a/hook.h b/hook.h
index 78a1a44690..ae502178b9 100644
--- a/hook.h
+++ b/hook.h
@@ -80,6 +80,14 @@ struct run_hooks_opt
 	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
 	 */
 	void *feed_pipe_cb_data;
+
+	/*
+	 * Populate this to capture output and prevent it from being printed to
+	 * stderr. This will be passed directly through to
+	 * run_command:run_parallel_processes(). See t/helper/test-run-command.c
+	 * for an example.
+	 */
+	consume_output_fn consume_output;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.2

[PATCH v5 10/11] receive-pack: convert update hooks to new API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Use the new hook sideband API introduced in the previous commit.

The hook API avoids creating a custom struct child_process and other
internal hook plumbing (e.g. calling find_hook()) and prepares for
the specification of hooks via configs or running parallel hooks.

Execution is still sequential through the current hook.[ch] via the
run_process_parallel_opts.processes=1 arg.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 64 +++++++++++++++++-------------------------
 1 file changed, 25 insertions(+), 39 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 9c49174616..d1c40a768d 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -918,6 +918,16 @@ static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
 	return 0;
 }
 
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
+{
+	if (!output)
+		BUG("output must be non-NULL");
+
+	/* buffer might be empty for keepalives */
+	if (output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
+}
+
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
@@ -941,29 +951,18 @@ static int run_receive_hook(struct command *commands,
 
 static int run_update_hook(struct command *cmd)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	int code;
-	const char *hook_path = find_hook(the_repository, "update");
-
-	if (!hook_path)
-		return 0;
-
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, cmd->ref_name);
-	strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
-	strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "update";
+	strvec_pushl(&opt.args,
+		     cmd->ref_name,
+		     oid_to_hex(&cmd->old_oid),
+		     oid_to_hex(&cmd->new_oid),
+		     NULL);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
 	if (use_sideband)
-		copy_to_sideband(proc.err, -1, NULL);
-	return finish_command(&proc);
+		opt.consume_output = hook_output_to_sideband;
+
+	return run_hooks_opt(the_repository, "update", &opt);
 }
 
 static struct command *find_command_by_refname(struct command *list,
@@ -1640,33 +1639,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 static void run_update_post_hook(struct command *commands)
 {
 	struct command *cmd;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	const char *hook;
-
-	hook = find_hook(the_repository, "post-update");
-	if (!hook)
-		return;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		if (cmd->error_string || cmd->did_not_exist)
 			continue;
-		if (!proc.args.nr)
-			strvec_push(&proc.args, hook);
-		strvec_push(&proc.args, cmd->ref_name);
+		strvec_push(&opt.args, cmd->ref_name);
 	}
-	if (!proc.args.nr)
+	if (!opt.args.nr)
 		return;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "post-update";
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
 
-	if (!start_command(&proc)) {
-		if (use_sideband)
-			copy_to_sideband(proc.err, -1, NULL);
-		finish_command(&proc);
-	}
+	run_hooks_opt(the_repository, "post-update", &opt);
 }
 
 static void check_aliased_update_internal(struct command *cmd,
-- 
2.51.2

[PATCH v5 11/11] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This converts the last remaining hooks to the new hook API, for
the same benefits as the previous conversions (no need to toggle
signals, manage custom struct child_process, call find_hook(),
prepares for specifyinig hooks via configs, etc.).

I noticed a performance degradation when processing large amounts
of hook input with just 1 line per callback, due to run-command's
poll loop, therefore I batched 500 lines per callback, to ensure
similar pipe throughput as before and to avoid hook child waiting
on stdin.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 214 ++++++++++++++++++-----------------------
 1 file changed, 95 insertions(+), 119 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index d1c40a768d..f22d975879 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)
 	return retval;
 }
 
-static void prepare_push_cert_sha1(struct child_process *proc)
+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)
 {
 	static int already_done;
 
@@ -775,23 +775,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)
 		nonce_status = check_nonce(sigcheck.payload);
 	}
 	if (!is_null_oid(&push_cert_oid)) {
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT=%s",
 			     oid_to_hex(&push_cert_oid));
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_SIGNER=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_SIGNER=%s",
 			     sigcheck.signer ? sigcheck.signer : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_KEY=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_KEY=%s",
 			     sigcheck.key ? sigcheck.key : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_STATUS=%c",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_STATUS=%c",
 			     sigcheck.result);
 		if (push_cert_nonce) {
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE=%s",
 				     push_cert_nonce);
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE_STATUS=%s",
 				     nonce_status);
 			if (nonce_status == NONCE_SLOP)
-				strvec_pushf(&proc->env,
+				strvec_pushf(&opt->env,
 					     "GIT_PUSH_CERT_NONCE_SLOP=%ld",
 					     nonce_stamp_slop);
 		}
@@ -803,119 +803,64 @@ struct receive_hook_feed_state {
 	struct ref_push_report *report;
 	int skip_broken;
 	struct strbuf buf;
-	const struct string_list *push_options;
 };
 
-typedef int (*feed_fn)(void *, const char **, size_t *);
-static int run_and_feed_hook(const char *hook_name, feed_fn feed,
-			     struct receive_hook_feed_state *feed_state)
+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct async muxer;
-	int code;
-	const char *hook_path = find_hook(the_repository, hook_name);
+	struct receive_hook_feed_state *state = pp_task_cb;
+	struct command *cmd = state->cmd;
+	unsigned int lines_batch_size = 500;
 
-	if (!hook_path)
-		return 0;
+	strbuf_reset(&state->buf);
 
-	strvec_push(&proc.args, hook_path);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = hook_name;
-
-	if (feed_state->push_options) {
-		size_t i;
-		for (i = 0; i < feed_state->push_options->nr; i++)
-			strvec_pushf(&proc.env,
-				     "GIT_PUSH_OPTION_%"PRIuMAX"=%s",
-				     (uintmax_t)i,
-				     feed_state->push_options->items[i].string);
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
-			     (uintmax_t)feed_state->push_options->nr);
-	} else
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
+	/* batch lines to avoid going through run-command's poll loop for each line */
+	for (unsigned int i = 0; i < lines_batch_size; i++) {
+		while (cmd &&
+		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
+			cmd = cmd->next;
 
-	if (tmp_objdir)
-		strvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));
+		if (!cmd)
+			break;  /* no more commands left */
 
-	if (use_sideband) {
-		memset(&muxer, 0, sizeof(muxer));
-		muxer.proc = copy_to_sideband;
-		muxer.in = -1;
-		code = start_async(&muxer);
-		if (code)
-			return code;
-		proc.err = muxer.in;
-	}
+		if (!state->report)
+			state->report = cmd->report;
 
-	prepare_push_cert_sha1(&proc);
+		if (state->report) {
+			struct object_id *old_oid;
+			struct object_id *new_oid;
+			const char *ref_name;
 
-	code = start_command(&proc);
-	if (code) {
-		if (use_sideband)
-			finish_async(&muxer);
-		return code;
-	}
+			old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
+			new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
+			ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(old_oid), oid_to_hex(new_oid),
+				    ref_name);
 
-	while (1) {
-		const char *buf;
-		size_t n;
-		if (feed(feed_state, &buf, &n))
-			break;
-		if (write_in_full(proc.in, buf, n) < 0)
-			break;
+			state->report = state->report->next;
+			if (!state->report)
+				cmd = cmd->next;
+		} else {
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
+				    cmd->ref_name);
+			cmd = cmd->next;
+		}
 	}
-	close(proc.in);
-	if (use_sideband)
-		finish_async(&muxer);
 
-	sigchain_pop(SIGPIPE);
-
-	return finish_command(&proc);
-}
+	state->cmd = cmd;
 
-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
-{
-	struct receive_hook_feed_state *state = state_;
-	struct command *cmd = state->cmd;
-
-	while (cmd &&
-	       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
-		cmd = cmd->next;
-	if (!cmd)
-		return -1; /* EOF */
-	if (!bufp)
-		return 0; /* OK, can feed something. */
-	strbuf_reset(&state->buf);
-	if (!state->report)
-		state->report = cmd->report;
-	if (state->report) {
-		struct object_id *old_oid;
-		struct object_id *new_oid;
-		const char *ref_name;
-
-		old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
-		new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
-		ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(old_oid), oid_to_hex(new_oid),
-			    ref_name);
-		state->report = state->report->next;
-		if (!state->report)
-			state->cmd = cmd->next;
-	} else {
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
-			    cmd->ref_name);
-		state->cmd = cmd->next;
-	}
-	if (bufp) {
-		*bufp = state->buf.buf;
-		*sizep = state->buf.len;
+	if (state->buf.len > 0) {
+		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
+		if (ret < 0) {
+			if (errno == EPIPE)
+				return 1; /* child closed pipe */
+			return ret;
+		}
 	}
-	return 0;
+
+	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
 }
 
 static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
@@ -933,20 +878,51 @@ static int run_receive_hook(struct command *commands,
 			    int skip_broken,
 			    const struct string_list *push_options)
 {
-	struct receive_hook_feed_state state;
-	int status;
-
-	strbuf_init(&state.buf, 0);
-	state.cmd = commands;
-	state.skip_broken = skip_broken;
-	state.report = NULL;
-	if (feed_receive_hook(&state, NULL, NULL))
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct command *iter = commands;
+	struct receive_hook_feed_state *feed_state;
+	int ret;
+
+	/* if there are no valid commands, don't invoke the hook at all. */
+	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
+		iter = iter->next;
+	if (!iter)
 		return 0;
-	state.cmd = commands;
-	state.push_options = push_options;
-	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
-	strbuf_release(&state.buf);
-	return status;
+
+	if (push_options) {
+		int i;
+		for (i = 0; i < push_options->nr; i++)
+			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
+				     push_options->items[i].string);
+		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
+					     (uintmax_t)push_options->nr);
+	} else
+		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+
+	if (tmp_objdir)
+		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+
+	prepare_push_cert_sha1(&opt);
+
+	/* set up sideband printer */
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
+
+	/* set up stdin callback */
+	feed_state = xmalloc(sizeof(struct receive_hook_feed_state));
+	feed_state->cmd = commands;
+	feed_state->skip_broken = skip_broken;
+	feed_state->report = NULL;
+	strbuf_init(&feed_state->buf, 0);
+	opt.feed_pipe_cb_data = feed_state;
+	opt.feed_pipe = feed_receive_hook_cb;
+
+	ret = run_hooks_opt(the_repository, hook_name, &opt);
+
+	strbuf_release(&feed_state->buf);
+	FREE_AND_NULL(opt.feed_pipe_cb_data);
+
+	return ret;
 }
 
 static int run_update_hook(struct command *cmd)
-- 
2.51.2

Re: [PATCH v5 11/11] receive-pack: convert receive hooks to hook API

[Patrick Steinhardt]

On Thu, Dec 18, 2025 at 07:11:25PM +0200, Adrian Ratiu wrote:
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index d1c40a768d..f22d975879 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -933,20 +878,51 @@ static int run_receive_hook(struct command *commands,
>  			    int skip_broken,
>  			    const struct string_list *push_options)
>  {
> -	struct receive_hook_feed_state state;
> -	int status;
> -
> -	strbuf_init(&state.buf, 0);
> -	state.cmd = commands;
> -	state.skip_broken = skip_broken;
> -	state.report = NULL;
> -	if (feed_receive_hook(&state, NULL, NULL))
> +	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
> +	struct command *iter = commands;
> +	struct receive_hook_feed_state *feed_state;
> +	int ret;
> +
> +	/* if there are no valid commands, don't invoke the hook at all. */
> +	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
> +		iter = iter->next;
> +	if (!iter)
>  		return 0;
> -	state.cmd = commands;
> -	state.push_options = push_options;
> -	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
> -	strbuf_release(&state.buf);
> -	return status;
> +
> +	if (push_options) {
> +		int i;

Nit: this variable could be declared in the loop.

> +		for (i = 0; i < push_options->nr; i++)
> +			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
> +				     push_options->items[i].string);
> +		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
> +					     (uintmax_t)push_options->nr);
> +	} else
> +		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");

Nit: this should also use curly braces according to our modern coding
guidelines:

	- When there are multiple arms to a conditional and some of them
	  require braces, enclose even a single line block in braces for
	  consistency.

> +	if (tmp_objdir)
> +		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
> +
> +	prepare_push_cert_sha1(&opt);
> +
> +	/* set up sideband printer */
> +	if (use_sideband)
> +		opt.consume_output = hook_output_to_sideband;
> +
> +	/* set up stdin callback */
> +	feed_state = xmalloc(sizeof(struct receive_hook_feed_state));

It feels somewhat unnecessary to allocate this structure as it could
have just as well be allocated on the stack.

> +	feed_state->cmd = commands;
> +	feed_state->skip_broken = skip_broken;
> +	feed_state->report = NULL;
> +	strbuf_init(&feed_state->buf, 0);
> +	opt.feed_pipe_cb_data = feed_state;
> +	opt.feed_pipe = feed_receive_hook_cb;
> +
> +	ret = run_hooks_opt(the_repository, hook_name, &opt);
> +
> +	strbuf_release(&feed_state->buf);
> +	FREE_AND_NULL(opt.feed_pipe_cb_data);
> +
> +	return ret;
>  }

All of these are nits, and the remaining patches all look good to me.
I'll leave it to you to decide whether you want to do one more (and
hopefully last) reroll.

Thanks!

Patrick

Re: [PATCH v5 11/11] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

On Fri, 19 Dec 2025, Patrick Steinhardt <ps@pks.im> wrote:
> On Thu, Dec 18, 2025 at 07:11:25PM +0200, Adrian Ratiu wrote:
>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
>> index d1c40a768d..f22d975879 100644
>> --- a/builtin/receive-pack.c
>> +++ b/builtin/receive-pack.c
>> @@ -933,20 +878,51 @@ static int run_receive_hook(struct command *commands,
>>  			    int skip_broken,
>>  			    const struct string_list *push_options)
>>  {
>> -	struct receive_hook_feed_state state;
>> -	int status;
>> -
>> -	strbuf_init(&state.buf, 0);
>> -	state.cmd = commands;
>> -	state.skip_broken = skip_broken;
>> -	state.report = NULL;
>> -	if (feed_receive_hook(&state, NULL, NULL))
>> +	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
>> +	struct command *iter = commands;
>> +	struct receive_hook_feed_state *feed_state;
>> +	int ret;
>> +
>> +	/* if there are no valid commands, don't invoke the hook at all. */
>> +	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
>> +		iter = iter->next;
>> +	if (!iter)
>>  		return 0;
>> -	state.cmd = commands;
>> -	state.push_options = push_options;
>> -	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
>> -	strbuf_release(&state.buf);
>> -	return status;
>> +
>> +	if (push_options) {
>> +		int i;
>
> Nit: this variable could be declared in the loop.
>
>> +		for (i = 0; i < push_options->nr; i++)
>> +			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
>> +				     push_options->items[i].string);
>> +		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
>> +					     (uintmax_t)push_options->nr);
>> +	} else
>> +		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
>
> Nit: this should also use curly braces according to our modern coding
> guidelines:
>
> 	- When there are multiple arms to a conditional and some of them
> 	  require braces, enclose even a single line block in braces for
> 	  consistency.
>
>> +	if (tmp_objdir)
>> +		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
>> +
>> +	prepare_push_cert_sha1(&opt);
>> +
>> +	/* set up sideband printer */
>> +	if (use_sideband)
>> +		opt.consume_output = hook_output_to_sideband;
>> +
>> +	/* set up stdin callback */
>> +	feed_state = xmalloc(sizeof(struct receive_hook_feed_state));
>
> It feels somewhat unnecessary to allocate this structure as it could
> have just as well be allocated on the stack.
>
>> +	feed_state->cmd = commands;
>> +	feed_state->skip_broken = skip_broken;
>> +	feed_state->report = NULL;
>> +	strbuf_init(&feed_state->buf, 0);
>> +	opt.feed_pipe_cb_data = feed_state;
>> +	opt.feed_pipe = feed_receive_hook_cb;
>> +
>> +	ret = run_hooks_opt(the_repository, hook_name, &opt);
>> +
>> +	strbuf_release(&feed_state->buf);
>> +	FREE_AND_NULL(opt.feed_pipe_cb_data);
>> +
>> +	return ret;
>>  }
>
> All of these are nits, and the remaining patches all look good to me.
> I'll leave it to you to decide whether you want to do one more (and
> hopefully last) reroll.

Thanks for the review, much appreciated as always.

Sure, I can do one more reroll to fix these latest nits.

Will give it about a week in case it gathers more feedback.

[PATCH v6 00/11] Convert remaining hooks to hook.h

[Adrian Ratiu]

Hello everyone,

This series finishes the hook.[ch] conversion for the remaining hooks in
preparation for adding config-based hooks and enabling parallel hook
execution where possible (that will be a separate series from this one).

v6 is minor refresh for some last nits. Details + range-diff below.

It is based on the latest master branch. There are no conflicts with
next and seen branches, the code is available on GitHub [1] and a
successful CI run [2] is also provided.

1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v6
2: https://github.com/10ne1/git/actions/runs/20436908624

Changes in v6:
* Last commit cb state is new allocated on the stack (Patrick)
* Small loop variable and brace fixes (Patrick)

Range-diff v5 -> v6:
 1:  8c16f1bcbf =  1:  8c16f1bcbf run-command: add first helper for pp child states
 2:  5e6e05ba92 =  2:  5e6e05ba92 run-command: add stdin callback for parallelization
 3:  3669acfe6a =  3:  3669acfe6a hook: provide stdin via callback
 4:  bf9d8680e4 =  4:  bf9d8680e4 hook: convert 'post-rewrite' hook in sequencer.c to hook API
 5:  bdcc1cff34 =  5:  bdcc1cff34 transport: convert pre-push to hook API
 6:  9c1d5e8726 =  6:  9c1d5e8726 reference-transaction: use hook API instead of run-command
 7:  0b986bf0fb =  7:  0b986bf0fb hook: allow overriding the ungroup option
 8:  5f07d07acc =  8:  5f07d07acc run-command: allow capturing of collated output
 9:  c4ff1e2270 =  9:  c4ff1e2270 hooks: allow callers to capture output
10:  15c831ca15 = 10:  15c831ca15 receive-pack: convert update hooks to new API
11:  4bd2c2974a ! 11:  432dc14871 receive-pack: convert receive hooks to hook API
    @@ builtin/receive-pack.c: struct receive_hook_feed_state {
     -		finish_async(&muxer);
      
     -	sigchain_pop(SIGPIPE);
    --
    --	return finish_command(&proc);
    --}
     +	state->cmd = cmd;
      
    +-	return finish_command(&proc);
    +-}
    +-
     -static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
     -{
     -	struct receive_hook_feed_state *state = state_;
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
     -	if (feed_receive_hook(&state, NULL, NULL))
     +	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
     +	struct command *iter = commands;
    -+	struct receive_hook_feed_state *feed_state;
    ++	struct receive_hook_feed_state feed_state;
     +	int ret;
     +
     +	/* if there are no valid commands, don't invoke the hook at all. */
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
     -	return status;
     +
     +	if (push_options) {
    -+		int i;
    -+		for (i = 0; i < push_options->nr; i++)
    ++		for (int i = 0; i < push_options->nr; i++)
     +			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
     +				     push_options->items[i].string);
     +		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
     +					     (uintmax_t)push_options->nr);
    -+	} else
    ++	} else {
     +		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
    ++	}
     +
     +	if (tmp_objdir)
     +		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
    @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
     +		opt.consume_output = hook_output_to_sideband;
     +
     +	/* set up stdin callback */
    -+	feed_state = xmalloc(sizeof(struct receive_hook_feed_state));
    -+	feed_state->cmd = commands;
    -+	feed_state->skip_broken = skip_broken;
    -+	feed_state->report = NULL;
    -+	strbuf_init(&feed_state->buf, 0);
    -+	opt.feed_pipe_cb_data = feed_state;
    ++	feed_state.cmd = commands;
    ++	feed_state.skip_broken = skip_broken;
    ++	feed_state.report = NULL;
    ++	strbuf_init(&feed_state.buf, 0);
    ++	opt.feed_pipe_cb_data = &feed_state;
     +	opt.feed_pipe = feed_receive_hook_cb;
     +
     +	ret = run_hooks_opt(the_repository, hook_name, &opt);
     +
    -+	strbuf_release(&feed_state->buf);
    -+	FREE_AND_NULL(opt.feed_pipe_cb_data);
    ++	strbuf_release(&feed_state.buf);
     +
     +	return ret;
      }

Adrian Ratiu (3):
  run-command: add first helper for pp child states
  reference-transaction: use hook API instead of run-command
  hook: allow overriding the ungroup option

Emily Shaffer (8):
  run-command: add stdin callback for parallelization
  hook: provide stdin via callback
  hook: convert 'post-rewrite' hook in sequencer.c to hook API
  transport: convert pre-push to hook API
  run-command: allow capturing of collated output
  hooks: allow callers to capture output
  receive-pack: convert update hooks to new API
  receive-pack: convert receive hooks to hook API

 builtin/hook.c              |   6 +
 builtin/receive-pack.c      | 270 +++++++++++++++---------------------
 commit.c                    |   3 +
 hook.c                      |  29 +++-
 hook.h                      |  51 +++++++
 refs.c                      | 100 ++++++-------
 run-command.c               | 142 +++++++++++++++----
 run-command.h               |  38 +++++
 sequencer.c                 |  42 +++---
 t/helper/test-run-command.c |  67 ++++++++-
 t/t0061-run-command.sh      |  38 +++++
 transport.c                 |  89 ++++++------
 12 files changed, 583 insertions(+), 292 deletions(-)

-- 
2.51.2

[PATCH v6 01/11] run-command: add first helper for pp child states

[Adrian Ratiu]

There is a recurring pattern of testing parallel process child states
and file descriptors to determine if a child is running, receiving any
input or if it's ready for cleanup.

Name the pp_child structure and introduce a first helper to make these
checks more readable. Next commits will add more helpers and checks.

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/run-command.c b/run-command.c
index e3e02475cc..3989673569 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1478,15 +1478,22 @@ enum child_state {
 	GIT_CP_WAIT_CLEANUP,
 };
 
+struct parallel_child {
+	enum child_state state;
+	struct child_process process;
+	struct strbuf err;
+	void *data;
+};
+
+static int child_is_working(const struct parallel_child *pp_child)
+{
+	return pp_child->state == GIT_CP_WORKING;
+}
+
 struct parallel_processes {
 	size_t nr_processes;
 
-	struct {
-		enum child_state state;
-		struct child_process process;
-		struct strbuf err;
-		void *data;
-	} *children;
+	struct parallel_child *children;
 	/*
 	 * The struct pollfd is logically part of *children,
 	 * but the system call expects it as its own array.
@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,
 			  int signo)
 {
 	for (size_t i = 0; i < opts->processes; i++)
-		if (pp->children[i].state == GIT_CP_WORKING)
+		if (child_is_working(&pp->children[i]))
 			kill(pp->children[i].process.pid, signo);
 }
 
@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 
 	/* Buffer output from all pipes. */
 	for (size_t i = 0; i < opts->processes; i++) {
-		if (pp->children[i].state == GIT_CP_WORKING &&
+		if (child_is_working(&pp->children[i]) &&
 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
 			int n = strbuf_read_once(&pp->children[i].err,
 						 pp->children[i].process.err, 0);
@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)
 {
 	size_t i = pp->output_owner;
 
-	if (pp->children[i].state == GIT_CP_WORKING &&
+	if (child_is_working(&pp->children[i]) &&
 	    pp->children[i].err.len) {
 		strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			 * running process time.
 			 */
 			for (i = 0; i < n; i++)
-				if (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)
+				if (child_is_working(&pp->children[(pp->output_owner + i) % n]))
 					break;
 			pp->output_owner = (pp->output_owner + i) % n;
 		}
-- 
2.51.2

[PATCH v6 02/11] run-command: add stdin callback for parallelization

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

If a user of the run_processes_parallel() API wants to pipe a large
amount of information to the stdin of each parallel command, that
data could exceed the pipe buffer of the process's stdin and can be
too big to store in-memory via strbuf & friends or to slurp to a file.

Generally this is solved by repeatedly writing to child_process.in
between calls to start_command() and finish_command(). For a specific
pre-existing example of this, see transport.c:run_pre_push_hook().

This adds a generic callback API to run_processes_parallel() to do
exactly that in a unified manner, similar to the existing callback APIs,
which can then be used by hooks.h to convert the remaining hooks to the
new, simpler parallel interface.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 87 ++++++++++++++++++++++++++++++++++---
 run-command.h               | 21 +++++++++
 t/helper/test-run-command.c | 52 +++++++++++++++++++++-
 t/t0061-run-command.sh      | 31 +++++++++++++
 4 files changed, 182 insertions(+), 9 deletions(-)

diff --git a/run-command.c b/run-command.c
index 3989673569..aaf0e4ecee 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)
 	return pp_child->state == GIT_CP_WORKING;
 }
 
+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)
+{
+	return child_is_working(pp_child) && !pp_child->process.in;
+}
+
+static int child_is_receiving_input(const struct parallel_child *pp_child)
+{
+	return child_is_working(pp_child) && pp_child->process.in > 0;
+}
+
 struct parallel_processes {
 	size_t nr_processes;
 
@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
+static void pp_buffer_stdin(struct parallel_processes *pp,
+			    const struct run_process_parallel_opts *opts)
+{
+	/* Buffer stdin for each pipe. */
+	for (size_t i = 0; i < opts->processes; i++) {
+		struct child_process *proc = &pp->children[i].process;
+		int ret;
+
+		if (!child_is_receiving_input(&pp->children[i]))
+			continue;
+
+		/*
+		 * child input is provided via path_to_stdin when the feed_pipe cb is
+		 * missing, so we just signal an EOF.
+		 */
+		if (!opts->feed_pipe) {
+			close(proc->in);
+			proc->in = 0;
+			continue;
+		}
+
+		/**
+		 * Feed the pipe:
+		 *   ret < 0 means error
+		 *   ret == 0 means there is more data to be fed
+		 *   ret > 0 means feeding finished
+		 */
+		ret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);
+		if (ret < 0)
+			die_errno("feed_pipe");
+
+		if (ret) {
+			close(proc->in);
+			proc->in = 0;
+		}
+	}
+}
+
 static void pp_buffer_stderr(struct parallel_processes *pp,
 			     const struct run_process_parallel_opts *opts,
 			     int output_timeout)
@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		pp->children[i].state = GIT_CP_FREE;
 		if (pp->pfd)
 			pp->pfd[i].fd = -1;
+		pp->children[i].process.in = 0;
 		child_process_init(&pp->children[i].process);
 
 		if (opts->ungroup) {
@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,
 	return result;
 }
 
+static void pp_handle_child_IO(struct parallel_processes *pp,
+				const struct run_process_parallel_opts *opts,
+				int output_timeout)
+{
+	/*
+	 * First push input, if any (it might no-op), to child tasks to avoid them blocking
+	 * after input. This also prevents deadlocks when ungrouping below, if a child blocks
+	 * while the parent also waits for them to finish.
+	 */
+	pp_buffer_stdin(pp, opts);
+
+	if (opts->ungroup) {
+		for (size_t i = 0; i < opts->processes; i++)
+			if (child_is_ready_for_cleanup(&pp->children[i]))
+				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
+	} else {
+		pp_buffer_stderr(pp, opts, output_timeout);
+		pp_output(pp);
+	}
+}
+
 void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	/*
+	 * Child tasks might receive input via stdin, terminating early (or not), so
+	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
+	 * actually writes the data to children stdin fds.
+	 */
+	sigchain_push(SIGPIPE, SIG_IGN);
+
 	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (i = 0;
@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts->ungroup) {
-			for (size_t i = 0; i < opts->processes; i++)
-				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
-		} else {
-			pp_buffer_stderr(&pp, opts, output_timeout);
-			pp_output(&pp);
-		}
+		pp_handle_child_IO(&pp, opts, output_timeout);
 		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 
 	pp_cleanup(&pp, opts);
 
+	sigchain_pop(SIGPIPE);
+
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
diff --git a/run-command.h b/run-command.h
index 0df25e445f..e1ca965b5b 100644
--- a/run-command.h
+++ b/run-command.h
@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * This callback is repeatedly called on every child process who requests
+ * start_command() to create a pipe by setting child_process.in < 0.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel, and
+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.
+ *
+ * Returns < 0 for error
+ * Returns == 0 when there is more data to be fed (will be called again)
+ * Returns > 0 when finished (child closed fd or no more data to be fed)
+ */
+typedef int (*feed_pipe_fn)(int child_in,
+				void *pp_cb,
+				void *pp_task_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -473,6 +488,12 @@ struct run_process_parallel_opts
 	 */
 	start_failure_fn start_failure;
 
+	/*
+	 * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any
+	 * special handling.
+	 */
+	feed_pipe_fn feed_pipe;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 3719f23cc2..4a56456894 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -23,19 +23,26 @@ static int number_callbacks;
 static int parallel_next(struct child_process *cp,
 			 struct strbuf *err,
 			 void *cb,
-			 void **task_cb UNUSED)
+			 void **task_cb)
 {
 	struct child_process *d = cb;
 	if (number_callbacks >= 4)
 		return 0;
 
 	strvec_pushv(&cp->args, d->args.v);
+	cp->in = d->in;
+	cp->no_stdin = d->no_stdin;
 	if (err)
 		strbuf_addstr(err, "preloaded output of a child\n");
 	else
 		fprintf(stderr, "preloaded output of a child\n");
 
 	number_callbacks++;
+
+	/* test_stdin callback will use this to count remaining lines */
+	*task_cb = xmalloc(sizeof(int));
+	*(int*)(*task_cb) = 2;
+
 	return 1;
 }
 
@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
-			 void *pp_task_cb UNUSED)
+			 void *pp_task_cb)
 {
 	if (err)
 		strbuf_addstr(err, "asking for a quick stop\n");
 	else
 		fprintf(stderr, "asking for a quick stop\n");
+
+	FREE_AND_NULL(pp_task_cb);
+
 	return 1;
 }
 
+static int task_finished_quiet(int result UNUSED,
+			       struct strbuf *err UNUSED,
+			       void *pp_cb UNUSED,
+			       void *pp_task_cb)
+{
+	FREE_AND_NULL(pp_task_cb);
+	return 0;
+}
+
+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)
+{
+	int *lines_remaining = task_cb;
+
+	if (*lines_remaining) {
+		struct strbuf buf = STRBUF_INIT;
+		strbuf_addf(&buf, "sample stdin %d\n", --(*lines_remaining));
+		if (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {
+			if (errno == EPIPE) {
+				/* child closed stdin, nothing more to do */
+				strbuf_release(&buf);
+				return 1;
+			}
+			die_errno("write");
+		}
+		strbuf_release(&buf);
+	}
+
+	return !(*lines_remaining);
+}
+
 struct testsuite {
 	struct string_list tests, failed;
 	int next;
@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)
 	struct run_process_parallel_opts opts = {
 		.get_next_task = next_test,
 		.start_failure = test_failed,
+		.feed_pipe = test_stdin_pipe_feed,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
 		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		opts.get_next_task = no_job;
 		opts.task_finished = task_finished;
+	} else if (!strcmp(argv[1], "run-command-stdin")) {
+		proc.in = -1;
+		proc.no_stdin = 0;
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished_quiet;
+		opts.feed_pipe = test_stdin_pipe_feed;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 76d4936a87..2f77fde0d9 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command listens to stdin' '
+	cat >expect <<-\EOF &&
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	preloaded output of a child
+	listening for stdin:
+	sample stdin 1
+	sample stdin 0
+	EOF
+
+	write_script stdin-script <<-\EOF &&
+	echo "listening for stdin:"
+	while read line
+	do
+		echo "$line"
+	done
+	EOF
+	test-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&
+	test_cmp expect actual
+'
+
 cat >expect <<-EOF
 preloaded output of a child
 asking for a quick stop
-- 
2.51.2

[PATCH v6 03/11] hook: provide stdin via callback

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This adds a callback mechanism for feeding stdin to hooks alongside
the existing path_to_stdin (which slurps a file's content to stdin).

The advantage of this new callback is that it can feed stdin without
going through the FS layer. This helps when feeding large amount of
data and uses the run-command parallel stdin callback introduced in
the preceding commit.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 hook.c | 23 ++++++++++++++++++++++-
 hook.h | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/hook.c b/hook.c
index b3de1048bf..5ddd7678d1 100644
--- a/hook.c
+++ b/hook.c
@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)
 static int pick_next_hook(struct child_process *cp,
 			  struct strbuf *out UNUSED,
 			  void *pp_cb,
-			  void **pp_task_cb UNUSED)
+			  void **pp_task_cb)
 {
 	struct hook_cb_data *hook_cb = pp_cb;
 	const char *hook_path = hook_cb->hook_path;
@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,
 
 	cp->no_stdin = 1;
 	strvec_pushv(&cp->env, hook_cb->options->env.v);
+
+	if (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	/* reopen the file for stdin; run_command closes it. */
 	if (hook_cb->options->path_to_stdin) {
 		cp->no_stdin = 0;
 		cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);
 	}
+
+	if (hook_cb->options->feed_pipe) {
+		cp->no_stdin = 0;
+		/* start_command() will allocate a pipe / stdin fd for us */
+		cp->in = -1;
+	}
+
 	cp->stdout_to_stderr = 1;
 	cp->trace2_hook_name = hook_cb->hook_name;
 	cp->dir = hook_cb->options->dir;
@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,
 	strvec_push(&cp->args, hook_path);
 	strvec_pushv(&cp->args, hook_cb->options->args.v);
 
+	/*
+	 * Provide per-hook internal state via task_cb for easy access, so
+	 * hook callbacks don't have to go through hook_cb->options.
+	 */
+	*pp_task_cb = hook_cb->options->feed_pipe_cb_data;
+
 	/*
 	 * This pick_next_hook() will be called again, we're only
 	 * running one hook, so indicate that no more work will be
@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
+		.feed_pipe = options->feed_pipe,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
 
+	if (options->path_to_stdin && options->feed_pipe)
+		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
+
 	if (options->invoked_hook)
 		*options->invoked_hook = 0;
 
diff --git a/hook.h b/hook.h
index 11863fa734..2169d4a6bd 100644
--- a/hook.h
+++ b/hook.h
@@ -1,6 +1,7 @@
 #ifndef HOOK_H
 #define HOOK_H
 #include "strvec.h"
+#include "run-command.h"
 
 struct repository;
 
@@ -37,6 +38,43 @@ struct run_hooks_opt
 	 * Path to file which should be piped to stdin for each hook.
 	 */
 	const char *path_to_stdin;
+
+	/**
+	 * Callback used to incrementally feed a child hook stdin pipe.
+	 *
+	 * Useful especially if a hook consumes large quantities of data
+	 * (e.g. a list of all refs in a client push), so feeding it via
+	 * in-memory strings or slurping to/from files is inefficient.
+	 * While the callback allows piecemeal writing, it can also be
+	 * used for smaller inputs, where it gets called only once.
+	 *
+	 * Add hook callback initalization context to `feed_pipe_ctx`.
+	 * Add hook callback internal state to `feed_pipe_cb_data`.
+	 *
+	 */
+	feed_pipe_fn feed_pipe;
+
+	/**
+	 * Opaque data pointer used to pass context to `feed_pipe_fn`.
+	 *
+	 * It can be accessed via the second callback arg 'pp_cb':
+	 * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_ctx;
+
+	/**
+	 * Opaque data pointer used to keep internal state across callback calls.
+	 *
+	 * It can be accessed directly via the third callback arg 'pp_task_cb':
+	 * struct ... *state = pp_task_cb;
+	 *
+	 * The caller is responsible for managing the memory for this data.
+	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
+	 */
+	void *feed_pipe_cb_data;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.2

[PATCH v6 04/11] hook: convert 'post-rewrite' hook in sequencer.c to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Replace the custom run-command calls used by post-rewrite with
the newer and simpler hook_run_opt(), which does not need to
create a custom 'struct child_process' or call find_hook().

Another benefit of using the hook API is that hook_run_opt()
handles the SIGPIPE toggle logic.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 sequencer.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 5476d39ba9..71ed31c774 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,
 	return ret;
 }
 
+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)
+{
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;
+	int ret;
+
+	if (!to_pipe)
+		BUG("pipe_from_strbuf called without feed_pipe_ctx");
+
+	ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
+
+	return 1; /* done writing */
+}
+
 static int run_rewrite_hook(const struct object_id *oldoid,
 			    const struct object_id *newoid)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 	int code;
 	struct strbuf sb = STRBUF_INIT;
-	const char *hook_path = find_hook(the_repository, "post-rewrite");
 
-	if (!hook_path)
-		return 0;
+	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
 
-	strvec_pushl(&proc.args, hook_path, "amend", NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "post-rewrite";
+	opt.feed_pipe_ctx = &sb;
+	opt.feed_pipe = pipe_from_strbuf;
+
+	strvec_push(&opt.args, "amend");
+
+	code = run_hooks_opt(the_repository, "post-rewrite", &opt);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
-	strbuf_addf(&sb, "%s %s\n", oid_to_hex(oldoid), oid_to_hex(newoid));
-	sigchain_push(SIGPIPE, SIG_IGN);
-	write_in_full(proc.in, sb.buf, sb.len);
-	close(proc.in);
 	strbuf_release(&sb);
-	sigchain_pop(SIGPIPE);
-	return finish_command(&proc);
+	return code;
 }
 
 void commit_post_rewrite(struct repository *r,
-- 
2.51.2

[PATCH v6 05/11] transport: convert pre-push to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Move the pre-push hook from custom run-command invocations to
the new hook API which doesn't require a custom child_process
structure and signal toggling.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 transport.c | 89 +++++++++++++++++++++++++++--------------------------
 1 file changed, 45 insertions(+), 44 deletions(-)

diff --git a/transport.c b/transport.c
index c7f06a7382..6d0f02be5d 100644
--- a/transport.c
+++ b/transport.c
@@ -1316,65 +1316,66 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)
 	die(_("Aborting."));
 }
 
-static int run_pre_push_hook(struct transport *transport,
-			     struct ref *remote_refs)
-{
-	int ret = 0, x;
-	struct ref *r;
-	struct child_process proc = CHILD_PROCESS_INIT;
+struct feed_pre_push_hook_data {
 	struct strbuf buf;
-	const char *hook_path = find_hook(the_repository, "pre-push");
+	const struct ref *refs;
+};
 
-	if (!hook_path)
-		return 0;
+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
+{
+	struct feed_pre_push_hook_data *data = pp_task_cb;
+	const struct ref *r = data->refs;
+	int ret = 0;
 
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, transport->remote->name);
-	strvec_push(&proc.args, transport->url);
+	if (!r)
+		return 1; /* no more refs */
 
-	proc.in = -1;
-	proc.trace2_hook_name = "pre-push";
+	data->refs = r->next;
 
-	if (start_command(&proc)) {
-		finish_command(&proc);
-		return -1;
+	switch (r->status) {
+	case REF_STATUS_REJECT_NONFASTFORWARD:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_UPTODATE:
+		return 0; /* skip refs which won't be pushed */
+	default:
+		break;
 	}
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	if (!r->peer_ref)
+		return 0;
 
-	strbuf_init(&buf, 256);
+	strbuf_reset(&data->buf);
+	strbuf_addf(&data->buf, "%s %s %s %s\n",
+		    r->peer_ref->name, oid_to_hex(&r->new_oid),
+		    r->name, oid_to_hex(&r->old_oid));
 
-	for (r = remote_refs; r; r = r->next) {
-		if (!r->peer_ref) continue;
-		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
-		if (r->status == REF_STATUS_REJECT_STALE) continue;
-		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
-		if (r->status == REF_STATUS_UPTODATE) continue;
+	ret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);
+	if (ret < 0 && errno != EPIPE)
+		return ret; /* We do not mind if a hook does not read all refs. */
 
-		strbuf_reset(&buf);
-		strbuf_addf( &buf, "%s %s %s %s\n",
-			 r->peer_ref->name, oid_to_hex(&r->new_oid),
-			 r->name, oid_to_hex(&r->old_oid));
+	return 0;
+}
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			/* We do not mind if a hook does not read all refs. */
-			if (errno != EPIPE)
-				ret = -1;
-			break;
-		}
-	}
+static int run_pre_push_hook(struct transport *transport,
+			     struct ref *remote_refs)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct feed_pre_push_hook_data data;
+	int ret = 0;
+
+	strvec_push(&opt.args, transport->remote->name);
+	strvec_push(&opt.args, transport->url);
 
-	strbuf_release(&buf);
+	strbuf_init(&data.buf, 0);
+	data.refs = remote_refs;
 
-	x = close(proc.in);
-	if (!ret)
-		ret = x;
+	opt.feed_pipe = pre_push_hook_feed_stdin;
+	opt.feed_pipe_cb_data = &data;
 
-	sigchain_pop(SIGPIPE);
+	ret = run_hooks_opt(the_repository, "pre-push", &opt);
 
-	x = finish_command(&proc);
-	if (!ret)
-		ret = x;
+	strbuf_release(&data.buf);
 
 	return ret;
 }
-- 
2.51.2

[PATCH v6 06/11] reference-transaction: use hook API instead of run-command

[Adrian Ratiu]

Convert the reference-transaction hook to the new hook API,
so it doesn't need to set up a struct child_process, call
find_hook or toggle the pipe signals.

The stdin feed callback is processing one ref update per
call. I haven't noticed any performance degradation due
to this, however we can batch as many we want in each call,
to ensure a good pipe throughtput (i.e. the child does not
wait after stdin).

Helped-by: Emily Shaffer <nasamuffin@google.com>
Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 refs.c | 100 ++++++++++++++++++++++++++++++---------------------------
 1 file changed, 52 insertions(+), 48 deletions(-)

diff --git a/refs.c b/refs.c
index 046b695bb2..e06e0cb072 100644
--- a/refs.c
+++ b/refs.c
@@ -2422,68 +2422,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-static int run_transaction_hook(struct ref_transaction *transaction,
-				const char *state)
+struct transaction_feed_cb_data {
+	size_t index;
+	struct strbuf buf;
+};
+
+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct strbuf buf = STRBUF_INIT;
-	const char *hook;
-	int ret = 0;
+	struct hook_cb_data *hook_cb = pp_cb;
+	struct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;
+	struct transaction_feed_cb_data *feed_cb_data = pp_task_cb;
+	struct strbuf *buf = &feed_cb_data->buf;
+	struct ref_update *update;
+	size_t i = feed_cb_data->index++;
+	int ret;
 
-	hook = find_hook(transaction->ref_store->repo, "reference-transaction");
-	if (!hook)
-		return ret;
+	if (i >= transaction->nr)
+		return 1; /* No more refs to process */
 
-	strvec_pushl(&proc.args, hook, state, NULL);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = "reference-transaction";
+	update = transaction->updates[i];
 
-	ret = start_command(&proc);
-	if (ret)
-		return ret;
+	if (update->flags & REF_LOG_ONLY)
+		return 0;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+	strbuf_reset(buf);
 
-	for (size_t i = 0; i < transaction->nr; i++) {
-		struct ref_update *update = transaction->updates[i];
+	if (!(update->flags & REF_HAVE_OLD))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->old_target)
+		strbuf_addf(buf, "ref:%s ", update->old_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->old_oid));
 
-		if (update->flags & REF_LOG_ONLY)
-			continue;
+	if (!(update->flags & REF_HAVE_NEW))
+		strbuf_addf(buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
+	else if (update->new_target)
+		strbuf_addf(buf, "ref:%s ", update->new_target);
+	else
+		strbuf_addf(buf, "%s ", oid_to_hex(&update->new_oid));
 
-		strbuf_reset(&buf);
+	strbuf_addf(buf, "%s\n", update->refname);
 
-		if (!(update->flags & REF_HAVE_OLD))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->old_target)
-			strbuf_addf(&buf, "ref:%s ", update->old_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->old_oid));
+	ret = write_in_full(hook_stdin_fd, buf->buf, buf->len);
+	if (ret < 0 && errno != EPIPE)
+		return ret;
 
-		if (!(update->flags & REF_HAVE_NEW))
-			strbuf_addf(&buf, "%s ", oid_to_hex(null_oid(the_hash_algo)));
-		else if (update->new_target)
-			strbuf_addf(&buf, "ref:%s ", update->new_target);
-		else
-			strbuf_addf(&buf, "%s ", oid_to_hex(&update->new_oid));
+	return 0; /* no more input to feed */
+}
+
+static int run_transaction_hook(struct ref_transaction *transaction,
+				const char *state)
+{
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct transaction_feed_cb_data feed_ctx = { 0 };
+	int ret = 0;
 
-		strbuf_addf(&buf, "%s\n", update->refname);
+	strvec_push(&opt.args, state);
 
-		if (write_in_full(proc.in, buf.buf, buf.len) < 0) {
-			if (errno != EPIPE) {
-				/* Don't leak errno outside this API */
-				errno = 0;
-				ret = -1;
-			}
-			break;
-		}
-	}
+	opt.feed_pipe = transaction_hook_feed_stdin;
+	opt.feed_pipe_ctx = transaction;
+	opt.feed_pipe_cb_data = &feed_ctx;
 
-	close(proc.in);
-	sigchain_pop(SIGPIPE);
-	strbuf_release(&buf);
+	strbuf_init(&feed_ctx.buf, 0);
+
+	ret = run_hooks_opt(transaction->ref_store->repo, "reference-transaction", &opt);
 
-	ret |= finish_command(&proc);
+	strbuf_release(&feed_ctx.buf);
 	return ret;
 }
 
-- 
2.51.2

[PATCH v6 07/11] hook: allow overriding the ungroup option

[Adrian Ratiu]

When calling run_process_parallel() in run_hooks_opt(), the
ungroup option is currently hardcoded to .ungroup = 1.

This causes problems when ungrouping should be disabled, for
example when sideband-reading collated output from child hooks,
because sideband-reading and ungrouping are mutually exclusive.

Thus a new hook.h option is added to allow overriding.

The existing ungroup=1 behavior is preserved in the run_hooks()
API and the "hook run" command. We could modify these to take
an option if necessary, so I added two code comments there.

Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/hook.c | 6 ++++++
 commit.c       | 3 +++
 hook.c         | 5 ++++-
 hook.h         | 5 +++++
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/builtin/hook.c b/builtin/hook.c
index 7afec380d2..73e7b8c2e8 100644
--- a/builtin/hook.c
+++ b/builtin/hook.c
@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,
 	if (!argc)
 		goto usage;
 
+	/*
+	 * All current "hook run" use-cases require ungrouped child output.
+	 * If this changes, a hook run argument can be added to toggle it.
+	 */
+	opt.ungroup = 1;
+
 	/*
 	 * Having a -- for "run" when providing <hook-args> is
 	 * mandatory.
diff --git a/commit.c b/commit.c
index 709c9eed58..2527f15d9d 100644
--- a/commit.c
+++ b/commit.c
@@ -1978,6 +1978,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,
 		strvec_push(&opt.args, arg);
 	va_end(args);
 
+	/* All commit hook use-cases require ungrouping child output. */
+	opt.ungroup = 1;
+
 	opt.invoked_hook = invoked_hook;
 	return run_hooks_opt(the_repository, name, &opt);
 }
diff --git a/hook.c b/hook.c
index 5ddd7678d1..00a1e2ad22 100644
--- a/hook.c
+++ b/hook.c
@@ -153,7 +153,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.tr2_label = hook_name,
 
 		.processes = 1,
-		.ungroup = 1,
+		.ungroup = options->ungroup,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
@@ -198,6 +198,9 @@ int run_hooks(struct repository *r, const char *hook_name)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
+	/* All use-cases of this API require ungrouping. */
+	opt.ungroup = 1;
+
 	return run_hooks_opt(r, hook_name, &opt);
 }
 
diff --git a/hook.h b/hook.h
index 2169d4a6bd..78a1a44690 100644
--- a/hook.h
+++ b/hook.h
@@ -34,6 +34,11 @@ struct run_hooks_opt
 	 */
 	int *invoked_hook;
 
+	/**
+	 * Allow hooks to set run_processes_parallel() 'ungroup' behavior.
+	 */
+	unsigned int ungroup:1;
+
 	/**
 	 * Path to file which should be piped to stdin for each hook.
 	 */
-- 
2.51.2

[PATCH v6 08/11] run-command: allow capturing of collated output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some callers, for example server-side hooks which wish to relay hook
output to clients across a transport, want to capture what would
normally print to stderr and do something else with it. Allow that via a
callback.

By calling the callback regardless of whether there's output available,
we allow clients to send e.g. a keepalive if necessary.

Because we expose a strbuf, not a fd or FILE*, there's no need to create
a temporary pipe or similar - we can just skip the print to stderr and
instead hand it to the caller.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 run-command.c               | 30 ++++++++++++++++++++++--------
 run-command.h               | 17 +++++++++++++++++
 t/helper/test-run-command.c | 15 +++++++++++++++
 t/t0061-run-command.sh      |  7 +++++++
 4 files changed, 61 insertions(+), 8 deletions(-)

diff --git a/run-command.c b/run-command.c
index aaf0e4ecee..2d3c2ac55c 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1595,7 +1595,10 @@ static void pp_cleanup(struct parallel_processes *pp,
 	 * When get_next_task added messages to the buffer in its last
 	 * iteration, the buffered output is non empty.
 	 */
-	strbuf_write(&pp->buffered_output, stderr);
+	if (opts->consume_output)
+		opts->consume_output(&pp->buffered_output, opts->data);
+	else
+		strbuf_write(&pp->buffered_output, stderr);
 	strbuf_release(&pp->buffered_output);
 
 	sigchain_pop_common();
@@ -1734,13 +1737,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,
 	}
 }
 
-static void pp_output(const struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp,
+		      const struct run_process_parallel_opts *opts)
 {
 	size_t i = pp->output_owner;
 
 	if (child_is_working(&pp->children[i]) &&
 	    pp->children[i].err.len) {
-		strbuf_write(&pp->children[i].err, stderr);
+		if (opts->consume_output)
+			opts->consume_output(&pp->children[i].err, opts->data);
+		else
+			strbuf_write(&pp->children[i].err, stderr);
 		strbuf_reset(&pp->children[i].err);
 	}
 }
@@ -1788,11 +1795,15 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		} else {
 			const size_t n = opts->processes;
 
-			strbuf_write(&pp->children[i].err, stderr);
+			/* Output errors, then all other finished child processes */
+			if (opts->consume_output) {
+				opts->consume_output(&pp->children[i].err, opts->data);
+				opts->consume_output(&pp->buffered_output, opts->data);
+			} else {
+				strbuf_write(&pp->children[i].err, stderr);
+				strbuf_write(&pp->buffered_output, stderr);
+			}
 			strbuf_reset(&pp->children[i].err);
-
-			/* Output all other finished child processes */
-			strbuf_write(&pp->buffered_output, stderr);
 			strbuf_reset(&pp->buffered_output);
 
 			/*
@@ -1829,7 +1840,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,
 				pp->children[i].state = GIT_CP_WAIT_CLEANUP;
 	} else {
 		pp_buffer_stderr(pp, opts, output_timeout);
-		pp_output(pp);
+		pp_output(pp, opts);
 	}
 }
 
@@ -1852,6 +1863,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 					   "max:%"PRIuMAX,
 					   (uintmax_t)opts->processes);
 
+	if (opts->ungroup && opts->consume_output)
+		BUG("ungroup and reading output are mutualy exclusive");
+
 	/*
 	 * Child tasks might receive input via stdin, terminating early (or not), so
 	 * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which
diff --git a/run-command.h b/run-command.h
index e1ca965b5b..7093252863 100644
--- a/run-command.h
+++ b/run-command.h
@@ -435,6 +435,17 @@ typedef int (*feed_pipe_fn)(int child_in,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * If this callback is provided, output is collated into a new pipe instead
+ * of the process stderr. Then `consume_output_fn` will be called repeatedly
+ * with output contained in the `output` arg. It will also be called with an
+ * empty `output` to allow for keepalives or similar operations if necessary.
+ *
+ * pp_cb is the callback cookie as passed into run_processes_parallel.
+ * No task cookie is provided because the callback receives collated output.
+ */
+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);
+
 /**
  * This callback is called on every child process that finished processing.
  *
@@ -494,6 +505,12 @@ struct run_process_parallel_opts
 	 */
 	feed_pipe_fn feed_pipe;
 
+	/*
+	 * consume_output: see consume_output_fn() above. This can be NULL
+	 * to omit any special handling.
+	 */
+	consume_output_fn consume_output;
+
 	/**
 	 * task_finished: See task_finished_fn() above. This can be
 	 * NULL to omit any special handling.
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 4a56456894..49eace8dce 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,
 	return 0;
 }
 
+static void test_divert_output(struct strbuf *output, void *cb UNUSED)
+{
+	FILE *output_file;
+
+	output_file = fopen("./output_file", "a");
+
+	strbuf_write(output, output_file);
+	fclose(output_file);
+}
+
 static int task_finished(int result UNUSED,
 			 struct strbuf *err,
 			 void *pp_cb UNUSED,
@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)
 		.get_next_task = next_test,
 		.start_failure = test_failed,
 		.feed_pipe = test_stdin_pipe_feed,
+		.consume_output = test_divert_output,
 		.task_finished = test_finished,
 		.data = &suite,
 	};
@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)
 		opts.get_next_task = parallel_next;
 		opts.task_finished = task_finished_quiet;
 		opts.feed_pipe = test_stdin_pipe_feed;
+	} else if (!strcmp(argv[1], "run-command-divert-output")) {
+		opts.get_next_task = parallel_next;
+		opts.consume_output = test_divert_output;
+		opts.task_finished = task_finished_quiet;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 2f77fde0d9..74529e219e 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than
 	test_line_count = 4 err
 '
 
+test_expect_success 'run_command can divert output' '
+	test_when_finished rm output_file &&
+	test-tool run-command run-command-divert-output 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
+	test_must_be_empty actual &&
+	test_cmp expect output_file
+'
+
 test_expect_success 'run_command listens to stdin' '
 	cat >expect <<-\EOF &&
 	preloaded output of a child
-- 
2.51.2

[PATCH v6 09/11] hooks: allow callers to capture output

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Some server-side hooks will require capturing output to send over
sideband instead of printing directly to stderr. Expose that capability.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c | 1 +
 hook.h | 8 ++++++++
 2 files changed, 9 insertions(+)

diff --git a/hook.c b/hook.c
index 00a1e2ad22..35211e5ed7 100644
--- a/hook.c
+++ b/hook.c
@@ -158,6 +158,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
 		.feed_pipe = options->feed_pipe,
+		.consume_output = options->consume_output,
 		.task_finished = notify_hook_finished,
 
 		.data = &cb_data,
diff --git a/hook.h b/hook.h
index 78a1a44690..ae502178b9 100644
--- a/hook.h
+++ b/hook.h
@@ -80,6 +80,14 @@ struct run_hooks_opt
 	 * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.
 	 */
 	void *feed_pipe_cb_data;
+
+	/*
+	 * Populate this to capture output and prevent it from being printed to
+	 * stderr. This will be passed directly through to
+	 * run_command:run_parallel_processes(). See t/helper/test-run-command.c
+	 * for an example.
+	 */
+	consume_output_fn consume_output;
 };
 
 #define RUN_HOOKS_OPT_INIT { \
-- 
2.51.2

[PATCH v6 10/11] receive-pack: convert update hooks to new API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

Use the new hook sideband API introduced in the previous commit.

The hook API avoids creating a custom struct child_process and other
internal hook plumbing (e.g. calling find_hook()) and prepares for
the specification of hooks via configs or running parallel hooks.

Execution is still sequential through the current hook.[ch] via the
run_process_parallel_opts.processes=1 arg.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 64 +++++++++++++++++-------------------------
 1 file changed, 25 insertions(+), 39 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 9c49174616..d1c40a768d 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -918,6 +918,16 @@ static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
 	return 0;
 }
 
+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
+{
+	if (!output)
+		BUG("output must be non-NULL");
+
+	/* buffer might be empty for keepalives */
+	if (output->len)
+		send_sideband(1, 2, output->buf, output->len, use_sideband);
+}
+
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
@@ -941,29 +951,18 @@ static int run_receive_hook(struct command *commands,
 
 static int run_update_hook(struct command *cmd)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	int code;
-	const char *hook_path = find_hook(the_repository, "update");
-
-	if (!hook_path)
-		return 0;
-
-	strvec_push(&proc.args, hook_path);
-	strvec_push(&proc.args, cmd->ref_name);
-	strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));
-	strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "update";
+	strvec_pushl(&opt.args,
+		     cmd->ref_name,
+		     oid_to_hex(&cmd->old_oid),
+		     oid_to_hex(&cmd->new_oid),
+		     NULL);
 
-	code = start_command(&proc);
-	if (code)
-		return code;
 	if (use_sideband)
-		copy_to_sideband(proc.err, -1, NULL);
-	return finish_command(&proc);
+		opt.consume_output = hook_output_to_sideband;
+
+	return run_hooks_opt(the_repository, "update", &opt);
 }
 
 static struct command *find_command_by_refname(struct command *list,
@@ -1640,33 +1639,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 static void run_update_post_hook(struct command *commands)
 {
 	struct command *cmd;
-	struct child_process proc = CHILD_PROCESS_INIT;
-	const char *hook;
-
-	hook = find_hook(the_repository, "post-update");
-	if (!hook)
-		return;
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		if (cmd->error_string || cmd->did_not_exist)
 			continue;
-		if (!proc.args.nr)
-			strvec_push(&proc.args, hook);
-		strvec_push(&proc.args, cmd->ref_name);
+		strvec_push(&opt.args, cmd->ref_name);
 	}
-	if (!proc.args.nr)
+	if (!opt.args.nr)
 		return;
 
-	proc.no_stdin = 1;
-	proc.stdout_to_stderr = 1;
-	proc.err = use_sideband ? -1 : 0;
-	proc.trace2_hook_name = "post-update";
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
 
-	if (!start_command(&proc)) {
-		if (use_sideband)
-			copy_to_sideband(proc.err, -1, NULL);
-		finish_command(&proc);
-	}
+	run_hooks_opt(the_repository, "post-update", &opt);
 }
 
 static void check_aliased_update_internal(struct command *cmd,
-- 
2.51.2

[PATCH v6 11/11] receive-pack: convert receive hooks to hook API

[Adrian Ratiu]

From: Emily Shaffer <emilyshaffer@google.com>

This converts the last remaining hooks to the new hook API, for
the same benefits as the previous conversions (no need to toggle
signals, manage custom struct child_process, call find_hook(),
prepares for specifyinig hooks via configs, etc.).

I noticed a performance degradation when processing large amounts
of hook input with just 1 line per callback, due to run-command's
poll loop, therefore I batched 500 lines per callback, to ensure
similar pipe throughput as before and to avoid hook child waiting
on stdin.

Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
---
 builtin/receive-pack.c | 212 ++++++++++++++++++-----------------------
 1 file changed, 93 insertions(+), 119 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index d1c40a768d..ef1f77be8c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)
 	return retval;
 }
 
-static void prepare_push_cert_sha1(struct child_process *proc)
+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)
 {
 	static int already_done;
 
@@ -775,23 +775,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)
 		nonce_status = check_nonce(sigcheck.payload);
 	}
 	if (!is_null_oid(&push_cert_oid)) {
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT=%s",
 			     oid_to_hex(&push_cert_oid));
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_SIGNER=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_SIGNER=%s",
 			     sigcheck.signer ? sigcheck.signer : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_KEY=%s",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_KEY=%s",
 			     sigcheck.key ? sigcheck.key : "");
-		strvec_pushf(&proc->env, "GIT_PUSH_CERT_STATUS=%c",
+		strvec_pushf(&opt->env, "GIT_PUSH_CERT_STATUS=%c",
 			     sigcheck.result);
 		if (push_cert_nonce) {
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE=%s",
 				     push_cert_nonce);
-			strvec_pushf(&proc->env,
+			strvec_pushf(&opt->env,
 				     "GIT_PUSH_CERT_NONCE_STATUS=%s",
 				     nonce_status);
 			if (nonce_status == NONCE_SLOP)
-				strvec_pushf(&proc->env,
+				strvec_pushf(&opt->env,
 					     "GIT_PUSH_CERT_NONCE_SLOP=%ld",
 					     nonce_stamp_slop);
 		}
@@ -803,119 +803,64 @@ struct receive_hook_feed_state {
 	struct ref_push_report *report;
 	int skip_broken;
 	struct strbuf buf;
-	const struct string_list *push_options;
 };
 
-typedef int (*feed_fn)(void *, const char **, size_t *);
-static int run_and_feed_hook(const char *hook_name, feed_fn feed,
-			     struct receive_hook_feed_state *feed_state)
+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)
 {
-	struct child_process proc = CHILD_PROCESS_INIT;
-	struct async muxer;
-	int code;
-	const char *hook_path = find_hook(the_repository, hook_name);
+	struct receive_hook_feed_state *state = pp_task_cb;
+	struct command *cmd = state->cmd;
+	unsigned int lines_batch_size = 500;
 
-	if (!hook_path)
-		return 0;
+	strbuf_reset(&state->buf);
 
-	strvec_push(&proc.args, hook_path);
-	proc.in = -1;
-	proc.stdout_to_stderr = 1;
-	proc.trace2_hook_name = hook_name;
-
-	if (feed_state->push_options) {
-		size_t i;
-		for (i = 0; i < feed_state->push_options->nr; i++)
-			strvec_pushf(&proc.env,
-				     "GIT_PUSH_OPTION_%"PRIuMAX"=%s",
-				     (uintmax_t)i,
-				     feed_state->push_options->items[i].string);
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
-			     (uintmax_t)feed_state->push_options->nr);
-	} else
-		strvec_pushf(&proc.env, "GIT_PUSH_OPTION_COUNT");
+	/* batch lines to avoid going through run-command's poll loop for each line */
+	for (unsigned int i = 0; i < lines_batch_size; i++) {
+		while (cmd &&
+		       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
+			cmd = cmd->next;
 
-	if (tmp_objdir)
-		strvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));
+		if (!cmd)
+			break;  /* no more commands left */
 
-	if (use_sideband) {
-		memset(&muxer, 0, sizeof(muxer));
-		muxer.proc = copy_to_sideband;
-		muxer.in = -1;
-		code = start_async(&muxer);
-		if (code)
-			return code;
-		proc.err = muxer.in;
-	}
+		if (!state->report)
+			state->report = cmd->report;
 
-	prepare_push_cert_sha1(&proc);
+		if (state->report) {
+			struct object_id *old_oid;
+			struct object_id *new_oid;
+			const char *ref_name;
 
-	code = start_command(&proc);
-	if (code) {
-		if (use_sideband)
-			finish_async(&muxer);
-		return code;
-	}
+			old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
+			new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
+			ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
 
-	sigchain_push(SIGPIPE, SIG_IGN);
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(old_oid), oid_to_hex(new_oid),
+				    ref_name);
 
-	while (1) {
-		const char *buf;
-		size_t n;
-		if (feed(feed_state, &buf, &n))
-			break;
-		if (write_in_full(proc.in, buf, n) < 0)
-			break;
+			state->report = state->report->next;
+			if (!state->report)
+				cmd = cmd->next;
+		} else {
+			strbuf_addf(&state->buf, "%s %s %s\n",
+				    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
+				    cmd->ref_name);
+			cmd = cmd->next;
+		}
 	}
-	close(proc.in);
-	if (use_sideband)
-		finish_async(&muxer);
 
-	sigchain_pop(SIGPIPE);
+	state->cmd = cmd;
 
-	return finish_command(&proc);
-}
-
-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)
-{
-	struct receive_hook_feed_state *state = state_;
-	struct command *cmd = state->cmd;
-
-	while (cmd &&
-	       state->skip_broken && (cmd->error_string || cmd->did_not_exist))
-		cmd = cmd->next;
-	if (!cmd)
-		return -1; /* EOF */
-	if (!bufp)
-		return 0; /* OK, can feed something. */
-	strbuf_reset(&state->buf);
-	if (!state->report)
-		state->report = cmd->report;
-	if (state->report) {
-		struct object_id *old_oid;
-		struct object_id *new_oid;
-		const char *ref_name;
-
-		old_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;
-		new_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;
-		ref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(old_oid), oid_to_hex(new_oid),
-			    ref_name);
-		state->report = state->report->next;
-		if (!state->report)
-			state->cmd = cmd->next;
-	} else {
-		strbuf_addf(&state->buf, "%s %s %s\n",
-			    oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),
-			    cmd->ref_name);
-		state->cmd = cmd->next;
-	}
-	if (bufp) {
-		*bufp = state->buf.buf;
-		*sizep = state->buf.len;
+	if (state->buf.len > 0) {
+		int ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);
+		if (ret < 0) {
+			if (errno == EPIPE)
+				return 1; /* child closed pipe */
+			return ret;
+		}
 	}
-	return 0;
+
+	return state->cmd ? 0 : 1;  /* 0 = more to come, 1 = EOF */
 }
 
 static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)
@@ -933,20 +878,49 @@ static int run_receive_hook(struct command *commands,
 			    int skip_broken,
 			    const struct string_list *push_options)
 {
-	struct receive_hook_feed_state state;
-	int status;
-
-	strbuf_init(&state.buf, 0);
-	state.cmd = commands;
-	state.skip_broken = skip_broken;
-	state.report = NULL;
-	if (feed_receive_hook(&state, NULL, NULL))
+	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+	struct command *iter = commands;
+	struct receive_hook_feed_state feed_state;
+	int ret;
+
+	/* if there are no valid commands, don't invoke the hook at all. */
+	while (iter && skip_broken && (iter->error_string || iter->did_not_exist))
+		iter = iter->next;
+	if (!iter)
 		return 0;
-	state.cmd = commands;
-	state.push_options = push_options;
-	status = run_and_feed_hook(hook_name, feed_receive_hook, &state);
-	strbuf_release(&state.buf);
-	return status;
+
+	if (push_options) {
+		for (int i = 0; i < push_options->nr; i++)
+			strvec_pushf(&opt.env, "GIT_PUSH_OPTION_%d=%s", i,
+				     push_options->items[i].string);
+		strvec_pushf(&opt.env, "GIT_PUSH_OPTION_COUNT=%"PRIuMAX"",
+					     (uintmax_t)push_options->nr);
+	} else {
+		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+	}
+
+	if (tmp_objdir)
+		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+
+	prepare_push_cert_sha1(&opt);
+
+	/* set up sideband printer */
+	if (use_sideband)
+		opt.consume_output = hook_output_to_sideband;
+
+	/* set up stdin callback */
+	feed_state.cmd = commands;
+	feed_state.skip_broken = skip_broken;
+	feed_state.report = NULL;
+	strbuf_init(&feed_state.buf, 0);
+	opt.feed_pipe_cb_data = &feed_state;
+	opt.feed_pipe = feed_receive_hook_cb;
+
+	ret = run_hooks_opt(the_repository, hook_name, &opt);
+
+	strbuf_release(&feed_state.buf);
+
+	return ret;
 }
 
 static int run_update_hook(struct command *cmd)
-- 
2.51.2

Re: [PATCH v6 00/11] Convert remaining hooks to hook.h

[Junio C Hamano]

Adrian Ratiu <adrian.ratiu@collabora.com> writes:

> Hello everyone,
>
> This series finishes the hook.[ch] conversion for the remaining hooks in
> preparation for adding config-based hooks and enabling parallel hook
> execution where possible (that will be a separate series from this one).
>
> v6 is minor refresh for some last nits. Details + range-diff below.

I didn't see anything iffy in the series.  Will replace.

Should we mark this for 'next'?