All of lore.kernel.org
 help / color / mirror / Atom feed
From: Patrick Steinhardt <ps@pks.im>
To: Adrian Ratiu <adrian.ratiu@collabora.com>
Cc: git@vger.kernel.org, "Jeff King" <peff@peff.net>,
	"Emily Shaffer" <emilyshaffer@google.com>,
	"Junio C Hamano" <gitster@pobox.com>,
	"Josh Steadmon" <steadmon@google.com>,
	"Kristoffer Haugsbakk" <kristofferhaugsbakk@fastmail.com>,
	"brian m . carlson" <sandals@crustytoothpaste.net>,
	"Emily Shaffer" <nasamuffin@google.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: Re: [PATCH v4 3/9] hook: allow parallel hook execution
Date: Tue, 24 Mar 2026 10:07:47 +0100	[thread overview]
Message-ID: <acJUY9N4oplsTgFY@pks.im> (raw)
In-Reply-To: <20260320135311.331463-4-adrian.ratiu@collabora.com>

On Fri, Mar 20, 2026 at 03:53:05PM +0200, Adrian Ratiu wrote:
> From: Emily Shaffer <nasamuffin@google.com>
> 
> Hooks always run in sequential order due to the hardcoded jobs == 1
> passed to run_process_parallel(). Remove that hardcoding to allow
> users to run hooks in parallel (opt-in).
> 
> Users need to decide which hooks to run in parallel, by specifying
> "parallel = true" in the config, because git cannot know if their

s/git/Git/

Sorry to be pedantic :)

> diff --git a/Documentation/config/hook.adoc b/Documentation/config/hook.adoc
> index b7847f9338..21800db648 100644
> --- a/Documentation/config/hook.adoc
> +++ b/Documentation/config/hook.adoc
> @@ -23,6 +23,19 @@ hook.<friendly-name>.enabled::
>  	in a system or global config file and needs to be disabled for a
>  	specific repository. See linkgit:git-hook[1].
>  
> +hook.<friendly-name>.parallel::
> +	Whether the hook `hook.<friendly-name>` may run in parallel with other hooks
> +	for the same event. Defaults to `false`. Set to `true` only when the
> +	hook script is safe to run concurrently with other hooks for the same
> +	event. If any hook for an event does not have this set to `true`,
> +	all hooks for that event run sequentially regardless of `hook.jobs`.
> +	Only configured (named) hooks need to declare this. Traditional hooks
> +	found in the hooks directory do not need to, and run in parallel when
> +	the effective job count is greater than 1. See linkgit:git-hook[1].

Thanks for adding this setting, this addresses my most important concern
with this patch series.

>  hook.jobs::
>  	Specifies how many hooks can be run simultaneously during parallelized
>  	hook execution. If unspecified, defaults to 1 (serial execution).
> ++
> +This setting has no effect unless all configured hooks for the event have
> +`hook.<friendly-name>.parallel` set to `true`.

I guess that's a fair constraint for now. We can still iterate going
forward and have those marked as parallelizable run in parallel, while
running the others sequentially.

> diff --git a/hook.c b/hook.c
> index c4872d8707..a60dac5a60 100644
> --- a/hook.c
> +++ b/hook.c
> @@ -120,6 +120,7 @@ struct hook_config_cache_entry {
>  	char *command;
>  	enum config_scope scope;
>  	unsigned int disabled:1;
> +	unsigned int parallel:1;
>  };

I'd recommend to use a proper bool here.

> @@ -223,6 +226,10 @@ static int hook_config_lookup_all(const char *key, const char *value,
>  		default:
>  			break; /* ignore unrecognised values */
>  		}
> +	} else if (!strcmp(subkey, "parallel")) {
> +		int v = git_parse_maybe_bool(value);
> +		if (v >= 0)
> +			strmap_put(&data->parallel_hooks, hook_name, (void *)(uintptr_t)v);

Do we want to warn on unparseable values?

> @@ -541,21 +553,75 @@ static void run_hooks_opt_clear(struct run_hooks_opt *options)
>  	strvec_clear(&options->args);
>  }
>  
> +/* Determine how many jobs to use for hook execution. */
> +static unsigned int get_hook_jobs(struct repository *r,
> +				  struct run_hooks_opt *options,
> +				  struct string_list *hook_list)
> +{
> +	/*
> +	 * Hooks needing separate output streams must run sequentially.
> +	 * Next commit will allow parallelizing these as well.
> +	 */
> +	if (!options->stdout_to_stderr)
> +		return 1;
> +
> +	/*
> +	 * An explicit job count overrides everything else: this covers both
> +	 * FORCE_SERIAL callers (for hooks that must never run in parallel)
> +	 * and the -j flag from the CLI. The CLI override is intentional: users
> +	 * may want to serialize hooks declared parallel or to parallelize more
> +	 * aggressively than the default.
> +	 */
> +	if (options->jobs)
> +		return options->jobs;

Hm, okay. I feel like this behaviour is somewhat surprising given that
it now operates different compared to the config value. But arguably,
there is no reason why a caller of git-hook(1) should explicitly set
this flag as it should be under control of the user, unless they have a
very good reason to override the number of jobs.

So maybe this is fine, but I think it needs to be called out explicitly
in our docs.

> +	/*
> +	 * Use hook.jobs from the already-parsed config cache (in-repo), or
> +	 * fallback to a direct config lookup (out-of-repo).
> +	 * Default to 1 (serial execution) on failure.
> +	 */
> +	if (r && r->gitdir && r->hook_config_cache)
> +		/* Use the already-parsed cache (in-repo) */
> +		options->jobs = r->hook_jobs ? r->hook_jobs : 1;
> +	else
> +		/* No cache present (out-of-repo call), use direct cfg lookup */
> +		if (repo_config_get_uint(r, "hook.jobs", &options->jobs))
> +			options->jobs = 1;

We first have a check for `r != NULL`, but if I'm not mistaken
`repo_config_get_uint()` will cause us to unconditionally deref `r`.

> diff --git a/hook.h b/hook.h
> index 7c8c3d471e..494f74345f 100644
> --- a/hook.h
> +++ b/hook.h
> @@ -35,6 +35,13 @@ struct hook {
>  		} configured;
>  	} u;
>  
> +	/**
> +	 * Whether this hook may run in parallel with other hooks for the same
> +	 * event. Only useful for configured (named) hooks. Traditional hooks
> +	 * always default to 0 (serial). Set via `hook.<name>.parallel = true`.
> +	 */
> +	unsigned int parallel:1;
> +
>  	/**
>  	 * Opaque data pointer used to keep internal state across callback calls.
>  	 *

This should likely also be a boolean.

Patrick

  reply	other threads:[~2026-03-24  9:07 UTC|newest]

Thread overview: 113+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-02-04 17:33 [PATCH 0/4] Run hooks in parallel Adrian Ratiu
2026-02-04 17:33 ` [PATCH 1/4] config: add a repo_config_get_uint() helper Adrian Ratiu
2026-02-04 17:33 ` [PATCH 2/4] hook: allow parallel hook execution Adrian Ratiu
2026-02-11 12:41   ` Patrick Steinhardt
2026-02-12 12:25     ` Adrian Ratiu
2026-02-04 17:33 ` [PATCH 3/4] hook: introduce extensions.hookStdoutToStderr Adrian Ratiu
2026-02-04 17:33 ` [PATCH 4/4] hook: allow runtime enabling extensions.hookStdoutToStderr Adrian Ratiu
2026-02-12 10:43 ` [PATCH 0/4] Run hooks in parallel Phillip Wood
2026-02-12 14:24   ` Adrian Ratiu
2026-02-13 14:39     ` Phillip Wood
2026-02-13 17:21       ` Adrian Ratiu
2026-02-22  0:28 ` [PATCH v2 00/10] " Adrian Ratiu
2026-02-22  0:28   ` [PATCH v2 01/10] repository: fix repo_init() memleak due to missing _clear() Adrian Ratiu
2026-02-22  0:28   ` [PATCH v2 02/10] config: add a repo_config_get_uint() helper Adrian Ratiu
2026-02-22  0:28   ` [PATCH v2 03/10] hook: refactor hook_config_cache from strmap to named struct Adrian Ratiu
2026-02-22  0:28   ` [PATCH v2 04/10] hook: parse the hook.jobs config Adrian Ratiu
2026-02-22  0:28   ` [PATCH v2 05/10] hook: allow parallel hook execution Adrian Ratiu
2026-02-22  0:29   ` [PATCH v2 06/10] hook: mark non-parallelizable hooks Adrian Ratiu
2026-02-22  0:29   ` [PATCH v2 07/10] hook: add -j/--jobs option to git hook run Adrian Ratiu
2026-02-22  0:29   ` [PATCH v2 08/10] hook: add per-event jobs config Adrian Ratiu
2026-02-22  0:29   ` [PATCH v2 09/10] hook: introduce extensions.hookStdoutToStderr Adrian Ratiu
2026-02-22  0:29   ` [PATCH v2 10/10] hook: allow runtime enabling extensions.hookStdoutToStderr Adrian Ratiu
2026-03-09 13:37 ` [PATCH v3 0/9] Run hooks in parallel Adrian Ratiu
2026-03-09 13:37   ` [PATCH v3 1/9] repository: fix repo_init() memleak due to missing _clear() Adrian Ratiu
2026-03-15  4:55     ` Junio C Hamano
2026-03-15  5:05     ` Junio C Hamano
2026-03-09 13:37   ` [PATCH v3 2/9] config: add a repo_config_get_uint() helper Adrian Ratiu
2026-03-09 13:37   ` [PATCH v3 3/9] hook: parse the hook.jobs config Adrian Ratiu
2026-03-15 16:13     ` Junio C Hamano
2026-03-09 13:37   ` [PATCH v3 4/9] hook: allow parallel hook execution Adrian Ratiu
2026-03-15 20:46     ` Junio C Hamano
2026-03-18 18:02       ` Adrian Ratiu
2026-03-09 13:37   ` [PATCH v3 5/9] hook: mark non-parallelizable hooks Adrian Ratiu
2026-03-15 20:56     ` Junio C Hamano
2026-03-18 18:40       ` Adrian Ratiu
2026-03-09 13:37   ` [PATCH v3 6/9] hook: add -j/--jobs option to git hook run Adrian Ratiu
2026-03-15 21:00     ` Junio C Hamano
2026-03-18 19:00       ` Adrian Ratiu
2026-03-09 13:37   ` [PATCH v3 7/9] hook: add per-event jobs config Adrian Ratiu
2026-03-16 18:40     ` Junio C Hamano
2026-03-18 19:21       ` Adrian Ratiu
2026-03-09 13:37   ` [PATCH v3 8/9] hook: introduce extensions.hookStdoutToStderr Adrian Ratiu
2026-03-16 18:44     ` Junio C Hamano
2026-03-18 19:50       ` Adrian Ratiu
2026-03-09 13:37   ` [PATCH v3 9/9] hook: allow runtime enabling extensions.hookStdoutToStderr Adrian Ratiu
2026-03-20 13:53 ` [PATCH v4 0/9] Run hooks in parallel Adrian Ratiu
2026-03-20 13:53   ` [PATCH v4 1/9] config: add a repo_config_get_uint() helper Adrian Ratiu
2026-03-20 13:53   ` [PATCH v4 2/9] hook: parse the hook.jobs config Adrian Ratiu
2026-03-24  9:07     ` Patrick Steinhardt
2026-03-24 18:59       ` Adrian Ratiu
2026-03-20 13:53   ` [PATCH v4 3/9] hook: allow parallel hook execution Adrian Ratiu
2026-03-24  9:07     ` Patrick Steinhardt [this message]
2026-03-20 13:53   ` [PATCH v4 4/9] hook: allow pre-push parallel execution Adrian Ratiu
2026-03-20 13:53   ` [PATCH v4 5/9] hook: mark non-parallelizable hooks Adrian Ratiu
2026-03-20 13:53   ` [PATCH v4 6/9] hook: add -j/--jobs option to git hook run Adrian Ratiu
2026-03-24  9:07     ` Patrick Steinhardt
2026-03-20 13:53   ` [PATCH v4 7/9] hook: add per-event jobs config Adrian Ratiu
2026-03-24  9:08     ` Patrick Steinhardt
2026-03-20 13:53   ` [PATCH v4 8/9] hook: warn when hook.<friendly-name>.jobs is set Adrian Ratiu
2026-03-24  9:08     ` Patrick Steinhardt
2026-03-20 13:53   ` [PATCH v4 9/9] hook: add hook.<event>.enabled switch Adrian Ratiu
2026-03-24  9:08     ` Patrick Steinhardt
2026-03-25 18:43       ` Adrian Ratiu
2026-03-20 17:24   ` [PATCH v4 0/9] Run hooks in parallel Junio C Hamano
2026-03-23 15:07     ` Adrian Ratiu
2026-03-24  9:07       ` Patrick Steinhardt
2026-03-26 10:18 ` [PATCH v5 00/12] " Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 01/12] repository: fix repo_init() memleak due to missing _clear() Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 02/12] config: add a repo_config_get_uint() helper Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 03/12] hook: parse the hook.jobs config Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 04/12] hook: allow parallel hook execution Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 05/12] hook: allow pre-push parallel execution Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 06/12] hook: mark non-parallelizable hooks Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 07/12] hook: add -j/--jobs option to git hook run Adrian Ratiu
2026-03-27 14:46     ` Patrick Steinhardt
2026-03-26 10:18   ` [PATCH v5 08/12] hook: add per-event jobs config Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 09/12] hook: warn when hook.<friendly-name>.jobs is set Adrian Ratiu
2026-03-27 14:46     ` Patrick Steinhardt
2026-03-26 10:18   ` [PATCH v5 10/12] hook: move is_known_hook() to hook.c for wider use Adrian Ratiu
2026-03-27 14:46     ` Patrick Steinhardt
2026-03-27 15:59       ` Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 11/12] hook: add hook.<event>.enabled switch Adrian Ratiu
2026-03-26 10:18   ` [PATCH v5 12/12] hook: allow hook.jobs=-1 to use all available CPU cores Adrian Ratiu
2026-04-04  8:29 ` [PATCH v6 00/12] Run hooks in parallel Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 01/12] repository: fix repo_init() memleak due to missing _clear() Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 02/12] config: add a repo_config_get_uint() helper Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 03/12] hook: parse the hook.jobs config Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 04/12] hook: allow parallel hook execution Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 05/12] hook: allow pre-push parallel execution Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 06/12] hook: mark non-parallelizable hooks Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 07/12] hook: add -j/--jobs option to git hook run Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 08/12] hook: add per-event jobs config Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 09/12] hook: warn when hook.<friendly-name>.jobs is set Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 10/12] hook: move is_known_hook() to hook.c for wider use Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 11/12] hook: add hook.<event>.enabled switch Adrian Ratiu
2026-04-04  8:29   ` [PATCH v6 12/12] hook: allow hook.jobs=-1 to use all available CPU cores Adrian Ratiu
2026-04-06 16:24   ` [PATCH v6 00/12] Run hooks in parallel Junio C Hamano
2026-04-08 10:17   ` Patrick Steinhardt
2026-04-08 16:57     ` Junio C Hamano
2026-04-10  9:05 ` [PATCH v7 00/13] " Adrian Ratiu
2026-04-10  9:05   ` [PATCH v7 01/13] repository: fix repo_init() memleak due to missing _clear() Adrian Ratiu
2026-04-10  9:05   ` [PATCH v7 02/13] config: add a repo_config_get_uint() helper Adrian Ratiu
2026-04-10  9:05   ` [PATCH v7 03/13] hook: parse the hook.jobs config Adrian Ratiu
2026-04-10  9:05   ` [PATCH v7 04/13] hook: allow parallel hook execution Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 05/13] hook: allow pre-push parallel execution Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 06/13] hook: mark non-parallelizable hooks Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 07/13] hook: add -j/--jobs option to git hook run Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 08/13] hook: add per-event jobs config Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 09/13] hook: warn when hook.<friendly-name>.jobs is set Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 10/13] hook: move is_known_hook() to hook.c for wider use Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 11/13] hook: add hook.<event>.enabled switch Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 12/13] hook: allow hook.jobs=-1 to use all available CPU cores Adrian Ratiu
2026-04-10  9:06   ` [PATCH v7 13/13] t1800: test SIGPIPE with parallel hooks Adrian Ratiu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=acJUY9N4oplsTgFY@pks.im \
    --to=ps@pks.im \
    --cc=adrian.ratiu@collabora.com \
    --cc=avarab@gmail.com \
    --cc=emilyshaffer@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=kristofferhaugsbakk@fastmail.com \
    --cc=nasamuffin@google.com \
    --cc=peff@peff.net \
    --cc=sandals@crustytoothpaste.net \
    --cc=steadmon@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.