Re: [PATCHv2 0/8] Expose the submodule parallelism to the user

[Ramsay Jones <ramsay@ramsayjones.plus.com>]

On 28/10/15 23:21, Stefan Beller wrote:
> This replaces origin/sb/submodule-parallel-update
> (anchoring at 74367d8938, Merge branch 'sb/submodule-parallel-fetch'
> into sb/submodule-parallel-update)
> 
> What does it do?
> ---
> This series should finish the on going efforts of parallelizing
> submodule network traffic. The patches contain tests for clone,
> fetch and submodule update to use the actual parallelism both via
> command line as well as a configured option. I decided to go with
> "submodule.jobs" for all three for now.
> 
> What is new in v2?
> ---
> * The patches got reordered slightly
> * Documentation was adapted
> 
> Interdiff below
> 
> Stefan Beller (8):
>   run_processes_parallel: Add output to tracing messages
>   submodule config: keep update strategy around
>   submodule config: remove name_and_item_from_var
>   submodule-config: parse_config
>   fetching submodules: Respect `submodule.jobs` config option
>   git submodule update: have a dedicated helper for cloning
>   submodule update: expose parallelism to the user
>   clone: allow an explicit argument for parallel submodule clones
> 
>  Documentation/config.txt        |   7 ++
>  Documentation/git-clone.txt     |   6 +-
>  Documentation/git-submodule.txt |   7 +-
>  builtin/clone.c                 |  23 +++-
>  builtin/fetch.c                 |   2 +-
>  builtin/submodule--helper.c     | 244 ++++++++++++++++++++++++++++++++++++++++
>  git-submodule.sh                |  54 ++++-----
>  run-command.c                   |   4 +
>  submodule-config.c              |  98 ++++++++++------
>  submodule-config.h              |   3 +
>  submodule.c                     |   5 +
>  t/t5526-fetch-submodules.sh     |  14 +++
>  t/t7400-submodule-basic.sh      |   4 +-
>  t/t7406-submodule-update.sh     |  27 +++++
>  14 files changed, 418 insertions(+), 80 deletions(-)
> 
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 0de0138..785721a 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -2643,12 +2643,12 @@ submodule.<name>.ignore::
>  	"--ignore-submodules" option. The 'git submodule' commands are not
>  	affected by this setting.
>  
> -submodule::jobs
> +submodule.jobs::
>  	This is used to determine how many submodules can be operated on in
>  	parallel. Specifying a positive integer allows up to that number
> -	of submodules being fetched in parallel. Specifying 0 the number
> -	of cpus will be taken as the maximum number. Currently this is
> -	used in fetch and clone operations only.
> +	of submodules being fetched in parallel. This is used in fetch
> +	and clone operations only. A value of 0 will give some reasonable
> +	default. The defaults may change with different versions of Git.
>  
>  tag.sort::
>  	This variable controls the sort ordering of tags when displayed by
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index affa52e..01bd6b7 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -216,9 +216,10 @@ objects from the source repository into a pack in the cloned repository.
>  	The result is Git repository can be separated from working
>  	tree.
>  
> --j::
> ---jobs::
> +-j <n>::
> +--jobs <n>::
>  	The number of submodules fetched at the same time.
> +	Defaults to the `submodule.jobs` option.

Hmm, is there a way to _not_ fetch in parallel (override the
config) from the command line for a given command?

ATB,
Ramsay Jones

>  
>  <repository>::
>  	The (possibly remote) repository to clone from.  See the
> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
> index f5429fa..c70fafd 100644
> --- a/Documentation/git-submodule.txt
> +++ b/Documentation/git-submodule.txt
> @@ -374,10 +374,11 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.
>  	clone with a history truncated to the specified number of revisions.
>  	See linkgit:git-clone[1]
>  
> --j::
> ---jobs::
> +-j <n>::
> +--jobs <n>::
>  	This option is only valid for the update command.
>  	Clone new submodules in parallel with as many jobs.
> +	Defaults to the `submodule.jobs` option.
>  
>  <path>...::
>  	Paths to submodule(s). When specified this will restrict the command
> diff --git a/builtin/clone.c b/builtin/clone.c
> index 5ac2d89..22b9924 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -727,10 +727,7 @@ static int checkout(void)
>  		struct argv_array args = ARGV_ARRAY_INIT;
>  		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>  
> -		if (max_jobs == -1)
> -			if (git_config_get_int("submodule.jobs", &max_jobs))
> -				max_jobs = 1;
> -		if (max_jobs != 1) {
> +		if (max_jobs != -1) {
>  			struct strbuf sb = STRBUF_INIT;
>  			strbuf_addf(&sb, "--jobs=%d", max_jobs);
>  			argv_array_push(&args, sb.buf);
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index c3d438a..67dba1c 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -476,9 +476,10 @@ static int update_clone(int argc, const char **argv, const char *prefix)
>  	/* Overlay the parsed .gitmodules file with .git/config */
>  	git_config(git_submodule_config, NULL);
>  
> -	if (max_jobs == -1)
> -		if (git_config_get_int("submodule.jobs", &max_jobs))
> -			max_jobs = 1;
> +	if (max_jobs < 0)
> +		max_jobs = config_parallel_submodules();
> +	if (max_jobs < 0)
> +		max_jobs = 1;
>  
>  	run_processes_parallel(max_jobs,
>  			       update_clone_get_next_task,
>