Re: [PATCHv4 06/14] run-command: add an asynchronous parallel child processor

[Stefan Beller <sbeller@google.com>]

On Wed, Sep 23, 2015 at 12:34 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Junio C Hamano <gitster@pobox.com> writes:
>
>> You are running "git fetch" that are is a lot more heavy-weight.
>> Because once each of them started fully they will be network bound,
>> it is likely that you would want to run more processes than you have
>> core.
>
> I thought the conclusion would be obvious, but just in case the
> readers need the flow of thought completed, from the above it
> follows that on a N-core box (say 8-core) you may want to run the
> fetch with -j16 (or more).  If we start everything at once, the time
> before one process starts to produce first meaningful response (I am
> not counting the "starting command for submodule-$i" message as
> "meaningful") would take twice as long under such condition even if
> you have infinite network bandwidth and talking to an infinitely
> fast server.
>
> I agree that a full slow-start ramping-up is not necessary in order
> to hide the start-up lag.  All you need to do is to start just one
> and make sure it becomes the foreground, and give it a time alone to
> make enough progress to produce an early output without getting
> slowed down by system activities caused by the other background
> processes in the group that start all at the same time.  And once
> the foreground process starts showing its output, you can unleash
> the remainder of the herd to thunder and nobody would notice as they
> run all in the background.  Again, I am assuming something like "git
> fetch" that takes reasonably large amount of resource and time to
> start-up and tear-down.
>

I can confirm this now.

    git fetch --recurse-submodules=yes -j 400

in an submodule-ified Android tree takes very long to start putting out useful
information, but if I hardcode the SPAWN_CAP to 4 it looks pretty amazing
fast.

I'd guess we could have a reasonable default for SPAWN_CAP by

    #define SPAWN_CAP min(pp.max_processes + 1, online_cpus)

>