Re: Resolving deltas dominates clone time

[Martin Fick <mfick@codeaurora.org>]

On Tuesday, April 30, 2019 2:02:32 PM MDT Jeff King wrote:
> On Tue, Apr 23, 2019 at 02:09:31PM -0600, Martin Fick wrote:
> > I think that if there were no default limit during a clone it could have
> > disastrous effects on people using the repo tool from the android project,
> > or any other "submodule like" tool that might clone many projects in
> > parallel. With the repo tool, people often use a large -j number such as
> > 24 which means they end up cloning around 24 projects at a time, and they
> > may do this for around 1000 projects. If git clone suddenly started as
> > many threads as there are CPUs for each clone, this would likely paralyze
> > the machine.
> 
> IMHO this is already a problem, because none of those 24 gits knows
> about the others. So they're already using 24*3 cores, though of course
> at any given moment some of those 24 may be bottle-necked on the
> network.

I think this is very different. In the first case this is a linear constraint 
that someone can use to intelligently adjust the number of clones they are 
doing at the same time. The second case cannot be accounted for in any 
intelligent way by the person running the clones, it is almost unconstrained.

...
> > I do suspect it would be nice to have a switch though that repo could use
> > to adjust this intelligently, is there some way to adjust threads from a
> > clone, I don't see one? I tried using 'GIT_FORCE_THREADS=28 git clone
> > ...' and it didn't seem to make a difference?
> 
> I think I led you astray earlier by mentioning GIT_FORCE_THREADS. It's
> actually just a boolean for "use threads even if we're only
> single-threaded". What you actually want is probably:
> 
>   git clone -c pack.threads=28 ...
> 
> (though I didn't test it to be sure).

Thanks, I will test this!

-Martin

-- 
The Qualcomm Innovation Center, Inc. is a member of Code 
Aurora Forum, hosted by The Linux Foundation