Re: Resolving deltas dominates clone time

[Martin Fick <mfick@codeaurora.org>]

On Tuesday, April 23, 2019 5:08:40 PM MDT Duy Nguyen wrote:
> On Tue, Apr 23, 2019 at 11:45 AM Jeff King <peff@peff.net> wrote:
> > On Mon, Apr 22, 2019 at 09:55:38PM -0400, Jeff King wrote:
> > > Here are my p5302 numbers on linux.git, by the way.
> > > 
> > >   Test                                           jk/p5302-repeat-fix
> > >   ------------------------------------------------------------------
> > >   5302.2: index-pack 0 threads                   307.04(303.74+3.30)
> > >   5302.3: index-pack 1 thread                    309.74(306.13+3.56)
> > >   5302.4: index-pack 2 threads                   177.89(313.73+3.60)
> > >   5302.5: index-pack 4 threads                   117.14(344.07+4.29)
> > >   5302.6: index-pack 8 threads                   112.40(607.12+5.80)
> > >   5302.7: index-pack default number of threads   135.00(322.03+3.74)
> > > 
> > > which still imply that "4" is a win over "3" ("8" is slightly better
> > > still in wall-clock time, but the total CPU rises dramatically; that's
> > > probably because this is a quad-core with hyperthreading, so by that
> > > point we're just throttling down the CPUs).
> > 
> > And here's a similar test run on a 20-core Xeon w/ hyperthreading (I
> > tweaked the test to keep going after eight threads):
> > 
> > Test                            HEAD
> > ----------------------------------------------------
> > 5302.2: index-pack 1 threads    376.88(364.50+11.52)
> > 5302.3: index-pack 2 threads    228.13(371.21+17.86)
> > 5302.4: index-pack 4 threads    151.41(387.06+21.12)
> > 5302.5: index-pack 8 threads    113.68(413.40+25.80)
> > 5302.6: index-pack 16 threads   100.60(511.85+37.53)
> > 5302.7: index-pack 32 threads   94.43(623.82+45.70)
> > 5302.8: index-pack 40 threads   93.64(702.88+47.61)
> > 
> > I don't think any of this is _particularly_ relevant to your case, but
> > it really seems to me that the default of capping at 3 threads is too
> > low.

Here are my index-pack results (I only ran them once since they take a while) 
using vgit 1.8.3.2:

Threads  real       user        sys
1     108m46.151s 106m14.420s  1m57.192s
2     58m14.274s  106m23.158s  5m32.736s
3     40m33.351s  106m42.281s  5m40.884s
4     31m40.342s  107m20.278s  5m40.675s
5     26m0.454s   106m54.370s  5m35.827s
12    13m25.304s  107m57.271s  6m26.493s
16    10m56.866s  107m46.107s  6m41.330s
18    10m18.112s  109m50.893s  7m1.369s
20    9m54.010s   113m51.028s  7m53.082s
24    9m1.104s    115m8.245s   7m57.156s
28    8m26.058s   116m46.311s  8m34.752s
32    8m42.967s   140m33.280s  9m59.514s
36    8m52.228s   151m28.939s  11m55.590s
40    8m22.719s   153m4.496s   12m36.041s
44    8m12.419s   166m41.594s  14m7.717s
48    8m0.377s    172m3.597s   16m32.041s
56    8m22.320s   188m31.426s  17m48.274s

 
> Looking back at the multithread commit, I think the trend was the same
> and I capped it because the gain was not proportional to the number of
> cores we threw at index-pack anymore. I would not be opposed to
> raising the cap though (or maybe just remove it)

I think that if there were no default limit during a clone it could have 
disastrous effects on people using the repo tool from the android project, or 
any other "submodule like" tool that might clone many projects in parallel. 
With the repo tool, people often use a large -j number such as 24 which means 
they end up cloning around 24 projects at a time, and they may do this for 
around 1000 projects. If git clone suddenly started as many threads as there 
are CPUs for each clone, this would likely paralyze the machine.

I do suspect it would be nice to have a switch though that repo could use to 
adjust this intelligently, is there some way to adjust threads from a clone, I 
don't see one? I tried using 'GIT_FORCE_THREADS=28 git clone ...' and it 
didn't seem to make a difference?

-Martin

-- 
The Qualcomm Innovation Center, Inc. is a member of Code 
Aurora Forum, hosted by The Linux Foundation