'git gc --aggressive' effectively unusable

'git gc --aggressive' effectively unusable

[Frans Pop]

Note: this is on a different repo from the 'git reflog expire --all' I
reported a bit earlier.

I have a git-svn checkout of a subversion repo which I wanted to compress
as much as possible. 'git gc --aggressive' starts to run fairly well, but
eats more and more memory and gets slower and slower. After it gets to
about 45% or 50% progress slows down noticeably and so far I haven't had
the patience to let it finish (40 minutes is already way too long).

A regular 'git gc' run completes without any problems.

$ du -sh .git/
612M    .git/

Special about this repo is that it contains two huge objects [1], which
could maybe be a factor:
     size    pack  SHA
- packages/po/sublevel4/da.po:
     495661  4654  801cd6451ece536c0ab41f79e09fc52efdf3361f
- packages/arch/powerpc/quik-installer/debian/po/da.po
     149515  1403  83a787b20817dc4d72db052de4055e7a7c9221d7  

Below some output from top and of the progress of the command showing the
problem. Check the change in number of compressed objects against the
timestamps from top.

Cheers,
FJP

[1] Caused by a bug in a script a couple of years back.

$ git gc --aggressive

Counting objects: 843342, done.
Delta compression using up to 2 threads.
Compressing objects:  53% (449663/836424)

top - 22:55:02 up 18 min,  1 user,  load average: 1.83, 1.68, 1.07
Tasks: 161 total,   1 running, 160 sleeping,   0 stopped,   0 zombie
Cpu0  : 91.4%us,  0.7%sy,  0.0%ni,  1.3%id,  6.6%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 97.7%us,  0.3%sy,  0.0%ni,  1.3%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2034284k total,  2018288k used,    15996k free,    10188k buffers
Swap:  2097148k total,    22612k used,  2074536k free,   449444k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5861 fjp       20   0 1775m 1.3g 194m S  188 66.7  21:10.89 git


Counting objects: 843342, done.
Delta compression using up to 2 threads.
Compressing objects:  58% (486001/836424)

top - 23:00:12 up 23 min,  1 user,  load average: 1.96, 1.84, 1.30
Tasks: 158 total,   2 running, 156 sleeping,   0 stopped,   0 zombie
Cpu0  : 98.3%us,  0.7%sy,  0.0%ni,  0.7%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 87.4%us,  1.7%sy,  0.0%ni,  0.0%id, 10.6%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   2034284k total,  2017516k used,    16768k free,     4696k buffers
Swap:  2097148k total,    22572k used,  2074576k free,   336944k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5861 fjp       20   0 1903m 1.4g 172m S  182 71.4  30:37.58 git


Counting objects: 843342, done.
Delta compression using up to 2 threads.
Compressing objects:  61% (515958/836424)

top - 23:05:56 up 29 min,  1 user,  load average: 1.68, 1.85, 1.48
Tasks: 159 total,   1 running, 158 sleeping,   0 stopped,   0 zombie
Cpu0  : 86.7%us,  1.7%sy,  0.0%ni,  2.0%id,  9.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 96.7%us,  0.0%sy,  0.0%ni,  0.7%id,  2.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2034284k total,  2018644k used,    15640k free,     2748k buffers
Swap:  2097148k total,    24312k used,  2072836k free,   343256k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5861 fjp       20   0 1903m 1.4g 189m S  176 72.3  40:29.50 git

Re: 'git gc --aggressive' effectively unusable

[Frans Pop]

The environment is the same as for the reflog problem, but I should have 
added that info anyway. Here it is again.

On Saturday 03 April 2010, Frans Pop wrote:
> I have a git-svn checkout of a subversion repo which I wanted to
> compress as much as possible. 'git gc --aggressive' starts to run fairly
> well, but eats more and more memory and gets slower and slower. After it
> gets to about 45% or 50% progress slows down noticeably and so far I
> haven't had the patience to let it finish (40 minutes is already way too
> long).

I'm seeing this with both git 1.6.6.1 and 1.7.0.3 on the same repo.
Environment:
- Debian amd64/Lenny; Core Duo x86_64 2.6.34-rc3 -> 1.6.6.1
- Debian i386/Sid; chroot on the same machine -> 1.7.0.3

Re: 'git gc --aggressive' effectively unusable

[Frans Pop]

On Saturday 03 April 2010, Frans Pop wrote:
> Special about this repo is that it contains two huge objects [1], which
> could maybe be a factor:
>      size    pack  SHA
> - packages/po/sublevel4/da.po:
>      495661  4654  801cd6451ece536c0ab41f79e09fc52efdf3361f
> - packages/arch/powerpc/quik-installer/debian/po/da.po
>      149515  1403  83a787b20817dc4d72db052de4055e7a7c9221d7  

To avoid confusion: these sizes are in kB.

Re: 'git gc --aggressive' effectively unusable

[Michael Witten]

On Fri, Apr 2, 2010 at 16:05, Frans Pop <elendil@planet.nl> wrote:
> I haven't had the patience to let it finish

There's your problem.

$ git help gc | sed -n /--aggressive$/,+3p
       --aggressive
           Usually git gc runs very quickly while
           providing good disk space utilization
           and performance. This option will
           cause git gc to more aggressively
           optimize the repository at the expense
           of taking much more time. The effects
           of this optimization are persistent, so
           this option only needs to be used
           occasionally; every few hundred
           changesets or so.

Last time I used this option (on Linus's Linux repo), I let the
algorithm do its thing for a couple of hours. Maybe the efficiency
could be vastly improved, but it does finish if you let it.

SIncerely,
Michael Witten

Re: 'git gc --aggressive' effectively unusable

[Michael Witten]

On Sat, Apr 3, 2010 at 15:33, Michael Witten <mfwitten@gmail.com> wrote:
> $ git help gc | sed -n /--aggressive$/,+3p

As an aside: I didn't realize I copied that in there; this would
probably be better:

$ git help gc | sed -n /--aggressive$/,/^$/p

Re: 'git gc --aggressive' effectively unusable

[Frans Pop]

On Saturday 03 April 2010, Michael Witten wrote:
> On Fri, Apr 2, 2010 at 16:05, Frans Pop <elendil@planet.nl> wrote:
> > I haven't had the patience to let it finish
>
> There's your problem.

Yes, I had seen that. But there's a difference between taking much more 
time and slowing down to such an extend that it never finishes.

I've tried it today on my linux-2.6 repo as well and the same thing 
happened. At first the progress is not fast but reasonable. When it gets 
to about 45% percent it starts slowing down a lot: from ~1500 objects per 
update of the counters to ~300 objects per update. And who knows what the 
progress is going to be when it reaches 70% or 90%: 10 per update?

With a total of over 2 milion objects in the repository such a low speed is 
simply not going to work, ever. So I maintain that it is effectively 
unusable.

Cheers,
FJP

Re: 'git gc --aggressive' effectively unusable

[Michael Witten]

On Sat, Apr 3, 2010 at 17:23, Frans Pop <elendil@planet.nl> wrote:
> On Saturday 03 April 2010, Michael Witten wrote:
>> On Fri, Apr 2, 2010 at 16:05, Frans Pop <elendil@planet.nl> wrote:
>> > I haven't had the patience to let it finish
> ...
> I've tried it today on my linux-2.6 repo as well and the same thing
> happened. At first the progress is not fast but reasonable. When it gets
> to about 45% percent it starts slowing down a lot: from ~1500 objects per
> update of the counters to ~300 objects per update. And who knows what
> the progress is going to be when it reaches 70% or 90%: 10 per update?
>
> With a total of over 2 milion objects in the repository such a low speed is
> simply not going to work, ever. So I maintain that it is effectively
> unusable.

Well, all I can do is quote myself:

    Last time I used this option (on Linus's Linux repo),
    I let the algorithm do its thing for a couple of hours.
    Maybe the efficiency could be vastly improved, but
    it does finish if you let it.

On Fri, Apr 2, 2010 at 16:12, Frans Pop <elendil@planet.nl> wrote:
> I'm seeing this with both git 1.6.6.1 and 1.7.0.3 on the same repo.

I think I must have run gc with 1.7.0.2.199.g90a2bf9; perhaps you
could use something like oprofile to figure out where gc is spending
most of its time.

Re: 'git gc --aggressive' effectively unusable

[Miles Bader]

Frans Pop <elendil@planet.nl> writes:
>> > I haven't had the patience to let it finish
>>
>> There's your problem.
>
> Yes, I had seen that. But there's a difference between taking much more 
> time and slowing down to such an extend that it never finishes.
>
> I've tried it today on my linux-2.6 repo as well and the same thing 
> happened. At first the progress is not fast but reasonable. When it gets 
> to about 45% percent it starts slowing down a lot: from ~1500 objects per 
> update of the counters to ~300 objects per update. And who knows what the 
> progress is going to be when it reaches 70% or 90%: 10 per update?

Are you sure it doesn't subsequently speed up again?

-Miles

-- 
Idiot, n. A member of a large and powerful tribe whose influence in human
affairs has always been dominant and controlling.

Re: 'git gc --aggressive' effectively unusable

[Michael Poole]

Miles Bader writes:

> Frans Pop <elendil@planet.nl> writes:
>>> > I haven't had the patience to let it finish
>>>
>>> There's your problem.
>>
>> Yes, I had seen that. But there's a difference between taking much more 
>> time and slowing down to such an extend that it never finishes.
>>
>> I've tried it today on my linux-2.6 repo as well and the same thing 
>> happened. At first the progress is not fast but reasonable. When it gets 
>> to about 45% percent it starts slowing down a lot: from ~1500 objects per 
>> update of the counters to ~300 objects per update. And who knows what the 
>> progress is going to be when it reaches 70% or 90%: 10 per update?
>
> Are you sure it doesn't subsequently speed up again?

I have seen asymptotic slowdown as "git gc --aggressive" progresses on
certain repositories.  It is particularly bad with
git://git.infradead.org/gcc.git (on an x86-64 system with 4 GB RAM).
git seemed to be thrashing swap badly as time went on.  I don't know
that git gc --aggressive would *never* finish on my gcc-git repository.
I just know that it got to about 80% done in less than an hour, to 90%
after twelve hours, and about 94% after another twelve hours.  (The same
operation on linux-2.6.git takes about 40 minutes with all the default
settings.)

I may have been dreaming, but I thought with some 1.6.x version of git,
reducing core.packedGitLimit and pack.windowLimit (now windowMemory?)
mostly made the thrashing go away.  When I try again with v1.7.0.2,
though, it doesn't seem to help very much -- there is still a lot of
swapping, and the git process got to about 7 GB virtual size before I
killed it after about 10 hours of operation.

Michael Poole

Re: 'git gc --aggressive' effectively unusable

[Jeff King]

On Sun, Apr 04, 2010 at 10:50:47AM -0400, Michael Poole wrote:

> > Are you sure it doesn't subsequently speed up again?
> 
> I have seen asymptotic slowdown as "git gc --aggressive" progresses on
> certain repositories.  It is particularly bad with
> git://git.infradead.org/gcc.git (on an x86-64 system with 4 GB RAM).
> git seemed to be thrashing swap badly as time went on.  I don't know
> that git gc --aggressive would *never* finish on my gcc-git repository.
> I just know that it got to about 80% done in less than an hour, to 90%
> after twelve hours, and about 94% after another twelve hours.  (The same
> operation on linux-2.6.git takes about 40 minutes with all the default
> settings.)
> 
> I may have been dreaming, but I thought with some 1.6.x version of git,
> reducing core.packedGitLimit and pack.windowLimit (now windowMemory?)
> mostly made the thrashing go away.  When I try again with v1.7.0.2,
> though, it doesn't seem to help very much -- there is still a lot of
> swapping, and the git process got to about 7 GB virtual size before I
> killed it after about 10 hours of operation.

I packed Frans' sample kernel repo with "git gc --aggressive" last
night. It did finish after about 9 hours. I didn't take memory usage
measurements, but here's what time said:

  real    535m38.898s
  user    216m46.437s
  sys     0m24.186s

That's 3.6 hours of CPU time over almost 9 hours (on a dual-core
machine). The non-agressive pack was about 680M, and the result was
480M. The machine has 2G of RAM, and not much else running. So I would
really not expect there to be much disk I/O required, but clearly we
were waiting quite a bit.

I'll try tweaking a few of the pack memory limits and try again.

-Peff

Re: 'git gc --aggressive' effectively unusable

[Jeff King]

On Sun, Apr 04, 2010 at 04:38:50PM -0400, Jeff King wrote:

> I packed Frans' sample kernel repo with "git gc --aggressive" last
> night. It did finish after about 9 hours. I didn't take memory usage
> measurements, but here's what time said:
> 
>   real    535m38.898s
>   user    216m46.437s
>   sys     0m24.186s
> 
> That's 3.6 hours of CPU time over almost 9 hours (on a dual-core
> machine). The non-agressive pack was about 680M, and the result was
> 480M. The machine has 2G of RAM, and not much else running. So I would
> really not expect there to be much disk I/O required, but clearly we
> were waiting quite a bit.
> 
> I'll try tweaking a few of the pack memory limits and try again.

Hmm, this may be relevant:

  http://thread.gmane.org/gmane.comp.version-control.git/67791/focus=94797

In my experiments, memory usage is increasing but valgrind doesn't
leaks. So perhaps it is fragmentation in the memory allocator.

-Peff

Re: 'git gc --aggressive' effectively unusable

[Nicolas Pitre]

On Sun, 4 Apr 2010, Jeff King wrote:

> On Sun, Apr 04, 2010 at 04:38:50PM -0400, Jeff King wrote:
> 
> > I packed Frans' sample kernel repo with "git gc --aggressive" last
> > night. It did finish after about 9 hours. I didn't take memory usage
> > measurements, but here's what time said:
> > 
> >   real    535m38.898s
> >   user    216m46.437s
> >   sys     0m24.186s
> > 
> > That's 3.6 hours of CPU time over almost 9 hours (on a dual-core
> > machine). The non-agressive pack was about 680M, and the result was
> > 480M. The machine has 2G of RAM, and not much else running. So I would
> > really not expect there to be much disk I/O required, but clearly we
> > were waiting quite a bit.
> > 
> > I'll try tweaking a few of the pack memory limits and try again.
> 
> Hmm, this may be relevant:
> 
>   http://thread.gmane.org/gmane.comp.version-control.git/67791/focus=94797
> 
> In my experiments, memory usage is increasing but valgrind doesn't
> leaks. So perhaps it is fragmentation in the memory allocator.

To verify this, simply try with pack.threads = 1.  That should help the 
memory allocator not to fragment memory allocation across threads 
randomly.

Also, going multithreaded _may_ be faster only if you can afford the 
increased memory usage.  Especially with gc --aggressive, each thread is 
adding its own share of memory usage in the delta window.

First thing to try for the biggest possible improvement is 
pack.threads=1.  On a quad core machine this means repacking 4 times 
slower, but this is certainly much faster than 100 times slower when the 
system starts swapping. That might even make the resulting pack a tad 
tighter due to delta windows not being fragmented across different 
threads.

If that is not enough, then try:

	pack.deltaCacheSize = 1
	core.packedGitWindowSize = 16m
	core.packedGitLimit = 128m

This should reduce Git's memory usage while making it slower without 
affecting the packing outcome.  Again "slower" could mean "much faster" 
if by reducing memory usage then swapping is completely avoided.

If that still doesn't help much, then the next tweaks will affect the 
packing result:

	pack.windowMemory = 256m

Here 256m is arbitrary and must be guessed from the size of the objects 
being packed.  The idea is to let smallish objects completely fill the 
search window (it has 250 entries by default with --aggressive) while 
not letting that many huge objects completely eat up all memory.  If 
there is still swapping going on then you can try 64m instead.  That 
means that if you have a large set of 1MB objects then the delta search 
window will be scaled down to less than 64 entries in that case.  This 
is why packing might be less optimal as there are fewer delta 
combinations being considered.

If this still doesn't prevent swapping then you should really consider 
installing more RAM.  There are fundamental object accounting structures 
that can hardly be shrunk such as struct object_entry in 
builtin/pack-objects.c, and one instance of such structure is needed for 
each object.  On a 64-bit machine this structure occupies 120 bytes, 
meaning 2M objects requires 240MB of RAM just for that.  The data set 
also has to fit in the file cache to avoid IO trashing.  So if your 
repository is larger than the available RAM then some trashing is almost 
unavoidable.  Sometimes a badly packed repository may require 2GB of 
disk space in the .git directory alone while the fully packed version is 
only a few hundred megabytes.  Such repositories may need to be repacked 
on a big machine first, before machines with less RAM are able to handle 
it afterwards.

Hope this helps.


Nicolas

Re: 'git gc --aggressive' effectively unusable

[Mike Galbraith]

On Sun, 2010-04-04 at 01:23 +0200, Frans Pop wrote:
> On Saturday 03 April 2010, Michael Witten wrote:
> > On Fri, Apr 2, 2010 at 16:05, Frans Pop <elendil@planet.nl> wrote:
> > > I haven't had the patience to let it finish
> >
> > There's your problem.
> 
> Yes, I had seen that. But there's a difference between taking much more 
> time and slowing down to such an extend that it never finishes.
> 
> I've tried it today on my linux-2.6 repo as well and the same thing 
> happened. At first the progress is not fast but reasonable. When it gets 
> to about 45% percent it starts slowing down a lot: from ~1500 objects per 
> update of the counters to ~300 objects per update. And who knows what the 
> progress is going to be when it reaches 70% or 90%: 10 per update?
> 
> With a total of over 2 milion objects in the repository such a low speed is 
> simply not going to work, ever. So I maintain that it is effectively 
> unusable.

As a data point, when I do gc, I routinely use --aggressive.  It takes a
while here, but not forever.  (I'm a tad short of 2 million objects)

Repo is mainline + next + tip + stable >= 2.6.22 + local branches.

git@marge:..git/linux-2.6> time git gc --aggressive
Counting objects: 1909894, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (1889774/1889774), done.
Writing objects: 100% (1909894/1909894), done.
Total 1909894 (delta 1674098), reused 0 (delta 0)

real    22m24.943s
user    55m33.756s
sys     0m8.149s

git is 1.7.0.3

	-Mike