gc getting called on each git command ... what's wrong?

gc getting called on each git command ... what's wrong?

[Geoff Russell]

Hi all,

I'm running git version 1.7.0.4 on Ubuntu 10.04 LTS

As of today, almost every time I do a git command, gc is getting
invoked. This is a multi-gigabyte repository with over
half a million objects, so this takes a while ... and I'm guessing
that it shouldn't be happening anyway!

I've run an fsck (which doesn't do a gc!) and the repository looks
clean ... no output.

I have packSizeLimit set to 30M ... not sure why I did this, was
investigating something I
didn't understand.  There are 96 pack files.

Any help greatly appreciated, many thanks,

Cheers,
Geoff

Re: gc getting called on each git command ... what's wrong?

[Peter Harris]

On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
>
> As of today, almost every time I do a git command, gc is getting
> invoked.

>   There are 96 pack files.

That's why. See gc.autopacklimit in "git help config" -- by default,
git will gc if there are more than 50 pack files.

Peter Harris

Re: gc getting called on each git command ... what's wrong?

[Drew Northup]

On Tue, 2011-06-07 at 21:48 -0400, Peter Harris wrote:
> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> >
> > As of today, almost every time I do a git command, gc is getting
> > invoked.
<re-added>
> >   I have packSizeLimit set to 30M 
</re-added>
> >   There are 96 pack files.
> 
> That's why. See gc.autopacklimit in "git help config" -- by default,
> git will gc if there are more than 50 pack files.

Do we want to consider ignoring (or automatically doubling, or something
like that) gc.autopacklimit if that number of packs meet or exceed
gc.packSizeLimit? I have no idea what the patch for this might look
like, but it seems to make more sense than this situation.

Just a random brain fart...

-- 
-Drew Northup
________________________________________________
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59

Re: gc getting called on each git command ... what's wrong?

[Brandon Casey]

On 06/08/2011 11:02 AM, Drew Northup wrote:
> 
> On Tue, 2011-06-07 at 21:48 -0400, Peter Harris wrote:
>> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
>>>
>>> As of today, almost every time I do a git command, gc is getting
>>> invoked.
> <re-added>
>>>   I have packSizeLimit set to 30M 
> </re-added>
>>>   There are 96 pack files.
>>
>> That's why. See gc.autopacklimit in "git help config" -- by default,
>> git will gc if there are more than 50 pack files.
> 
> Do we want to consider ignoring (or automatically doubling, or something
> like that) gc.autopacklimit if that number of packs meet or exceed
> gc.packSizeLimit? I have no idea what the patch for this might look
> like, but it seems to make more sense than this situation.
> 
> Just a random brain fart...
> 

Or just ignore the packs that exceed pack.packSizeLimit...

diff --git a/builtin/gc.c b/builtin/gc.c
index ff5f73b..7be14ab 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -26,6 +26,7 @@ static int pack_refs = 1;
 static int aggressive_window = 250;
 static int gc_auto_threshold = 6700;
 static int gc_auto_pack_limit = 50;
+static off_t pack_size_limit;
 static const char *prune_expire = "2.weeks.ago";
 
 #define MAX_ADD 10
@@ -64,6 +65,10 @@ static int gc_config(const char *var, const char *value, void *cb)
 		}
 		return git_config_string(&prune_expire, var, value);
 	}
+	if (!strcmp(var, "pack.packsizelimit")) {
+		pack_size_limit = git_config_ulong(var, value);
+		return 0;
+	}
 	return git_default_config(var, value, cb);
 }
 
@@ -135,10 +140,8 @@ static int too_many_packs(void)
 			continue;
 		if (p->pack_keep)
 			continue;
-		/*
-		 * Perhaps check the size of the pack and count only
-		 * very small ones here?
-		 */
+		if (pack_size_limit && p->pack_size >= pack_size_limit)
+			continue;
 		cnt++;
 	}
 	return gc_auto_pack_limit <= cnt;

Re: gc getting called on each git command ... what's wrong?

[Junio C Hamano]

Drew Northup <drew.northup@maine.edu> writes:

>> That's why. See gc.autopacklimit in "git help config" -- by default,
>> git will gc if there are more than 50 pack files.
>
> Do we want to consider ignoring (or automatically doubling, or something
> like that) gc.autopacklimit if that number of packs meet or exceed
> gc.packSizeLimit? I have no idea what the patch for this might look
> like, but it seems to make more sense than this situation.

This is unrelated to the auto-gc, but it also would be fruitful to
question if it is a sane setting to limit packfiles to 30M, when the
repository needs 100 of them (total around 3G??). Just like having too
many loose object files degrade performance (and that is one of the
reasons we pack them in the first place), having many packs will degrade
performance unnecessarily and to a worse degree, as "check which pack has
this particular object" code has to examine all packs, unlike the loose
object case where we let the .git/objects/?? fan-out to give us some
hashing and the filesystem to do the heavylifting for us.

Re: gc getting called on each git command ... what's wrong?

[Jakub Narebski]

Peter Harris <git@peter.is-a-geek.org> writes:

> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> >
> > As of today, almost every time I do a git command, gc is getting
> > invoked.
> 
> >   There are 96 pack files.
> 
> That's why. See gc.autopacklimit in "git help config" -- by default,
> git will gc if there are more than 50 pack files.

Actually it looks like it is combination of this and packSizeLimit set
to 30M.  Git notices that it has too many packfiles, and tries to
repack them, but packlimit forces Git to split it into small
packfiles... and end up with more packfiles than limit anyway.

Perhaps git should notice that it has nonsensical combination of
options...
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: gc getting called on each git command ... what's wrong?

[Geoff Russell]

On Thu, Jun 9, 2011 at 2:39 AM, Jakub Narebski <jnareb@gmail.com> wrote:
>
> Peter Harris <git@peter.is-a-geek.org> writes:
>
> > On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> > >
> > > As of today, almost every time I do a git command, gc is getting
> > > invoked.
> >
> > >   There are 96 pack files.
> >
> > That's why. See gc.autopacklimit in "git help config" -- by default,
> > git will gc if there are more than 50 pack files.

Thanks to everybody. This is exactly what was happening and the problems went
away when I set the packSizeLimit higher ... 3000M

>
> Actually it looks like it is combination of this and packSizeLimit set
> to 30M.  Git notices that it has too many packfiles, and tries to
> repack them, but packlimit forces Git to split it into small
> packfiles... and end up with more packfiles than limit anyway.
>
> Perhaps git should notice that it has nonsensical combination of
> options...

That would be nice. It should be reasonably easy to work out that the
packSizeLimit
will guarantee too many pack files after the gc. Disobeying a users
wishes shouldn't
be undertaken lightly, but sometimes we stuff up :)

Cheers,
Geoff.

--
6 Fifth Ave,
St Morris, S.A. 5068
Australia
Ph: 041 8805 184 / 08 8332 5069
http://perfidy.com.au

Re: gc getting called on each git command ... what's wrong?

[Jakub Narebski]

On Wed, 15 Jun 2011, Geoff Russell wrote:
> On Thu, Jun 9, 2011 at 2:39 AM, Jakub Narebski <jnareb@gmail.com> wrote:
> > Peter Harris <git@peter.is-a-geek.org> writes:
> > > On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> > > >
> > > > As of today, almost every time I do a git command, gc is getting
> > > > invoked.
> > >
> > > >   There are 96 pack files.
> > >
> > > That's why. See gc.autopacklimit in "git help config" -- by default,
> > > git will gc if there are more than 50 pack files.
> >
> > Actually it looks like it is combination of this and packSizeLimit set
> > to 30M.  Git notices that it has too many packfiles, and tries to
> > repack them, but packlimit forces Git to split it into small
> > packfiles... and end up with more packfiles than limit anyway.
> 
> Thanks to everybody. This is exactly what was happening and the problems
> went away when I set the packSizeLimit higher ... 3000M
 
Why did you set packSizeLimit at all?

 
> >
> > Perhaps git should notice that it has nonsensical combination of
> > options...
> 
> That would be nice. It should be reasonably easy to work out that the
> packSizeLimit will guarantee too many pack files after the gc.
> Disobeying a users wishes shouldn't be undertaken lightly, but sometimes
> we stuff up :) 

Well, git can simply notice that each except perhaps on file has
size greater or equal to gc.packSizeLimit, and then ignore gc.autopacklimit
hint, because repacking would not reduce number of packs, and not lower
it below gc.autopacklimit.

If `git gc` is called interactively, we can warn user about this situation...

-- 
Jakub Narebski
Poland

Re: gc getting called on each git command ... what's wrong?

[Geoff Russell]

2011/6/16 Jakub Narebski <jnareb@gmail.com>
>
>
> Why did you set packSizeLimit at all?
>
>

Some time ago (31/8/2010) I had a problem which seemed to be caused by
large packs (>4GB), you
can find it in the git list with a subject of "Large pack causes git
clone failures ... what to do?"

Anyway, I set packSizeLimit and fiddled around for a bit ...
eventually the problem went
away when I moved the central repository to another machine with less load and
more memory. At which point I gave a sigh of relief and forgot to
remove the packSizeLimit
until recently bitten. But the original problem was probably nothing
to do with large
packs and hasn't recurred.

Cheers,
Geoff.

Re: gc getting called on each git command ... what's wrong?

[Jakub Narebski]

On Thu, 16 Jun 2011, Geoff Russell wrote:
> 2011/6/16 Jakub Narebski <jnareb@gmail.com>
> >
> > Why did you set packSizeLimit at all?
> 
> Some time ago (31/8/2010) I had a problem which seemed to be caused by
> large packs (>4GB), you can find it in the git list with a subject of
> "Large pack causes git clone failures ... what to do?"

So why did you set packSizeLimit to such ridiculous low value, instead
of 2g (2 GB) or something?

-- 
Jakub Narebski
Poland