git gc taking forever to reflog expire --all

git gc taking forever to reflog expire --all

[Geert Bosch]

On my recent MacBook Pro, I gave up after reflog expire --all
had chewed up 30 minutes of perfectly fine CPU time.

Basically, I have a SVN import of a single branch of a repository
with just a single directory with 3K files or so (this SVN import
is part of a bigger repository). There are about 117K commits
in the GIT history. I have a few local branches of just a couple
commits each. The total .git size is 450MB, with about 3K loose
objects.

This seems something that should be "git gc"-ed in about a
minute or less, so I'm puzzled what is going on in this
git reflog expire. From a cursory look it seems that
we're in a loop finding a merge base for each commit, or
similar. I'm including a log of a random interruption
(using Ubuntu here instead of MacOS for better debugging).

  -Geert

#0  0x080b1c8c in clear_commit_marks (commit=0x87b2d60, mark=983040)
    at commit.c:403
#1  0x080b2d44 in get_merge_bases_many (one=0x90a9120, n=1, twos=0xbffff344, 
    cleanup=1) at commit.c:655
#2  0x080b2e67 in get_merge_bases (one=0x90a9120, two=0x819f720, cleanup=1)
    at commit.c:704
#3  0x080b2e9b in in_merge_bases (commit=0x90a9120, reference=0xbffff51c, 
    num=1) at commit.c:728
#4  0x08092ffa in unreachable (cb=0xbffff514, commit=0x5, 
    sha1=0xbffff484 "\264\252\244\232\320=Q\213\243Ҋ\323\070%[T\232\360,\032\214=\226\340\001\253\335tZ ", <incomplete sequence \303>) at builtin/reflog.c:233
#5  0x080930f5 in expire_reflog_ent (
    osha1=0xbffff498 "\232\360,\032\214=\226\340\001\253\335tZ ", <incomplete sequence \303>, 
    nsha1=0xbffff484 "\264\252\244\232\320=Q\213\243Ҋ\323\070%[T\232\360,\032\214=\226\340\001\253\335tZ ", <incomplete sequence \303>, 
    email=0x81c0ada "Geert Bosch <bosch@gnat.com>", timestamp=1265908541, 
    tz=0, message=0x81c0b08 "r4741\n", cb_data=0xbffff514)
    at builtin/reflog.c:298
#6  0x080e2c1f in for_each_recent_reflog_ent (
    ref=0x818066c "refs/remotes/git-svn@130115", 
    fn=0x8093030 <expire_reflog_ent>, ofs=0, cb_data=0xbffff514) at refs.c:1623
#7  0x080e2d27 in for_each_reflog_ent (
    ref=0x818066c "refs/remotes/git-svn@130115", 
    fn=0x8093030 <expire_reflog_ent>, cb_data=0xbffff514) at refs.c:1635
#8  0x080924d7 in expire_reflog (ref=0x818066c "refs/remotes/git-svn@130115", 
    sha1=0x819f720 "\005", unused=<value optimized out>, cb_data=0xbffff594)
    at builtin/reflog.c:353
#9  0x08092ab6 in cmd_reflog_expire (argc=2, argv=0xbffffb2c, prefix=0x0)
    at builtin/reflog.c:599
#10 0x0804b374 in run_builtin (argc=3, argv=0xbffffb28) at git.c:260
#11 handle_internal_command (argc=3, argv=0xbffffb28) at git.c:416
#12 0x0804b572 in run_argv (argc=3, argv=0xbffffb28) at git.c:458
#13 main (argc=3, argv=0xbffffb28) at git.c:529

Re: git gc taking forever to reflog expire --all

[Jeff King]

On Mon, May 10, 2010 at 01:58:11PM -0400, Geert Bosch wrote:

> On my recent MacBook Pro, I gave up after reflog expire --all
> had chewed up 30 minutes of perfectly fine CPU time.

There are some corner cases that can cause the expire-unreachable
algorithm to behave badly. The fix is b4ca1db (reflog
--expire-unreachable: avoid merge-base computation, 2010-04-07), but
hasn't been included in a release yet.

In the meantime, you can work around it with:

  git config gc.reflogExpireUnreachable never

-Peff

Re: git gc taking forever to reflog expire --all

[Geert Bosch]

On May 11, 2010, at 01:01, Jeff King wrote:

> On Mon, May 10, 2010 at 01:58:11PM -0400, Geert Bosch wrote:
> 
>> On my recent MacBook Pro, I gave up after reflog expire --all
>> had chewed up 30 minutes of perfectly fine CPU time.
> 
> There are some corner cases that can cause the expire-unreachable
> algorithm to behave badly. The fix is b4ca1db (reflog
> --expire-unreachable: avoid merge-base computation, 2010-04-07), but
> hasn't been included in a release yet.
> 
I tried with v1.7.1 + cherry-pick b4ca1db968eb57d9cd869337bffa254e0b2c83bd,
but that didn't solve the issue.
> In the meantime, you can work around it with:
> 
>  git config gc.reflogExpireUnreachable never

I had tried that too, but I guess I'll just have to be very patient:
22707 ttys001   11:14.27 git reflog expire --all

What would be the easiest way for me to repack my repository
without running reflog expire at all?

  -Geert

Re: git gc taking forever to reflog expire --all

[Jeff King]

On Tue, May 11, 2010 at 06:05:14AM -0400, Geert Bosch wrote:

> > There are some corner cases that can cause the expire-unreachable
> > algorithm to behave badly. The fix is b4ca1db (reflog
> > --expire-unreachable: avoid merge-base computation, 2010-04-07), but
> > hasn't been included in a release yet.
> > 
> I tried with v1.7.1 + cherry-pick b4ca1db968eb57d9cd869337bffa254e0b2c83bd,
> but that didn't solve the issue.
> > In the meantime, you can work around it with:
> > 
> >  git config gc.reflogExpireUnreachable never
> 
> I had tried that too, but I guess I'll just have to be very patient:
> 22707 ttys001   11:14.27 git reflog expire --all

Hmm. If that didn't solve it, then presumably it is a different problem,
and I'd like to take a closer look. Is it possible for you to make a
tarball of the .git directory available?

> What would be the easiest way for me to repack my repository
> without running reflog expire at all?

Try "git repack -Ad".

-Peff

Re: git gc taking forever to reflog expire --all

[Jeff King]

[for those reading the list, Geert was kind enough to send
 me a copy off his repo off-list. Here's what I found.]

On Tue, May 11, 2010 at 06:05:14AM -0400, Geert Bosch wrote:

> > There are some corner cases that can cause the expire-unreachable
> > algorithm to behave badly. The fix is b4ca1db (reflog
> > --expire-unreachable: avoid merge-base computation, 2010-04-07), but
> > hasn't been included in a release yet.
> > 
> I tried with v1.7.1 + cherry-pick b4ca1db968eb57d9cd869337bffa254e0b2c83bd,
> but that didn't solve the issue.

Are you absolutely sure you weren't accidentally running an older
version of git? I was able to reproduce your problem with git v1.7.0,
but current 'master' fixes it. I bisected and ended up at b4ca1db.

There is the possibility that cherry-picking b4ca1db isn't sufficient
for some reason, but I didn't see any related commits that would make a
difference. In either case, your problem is just another manifestation
of the same issue, so it should be taken care of in v1.7.2.

> > In the meantime, you can work around it with:
> > 
> >  git config gc.reflogExpireUnreachable never
> 
> I had tried that too, but I guess I'll just have to be very patient:
> 22707 ttys001   11:14.27 git reflog expire --all

Sorry, this was my fault. Apparently the "never" token had a bug which
was fixed in v1.7.0.3. If the git you were testing on was before that,
then it would have done nothing.

-Peff