Re: git-reflog 70 minutes at 100% cpu and counting

[Eric Paris <eparis@redhat.com>]

On Wed, 2009-12-16 at 16:06 -0500, Nicolas Pitre wrote:
> On Wed, 16 Dec 2009, Eric Paris wrote:
> 
> > On Tue, 2009-12-15 at 22:03 -0500, Nicolas Pitre wrote:
> > > On Mon, 14 Dec 2009, Eric Paris wrote:
> > > 
> > > > The alternative repo is slowing pushing up to that same location.  That
> > > > tar is 855838982, so just a tad bit smaller.
> > > 
> > > It doesn't appear to be complete yet, and not progressing either.
> > 
> > The alternative repo is now available (but the original is down)
> > 
> > I tried to run git gc --aggressive last night while I slept and got this
> > as output, maybe it helps point to a solution/problem?  The git reflog
> > portion ran for 5 hours and 36 minutes and appears to have finished.
> 
> Yes.  I was able to reproduce your issue.  And because of the *horrible* 
> repository packing, the reflog expiration process is taking ages when 
> determining object reachability at a rate of one reflog entry every 2 
> seconds or so.  With 4214 entries for the fsnotify-syscall branch, and 
> 1352 entries for the fsnotify branch, this already takes up asignificant 
> portion of the actual run time.  I'm sure if your repository was 
> properly packed this would take less than a minute.

I'm guessing this is a result of stgit.?  These branches really should
be just a branch from a tag (which exists in kernel-1) and about 30-50
patches linearly applied on top.  I don't know how I get that many
objects.  I'm guessing many/most of them are crap that should be able to
be cleaned/deleted entirely as the rebasing/pushing/poping/updating that
stgit does under the covers should have rendered them pointless.  Not
really sure when/how that should/could have happened.

Should I be running git-gc every night?

> Now, repacking doesn't work because...
> 
> > $ git gc --aggressive
> > error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2
> > error: Could not read 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f

/me is pretty git dumb, but is there some way to figure out the parents
or children of these?  I just trolled through all of my directories
doing git show and didn't get any hits.  I guess I'll just clean up and
start over....

> Those objects are indeed missing from the repository.  Without them your 
> repository is "broken".  Either you can find them somewhere else and 
> copy them over, or salvage as much as you can by fetching the 
> interesting branches into another freshly made repository.  This is 
> unfortunate because I would have liked to see by how much this 
> repository would have shrunk after a successful repack.
> 
> Of course, usage of alternates is recommended _only_ with repositories 
> that are stable, i.e. don't ever add repositories to 
> .git/objects/info/alternates if those repositories are rewinded/rebased 
> and/or branches in them are deleted/replaced.  That could be a reason 
> why some objects are now missing from the repository using alternates.

So I'm not sure how I did things wrong.  my kernel-1 has those bunch of
remotes.  The linux-next remote, like I said, basically rebases to
linus' tree, then merges 150 random branches.  It tags that tree every
day and I pull those tags.  So I would never expect any objects from
those remote trees to ever disappear.

Now I created branches in kernel-1 and I certainly have done lots of
things like so

git checkout -b testing remotes/linux-next/master
[edit]
git commit -a
git checkout -b testing1 remotes/linux-next/master
git branch -D testing

My assumption though was that this wouldn't ever affect my other
repositories.  My other repository branches always started by checking
out a branch with remotes/*/* as the base.

My understanding was that I would only run into problems if I used
something on a branch I created myself in the alternatives repo in other
repos (and I didn't remove remotes)

I guess it's not impossible to believe that at some point in time i
would have exported patches to and mbox from kernel-1 and applied them
to kernel-2 or vice versa.  I guess this would end up with the same
objects, right?  Then if I deleted the branch in kernel-1 I would have
problems in kernel-2?

I guess I'll rebuild my setup

new kernel-alt has just the remotes, and my kernel-1,2,3 all alt to it
I'll never have local branches in my kernel-alt
I'll run git-gc every night
I'll hope to never have problem again.

Sound good?