git interaction between push and auto-gc

git interaction between push and auto-gc

[Bob Lavey]

Greetings,

We are seeing some unexpected behavior with git that we'd like to better 
understand.  We are running git 1.7.11.1 on the remote repo server 
(CentOS 6.3-x86) and the clients (mostly Windows 7), and we are running 
gitolite 3.04 on the server.

At times, our repos can get many unreachable loose objects, and if we 
don't clean them up, we will see instances where a push to the remote 
repo does not work correctly.  This almost always occurs when we try to 
delete a remote branch with a "git push origin :branch" call, but this 
week we saw two occurrences where a push failed to create a branch (as 
well as about a dozen cases where a push failed to delete the branch). 
This behavior does not occur for every push, however: I'd guess 10% of 
pushes fail.

Here is output from a user's machine:

d:\git\jedi\jedifw>git push origin 
configMgrTestBuilder:refs/23s/iq/qbar/user.name@hp.com/jit20073
Counting objects: 59, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (29/29), done.
Writing objects: 100% (30/30), 129.69 KiB, done.
Total 30 (delta 24), reused 1 (delta 0)
Auto packing the repository for optimum performance.
warning: There are too many unreachable loose objects; run 'git prune' 
to remove them.
To git@git.boi.hp.com:/jedi/jedifw.git


After a couple of hours, that user was concerned that their branch 
hadn't started processing, so they sent an email to support and the 
re-pushed.  Here are the lines from the gitolite log showing both of 
those pushes:

gitolite-2013-01-07.log:2013-01-07.15:57:58     23285   update 
jedi/jedifw     user.name@hp.com    W 
refs/23s/iq/qbar/user.name@hp.com/jit20073 
0000000000000000000000000000000000000000 
2ab0e6626c7a51799179993ea0fdbb829a1ea852

gitolite-2013-01-07.log:2013-01-07.19:57:16     30374   update 
jedi/jedifw     user.name@hp.com    W 
refs/23s/iq/qbar/user.name@hp.com/jit20073 
0000000000000000000000000000000000000000 
2ab0e6626c7a51799179993ea0fdbb829a1ea852


There are no log entries for that branch or that SHA in between those 
two lines.  My understanding is that both of those lines show a new 
branch being created on the remote repo.  I'm not sure what happened to 
the first push, though: I expected that the branch would have been 
created at that time, and subsequent fetches from the remote repo would 
show the branch.  However, fetches from the remote repo did not show the 
branch until after the second push.

We've been running git for about 3 years, and this happened occasionally 
when we first started with git, but we always found it was related to a 
huge number of unreachable loose objects, which triggered an auto-gc on 
the remote repo.  When we performed manual gc --aggressive and prune 
operations on the remote repo, the problem stopped.  It happened this 
week because we'd deleted over 4,000 refs/topics branches on 
Friday/Saturday, which left a huge number of unreachable loose objects. 
  Our cleanup script was only pruning objects older than 2.weeks.ago, 
and when I pruned objects older than 2.days.ago the problem again stopped.

Is this expected behavior for the interaction between pushes and auto-gc 
on the remote repo?

Also, I went through the release notes for the latest versions of git, 
and I found this for 1.7.11.6:

* When "git push" triggered the automatic gc on the receiving end, a
    message from "git prune" that said it was removing cruft leaked to
    the standard output, breaking the communication protocol.

Could that be related to what we're seeing?

Thanks,
Bob Lavey