Re: git pull & git gc

[Jeff King <peff@peff.net>]

On Wed, Mar 18, 2015 at 02:58:15PM +0000, John Keeping wrote:

> On Wed, Mar 18, 2015 at 09:41:59PM +0700, Duy Nguyen wrote:
> > On Wed, Mar 18, 2015 at 9:33 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> > > If not, I made some mistake in analyzing this and we'll start again.
> > 
> > I did make one mistake, the first "gc" should have reduced the number
> > of loose objects to zero. Why didn't it.?  I'll come back to this
> > tomorrow if nobody finds out first :)
> 
> Most likely they are not referenced by anything but are younger than 2
> weeks.
> 
> I saw a similar issue with automatic gc triggering after every operation
> when I did something equivalent to:
> 
> 	git add <lots of files>
> 	git commit
> 	git reset --hard HEAD^
> 
> which creates a log of unreachable objects which are not old enough to
> be pruned.

Yes, this is almost certainly the problem. Though to be pedantic, the
command above will still have a reflog entry, so the objects will be
reachable (and packed). But there are other variants that don't leave
the objects reachable from even reflogs.

I don't know if there is an easy way around this. Auto-gc's object count
is making the assumption that running the gc will reduce the number of
objects, but obviously it does not always do so. We could do a more
thorough check and find the number of actual packable and prune-able
objects. The "prune-able" part of that is easy; just omit objects from
the count that are newer than 2 weeks. But "packable" is expensive. You
would have to compute reachability by walking from the tips. That can
take tens of seconds on a large repo.

You could perhaps cut off the walk early when you hit a packed commit
(this does not strictly imply that all of the related objects are
packed, but it would be good enough for a heuristic). But even that is
probably too expensive for "gc --auto".

-Peff

PS Note that in git v2.2.0 and up, prune will leave not only "recent"
   unreachable objects, but older objects which are reachable from those
   recent ones (so that we keep or prune whole chunks of history, rather
   than dropping part and leaving the rest broken). Technically this
   exacerbates the problem (we keep more objects), though I doubt it
   makes much difference in practice (most chunks of history were
   created at similar times, so the mtimes of the whole chunk will be
   close together).