Re: Git is not scalable with too many refs/*

[Martin Fick <mfick@codeaurora.org>]

On Wednesday, September 28, 2011 01:38:04 pm Martin Fick 
wrote:
> On Monday, September 26, 2011 06:41:04 am Christian
> Couder
> 
> wrote:
> > On Sun, Sep 25, 2011 at 10:43 PM, Martin Fick
> 
> <mfick@codeaurora.org> wrote:
> ...
> 
> > >  git checkout
> > > 
> > > can also take rather long periods of time > 3 mins
> > > when run on a repo with ~100K refs.
> 
> ...
> 
> > >  So, I bisected this issue also, and it seems that
> > >  the
> > > 
> > > "offending" commit is
> 
> ...
> 
> > > commit 680955702990c1d4bfb3c6feed6ae9c6cb5c3c07
> > > Author: Christian Couder <chriscool@tuxfamily.org>
> > > 
> > >    replace_object: add mechanism to replace objects
> > > 
> > > found in "refs/replace/"
> 
> ...
> 
> > I don't think there is an obvious problem with it, but
> > it would be nice if you could dig a bit deeper.
> > 
> > The first thing that could take a lot of time is the
> > call to for_each_replace_ref() in this function:
> > 
> > +static void prepare_replace_object(void)
> > +{
> > +       static int replace_object_prepared;
> > +
> > +       if (replace_object_prepared)
> > +               return;
> > +
> > +       for_each_replace_ref(register_replace_ref,
> > NULL); +       replace_object_prepared = 1;
> > +}
> 
> The time was actually spent in for_each_replace_ref()
> which calls get_loose_refs() which has the recursive bug
> that Julian Phillips fixed 2 days ago.  Good to see that
> this fix helps other use cases too.
> 
> So with that bug fixed, the thing taking the most time
> now for a git checkout with ~100K refs seems to be the
> orphan check as Thomas predicted.  The strange part with
> this, is that the orphan check seems to take only about
> ~20s in the repo where the refs aren't packed.  However,
> in the repo where they are packed, this check takes at
> least 5min!  This seems a bit unusual, doesn't it?  Is
> the filesystem that much better at indexing refs than
> git's pack mechanism? Seems unlikely, the unpacked refs
> take 312M in the FS, the packed ones only take about
> 4.3M.  I suspect their is something else unexpected
> going on here in the packed ref case.
> 
> Any thoughts?  I will dig deeper...

I think the problem is that resolve_ref() walks a linked 
list of searching for the packed ref.  Does this mean that 
packed refs are not indexed at all?

> 
> -Martin

-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum