Re: Slow "git rev-list origin/master --not --all" or "git fetch" slow when downloading nothing

[Linus Torvalds <torvalds@linux-foundation.org>]

On Wed, 5 Nov 2008, Santi Béjar wrote:
> >
> > Hmm. It sounds like you possibly don't have packed refs.
> 
> They are packed up to v2.6.27.

Yeah, your strace isn't at all horrible. You don't open very many files at 
all, and you don't have any big directories.

The biggest cost when things are cold-cache is probably the seeking from 
just opening all those index files. 

> It is an old computer (Pentium 4 2.5 GHz) and the repo is on an external 
> USB drive.

There should be basically no CPU spent on that load, so your computer is 
fine. But I think the issue is the dog-slow IO on the USB drive, 
especially since there are multiple pack-files and thus index files.

Your strace would be more interesting with "-Ttt", but much of the cost is 
likely in the page faulting of the mmap'ed data, and none of that would 
show up in the trace, except indirectly (ie just looking at the times 
between the system calls).

> Yes, in the general case it is, but in this case we can bypass the
> checking of the --all refs after checking if all the given refs are
> equal to some of the --all refs.

I don't think we'll actually walk anything, because all commits will end 
up being negative.

But we'll look up the objects for even the negative commits, yes. So we're 
doing several "unnecessary" object lookups, and in that sense we could 
make this much faster by not even bothering to look them up.

But we do that to validate that the refs are _valid_, so in that sense the 
object lookup is not "unnecessary" at all. Oh, and we need to peel them to 
see if they are tag objects, in order to mark the _commit_ uninteresting 
if the object itself was uninteresting.

So in practice we do end up having to pretty much parse them all. 

We could do some crazy special case for the empty set, but it would be 
better to see if you can improve performance with a slow disk some other, 
less hacky, way. If you use a USB stick to move between machines, maybe 
you can make sure that it's fully packed (ie a single index file) before 
moving it to the USB stick? That would likely help quite a bit.

		Linus