git-local-pull

git-local-pull

[David S. Miller]

I tried to start using git-clone-script to clone repositories
locally.  It crunches on the disk for a couple of seconds,
that's fine, but then I notice the disk activity stop and
git-local-pull becomes cpu bound and grows to 80MB in size
over the course of 5 minutes.

Is this a side effect of the new pack/unpack stuff?

Compared to what this thing is doing, manually symlinking
the object database, copying over the HEAD, and building
the index file is significantly faster.

Actually, git-clone-script didn't build an index file.
So the compute time definitely came from something else
entirely.

Re: git-local-pull

[Linus Torvalds]

On Sun, 26 Jun 2005, David S. Miller wrote:
> 
> I tried to start using git-clone-script to clone repositories
> locally.  It crunches on the disk for a couple of seconds,
> that's fine, but then I notice the disk activity stop and
> git-local-pull becomes cpu bound and grows to 80MB in size
> over the course of 5 minutes.

Try adding "-l" to the git-local-pull command line, to make it do 
hardlinks by default (and fall back to a copy if that fails).

> Is this a side effect of the new pack/unpack stuff?

Nope, nothing uses that yet. 

I think the 80MB is just because it built up the object relationship for 
the whole kernel db there. I don't know what the 5 minutes of CPU are, 
though, it should just be copying stuff (and at 250MB in and 250MB out, 
that's a lot of stuff to copy, but it shouldn't be five minutes worth, 
especially if it's all cached and you see a pegged CPU).

That copy is stupid, btw. Doing a "mmap + write" is a lot slower than
doing a "while (data) { read(8kB) + write(8kB) }" loop. The read() will
generally do better read-ahead and the small read/write will be better for 
the caches. But that still doesn't explain 5 minutes worth.

> Compared to what this thing is doing, manually symlinking
> the object database, copying over the HEAD, and building
> the index file is significantly faster.

Well, with "-l" that is what git-local-pull will do too. And if you want 
symlinks, use "-s". Or use both, in which case it will try a hardlink 
first, and then a symlink if it gets an EXDEV.

		Linus

Re: git-local-pull

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Sun, 26 Jun 2005, David S. Miller wrote:
> 
>>I tried to start using git-clone-script to clone repositories
>>locally.  It crunches on the disk for a couple of seconds,
>>that's fine, but then I notice the disk activity stop and
>>git-local-pull becomes cpu bound and grows to 80MB in size
>>over the course of 5 minutes.
> 
> 
> Try adding "-l" to the git-local-pull command line, to make it do 
> hardlinks by default (and fall back to a copy if that fails).
> 
> 
>>Is this a side effect of the new pack/unpack stuff?
> 
> 
> Nope, nothing uses that yet. 
> 
> I think the 80MB is just because it built up the object relationship for 
> the whole kernel db there. I don't know what the 5 minutes of CPU are, 
> though, it should just be copying stuff (and at 250MB in and 250MB out, 
> that's a lot of stuff to copy, but it shouldn't be five minutes worth, 
> especially if it's all cached and you see a pegged CPU).

FWIW, I notice that git-diff-tree uses >100MB of RAM, on a large 
git-pull-script run.  It's generating the diffstat output, AFAICS.

	Jeff

Re: git-local-pull

[Linus Torvalds]

On Mon, 27 Jun 2005, Jeff Garzik wrote:
> 
> FWIW, I notice that git-diff-tree uses >100MB of RAM, on a large 
> git-pull-script run.  It's generating the diffstat output, AFAICS.

Ahh, yes, I should make the diffstat be conditional, not everybody wants 
it, and it's often the most expensive part of the pull..

		Linus