git fetch slow as molasses due to tag downloading

git fetch slow as molasses due to tag downloading

[Han-Wen Nienhuys]

Hello,

just upgraded our autobuilder from 1.4.3 to 1.4.4.2.

Now, our standard download command comes to a complete halt.  Judging
from the "ps -ef" apparently, it does 

   git-show-ref --verify --quiet -- [TAG]

This is done for every one of the 1500 tags that are in my repository. 
At approx 20 tags per second this takes an awful lot of time. 

1. Is this necessary? 

2. Is this efficient?  Wouldn't doing all tags in a single git-show-ref
invocation be potentially quicker?


-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen

Re: git fetch slow as molasses due to tag downloading

[Johannes Schindelin]

Hi,

On Thu, 14 Dec 2006, Han-Wen Nienhuys wrote:

> just upgraded our autobuilder from 1.4.3 to 1.4.4.2.
> 
> Now, our standard download command comes to a complete halt.  Judging
> >from the "ps -ef" apparently, it does 
> 
>    git-show-ref --verify --quiet -- [TAG]
> 
> This is done for every one of the 1500 tags that are in my repository. 
> At approx 20 tags per second this takes an awful lot of time. 
> 
> 1. Is this necessary? 

Yes. The purpose is to check which tags have not yet been fetched.

> 2. Is this efficient?  Wouldn't doing all tags in a single git-show-ref 
> invocation be potentially quicker?

It is not efficient. But it cannot be solved like you propose, since it is 
inside a loop, and a "continue" is executed when the tag exists already.

IMHO this should be solved as a filter: "git-show-ref --stdin 
--show-invalid". Thus, git does not have to traverse _every_ ref for 
_every_ incoming tag.

Ciao,
Dscho

Re: git fetch slow as molasses due to tag downloading

[Jakub Narebski]

Johannes Schindelin wrote:

> IMHO this should be solved as a filter: "git-show-ref --stdin 
> --show-invalid". Thus, git does not have to traverse _every_ ref for 
> _every_ incoming tag.

Can you make it so? TIA.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: git fetch slow as molasses due to tag downloading

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> IMHO this should be solved as a filter: "git-show-ref --stdin 
> --show-invalid". Thus, git does not have to traverse _every_ ref for 
> _every_ incoming tag.

That sounds like a too specialized hack to me.  We should first
speed up the general "--verify $ref" case; as you corrected me
earlier it still has a useless loop.

If it is still too slow (which I suspect it could be the case,
with fork+exec overhead), we should _also_ pursue the filter
approach, but even then I think "filter out the valid ones" is a
specialized hack, if you mean "show only the invalid one's
names" by --show-invalid.

It would make sense to do

        $ git show-ref --show-invalid v1.0.0 v2.6.18
        f665776185ad074b236c00751d666da7d1977dbe refs/tags/v1.0.0
        - refs/tags/v2.6.18

(and its equivalent to take refs from --stdin) though.

Re: git fetch slow as molasses due to tag downloading

[Jakub Narebski]

Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
>> IMHO this should be solved as a filter: "git-show-ref --stdin 
>> --show-invalid". Thus, git does not have to traverse _every_ ref for 
>> _every_ incoming tag.
> 
> That sounds like a too specialized hack to me.  We should first
> speed up the general "--verify $ref" case; as you corrected me
> earlier it still has a useless loop.
> 
> If it is still too slow (which I suspect it could be the case,
> with fork+exec overhead), we should _also_ pursue the filter
> approach, but even then I think "filter out the valid ones" is a
> specialized hack, if you mean "show only the invalid one's
> names" by --show-invalid.
> 
> It would make sense to do
> 
>         $ git show-ref --show-invalid v1.0.0 v2.6.18
>         f665776185ad074b236c00751d666da7d1977dbe refs/tags/v1.0.0
>         - refs/tags/v2.6.18
> 
> (and its equivalent to take refs from --stdin) though.

Nice idea. And having '-' is probably better than using
00000000000000000000000000000000000000000 as for non-existing objects.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git