Cannot clone redirecting stdout

Cannot clone redirecting stdout

[Aloisio]

Hi all,

I faced a problem when trying to clone git://gitorious.org/qt/qt.git

this works:
git clone -n git://gitorious.org/qt/qt.git repo

this doesn't:
git clone -n git://gitorious.org/qt/qt.git repo >log
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

I reproduced the error in the following versions:
git version 1.6.0.4
git version 1.6.4.2
git version 1.5.4.7

Any clues?
Thanks

Aloisio Almeida

Re: Cannot clone redirecting stdout

[Stefan Naewe]

On 9/11/2009 12:33 AM, Aloisio wrote:
> Hi all,
> 
> I faced a problem when trying to clone git://gitorious.org/qt/qt.git
> 
> this works:
> git clone -n git://gitorious.org/qt/qt.git repo
> 
> this doesn't:
> git clone -n git://gitorious.org/qt/qt.git repo >log
> fatal: The remote end hung up unexpectedly
> fatal: early EOF
> fatal: index-pack failed
> 
> I reproduced the error in the following versions:
> git version 1.6.0.4
> git version 1.6.4.2
> git version 1.5.4.7
> 
> Any clues?
> Thanks

I'd say:

Take a look at /proc/<pid-of-git>/fd

fd 0 (stdout) is linked through a pipe to 'git index-pack'.
Redirecting this link breaks everything.

Regards,
Stefan

Re: Cannot clone redirecting stdout

[Stefan Naewe]

On 9/11/2009 8:39 AM, Stefan Naewe wrote:
> On 9/11/2009 12:33 AM, Aloisio wrote:
>> Hi all,
>>
>> I faced a problem when trying to clone git://gitorious.org/qt/qt.git
>>
>> this works:
>> git clone -n git://gitorious.org/qt/qt.git repo
>>
>> this doesn't:
>> git clone -n git://gitorious.org/qt/qt.git repo >log
>> fatal: The remote end hung up unexpectedly
>> fatal: early EOF
>> fatal: index-pack failed
>>
>> I reproduced the error in the following versions:
>> git version 1.6.0.4
>> git version 1.6.4.2
>> git version 1.5.4.7
>>
>> Any clues?
>> Thanks
> 
> I'd say:
> 
> Take a look at /proc/<pid-of-git>/fd
> 
> fd 0 (stdout) is linked through a pipe to 'git index-pack'.
> Redirecting this link breaks everything.

stdout is of course fd 1 ;-)

Regards,
Stefan
-- 
/dev/random says: Never trust a computer you can't lift. - Stan Masor

Re: Cannot clone redirecting stdout

[Jean-Luc Herren]

Hi list!

Aloisio wrote:
> this doesn't:
> git clone -n git://gitorious.org/qt/qt.git repo >log
> fatal: The remote end hung up unexpectedly
> fatal: early EOF
> fatal: index-pack failed

What are you trying to store in that file?  The only line I
observe being printed to stdout while cloning is this one:

    Initialized empty Git repository in [...]/repo/.git/

Everything else (progress status) is printed to stderr and I
presume that any error messages would also go to stderr too, if
any.  So possibly you would be happy with this:

    git clone -n git://gitorious.org/qt/qt.git repo 2> log

which works fine here with git-1.6.4.2.  Though I agree that I
would expect redirecting stdout to work.

Cheers,
jlh

Re: Cannot clone redirecting stdout

[Jeff King]

On Thu, Sep 10, 2009 at 07:33:21PM -0300, Aloisio wrote:

> I faced a problem when trying to clone git://gitorious.org/qt/qt.git
> 
> this works:
> git clone -n git://gitorious.org/qt/qt.git repo
> 
> this doesn't:
> git clone -n git://gitorious.org/qt/qt.git repo >log
> fatal: The remote end hung up unexpectedly
> fatal: early EOF
> fatal: index-pack failed

I can reproduce the problem here. But after staring at the strace for a
long time, I don't think the problem is on the client side. The remote
end _does_ hang up unexpectedly.

Looking at what we send, the only difference between the redirected and
unredirected case I could find is that we send the "no-progress" flag to
the server, which then hangs up on us instead of sending us the pack.
Which makes no sense.

Confused,
-Peff

Re: Cannot clone redirecting stdout

[Jeff King]

On Fri, Sep 11, 2009 at 06:23:29AM -0400, Jeff King wrote:

> > I faced a problem when trying to clone git://gitorious.org/qt/qt.git
> > 
> > this works:
> > git clone -n git://gitorious.org/qt/qt.git repo
> > 
> > this doesn't:
> > git clone -n git://gitorious.org/qt/qt.git repo >log
> > fatal: The remote end hung up unexpectedly
> > fatal: early EOF
> > fatal: index-pack failed
> 
> I can reproduce the problem here. But after staring at the strace for a
> long time, I don't think the problem is on the client side. The remote
> end _does_ hang up unexpectedly.
> 
> Looking at what we send, the only difference between the redirected and
> unredirected case I could find is that we send the "no-progress" flag to
> the server, which then hangs up on us instead of sending us the pack.
> Which makes no sense.

I did a little more testing, and I can't reproduce the problem against a
local git-daemon. I tried using several versions for the server, going
all the way back to v1.5.0, which pre-dates no-progress, and all of them
worked just fine.

So I am inclined to think there is something non-standard or broken at
gitorious.org. I'm cc'ing support@gitorious to see if they have any
comment.

-Peff

Re: Cannot clone redirecting stdout

[Johan Sørensen]

On Fri, Sep 11, 2009 at 3:51 PM, Jeff King <peff@peff.net> wrote:
> On Fri, Sep 11, 2009 at 06:23:29AM -0400, Jeff King wrote:
>
>> > I faced a problem when trying to clone git://gitorious.org/qt/qt.git
>> >
>> > this works:
>> > git clone -n git://gitorious.org/qt/qt.git repo
>> >
>> > this doesn't:
>> > git clone -n git://gitorious.org/qt/qt.git repo >log
>> > fatal: The remote end hung up unexpectedly
>> > fatal: early EOF
>> > fatal: index-pack failed
>>
>> I can reproduce the problem here. But after staring at the strace for a
>> long time, I don't think the problem is on the client side. The remote
>> end _does_ hang up unexpectedly.
>>
>> Looking at what we send, the only difference between the redirected and
>> unredirected case I could find is that we send the "no-progress" flag to
>> the server, which then hangs up on us instead of sending us the pack.
>> Which makes no sense.
>
> I did a little more testing, and I can't reproduce the problem against a
> local git-daemon. I tried using several versions for the server, going
> all the way back to v1.5.0, which pre-dates no-progress, and all of them
> worked just fine.
>
> So I am inclined to think there is something non-standard or broken at
> gitorious.org. I'm cc'ing support@gitorious to see if they have any
> comment.

Some quick tests seem to indicate it's related to the fact that our
wonderful little fork+exec git-daemon[1] (which is different from the
one distributed with git) exec's to "git-upload-pack --strict
--timeout=30 /path/to/repo". Now, why exactly that'll trigger when the
no-progress flag is given I'm not sure of. The daemon itself execs as
soon as it figures out what repo the client requested, so apart from
the timeout the only thing it reacts to is the header (the
"${headersize}git-upload-pack project/repo.git\0host=gitorious.org\0"
part).

We also do redirect stderr to /dev/null for reasons I cannot remember
(so probably not good ones), that may be related as well. Well run
some more tests...

[1]: http://gitorious.org/gitorious/mainline/blobs/master/script/git-daemon

>
> -Peff
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Cannot clone redirecting stdout

[Jeff King]

On Fri, Sep 11, 2009 at 05:46:23PM +0200, Johan Sørensen wrote:

> Some quick tests seem to indicate it's related to the fact that our
> wonderful little fork+exec git-daemon[1] (which is different from the
> one distributed with git) exec's to "git-upload-pack --strict
> --timeout=30 /path/to/repo". Now, why exactly that'll trigger when the
> no-progress flag is given I'm not sure of. The daemon itself execs as
> soon as it figures out what repo the client requested, so apart from
> the timeout the only thing it reacts to is the header (the
> "${headersize}git-upload-pack project/repo.git\0host=gitorious.org\0"
> part).
> 
> We also do redirect stderr to /dev/null for reasons I cannot remember
> (so probably not good ones), that may be related as well. Well run
> some more tests...

Ah. I have a theory. If I do a clone of git://gitorious.org/qt/qt.git,
the counting/compressing stages take a long time (I timed it at 1m40
before it actually sends any data). And looking at upload-pack.c, we
leave the 30-second alarm set while creating the pack. Meaning we die 30
seconds into creating the pack.

When progress is being displayed, however, the progress timer actually
uses SIGALRM, as well. So we are constantly resetting the timer and it
never goes off.

And we should be able to test this theory. How long does it take for the
failure case to fail:

  $ time git clone -n git://gitorious.org/qt/qt.git repo >log
  fatal: The remote end hung up unexpectedly
  fatal: early EOF
  fatal: index-pack failed

  real    0m31.106s
  user    0m0.000s
  sys     0m0.012s

Hmm. Suspicious. :)

So that implies to me a few things:

  1. You guys should really pack your repos, as you are wasting over a
     minute of CPU time every time somebody clones this repo.

  2. Upload-pack has what I consider a bug. The --timeout should be
     suspended while we are actually crunching numbers to create the
     pack. We probably don't want it when sending the pack, either, as
     people with slow connections (or big repos) will get timed out
     during the send. Probably we just want to apply it to times when we
     are waiting to get the list of refs from the client.

  3. Upload-pack and the progress code are both using the global alarm
     timer and signal, and that is papering over the bug in (2) when
     progress is enabled. I'm not sure of the simplest way of having
     those interact. But maybe we can just ignore it, because we
     probably don't want to using the --timeout alarm during the packing
     phase, anyway.

-Peff

Re: Cannot clone redirecting stdout

[Jeff King]

On Fri, Sep 11, 2009 at 12:05:10PM -0400, Jeff King wrote:

> Ah. I have a theory. If I do a clone of git://gitorious.org/qt/qt.git,
> the counting/compressing stages take a long time (I timed it at 1m40
> before it actually sends any data). And looking at upload-pack.c, we
> leave the 30-second alarm set while creating the pack. Meaning we die 30
> seconds into creating the pack.
> 
> When progress is being displayed, however, the progress timer actually
> uses SIGALRM, as well. So we are constantly resetting the timer and it
> never goes off.

Hmm. Actually, this is not quite right. It looks like we call out to
pack-objects as an external program, so there is no conflict with the
signal. And we do proxy the output of pack-objects, which will keep our
timer resetting every time we see a chunk of output. But pack-objects
produces no output during the deltification phase, unless progress is
turned on.  So we still hit our timeout in upload-pack during that
phase.

So our options are:

  1. Turn off the timer during deltification, which could mean that it
     would potentially go forever. But it's not controlled by the user.
     I think the 'timeout' feature is really about the client just
     opening the connection and sitting.

  2. Keep progress on during deltification, but just throw it away
     instead of relaying it if no-progress is in effect.

  3. Accept that hitting the timeout during deltification _should_ cause
     it to die. In that case, then the case with progress is wrong, and
     we should stop resetting the timer just because we got some
     progress output from pack-objects. But this may be redefining the
     intent of --timeout. I don't know what the original intent was, or
     what users of the feature are expecting.

-Peff

Re: Cannot clone redirecting stdout

[Daniel Barkalow]

On Fri, 11 Sep 2009, Jeff King wrote:

> On Fri, Sep 11, 2009 at 12:05:10PM -0400, Jeff King wrote:
> 
> > Ah. I have a theory. If I do a clone of git://gitorious.org/qt/qt.git,
> > the counting/compressing stages take a long time (I timed it at 1m40
> > before it actually sends any data). And looking at upload-pack.c, we
> > leave the 30-second alarm set while creating the pack. Meaning we die 30
> > seconds into creating the pack.
> > 
> > When progress is being displayed, however, the progress timer actually
> > uses SIGALRM, as well. So we are constantly resetting the timer and it
> > never goes off.
> 
> Hmm. Actually, this is not quite right. It looks like we call out to
> pack-objects as an external program, so there is no conflict with the
> signal. And we do proxy the output of pack-objects, which will keep our
> timer resetting every time we see a chunk of output. But pack-objects
> produces no output during the deltification phase, unless progress is
> turned on.  So we still hit our timeout in upload-pack during that
> phase.
> 
> So our options are:
> 
>   1. Turn off the timer during deltification, which could mean that it
>      would potentially go forever. But it's not controlled by the user.
>      I think the 'timeout' feature is really about the client just
>      opening the connection and sitting.
> 
>   2. Keep progress on during deltification, but just throw it away
>      instead of relaying it if no-progress is in effect.
> 
>   3. Accept that hitting the timeout during deltification _should_ cause
>      it to die. In that case, then the case with progress is wrong, and
>      we should stop resetting the timer just because we got some
>      progress output from pack-objects. But this may be redefining the
>      intent of --timeout. I don't know what the original intent was, or
>      what users of the feature are expecting.

You don't remember October 2005? HPA introduced it in 960decc, which has a 
pretty good explanation: we doesn't want to get DoS'd if clients just send 
SYNs. So it's supposed to time out only if we spend that long waiting 
for a protocol item from the client.

	-Daniel
*This .sig left intentionally blank*