Git clone fails with fatal: the remote end hung up unexpectedly

Git clone fails with fatal: the remote end hung up unexpectedly

[Owen Ofiesh]

We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails with the following error using the HTTP protocol.

The error on the client is:
   fatal: the remote end hung up unexpectedly
   fatal: early EOF
   fatal: index-pack failed

The client is writing to an NFS volume.

The HTTP POST error on the server is:
   c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection
   c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection

Our git repositories are managed by Atlassian's bitbucket server v5.5.1.

We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem to matter the client SW version).

The Linux git server is SW v2.9.5.

Per discussions about this we've tried the following:
1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server to 100MBs.
2. We set the http.postBuffer client value to 100MBs.
Neither of these helped.

Any thoughts about this please?

Note: 
For reasons I will not enter into, we cannot use SSH for this.

Thank you.

end

Regards,
Owen Ofiesh, SCM, Software Test and Development Manager
MaxLinear, Inc.  |  www.maxlinear.com 
5966 La Place Court Suite 100 Carlsbad, CA 92008
Phone: 760-517-1109
Cell: 858-335-1690
oofiesh@maxlinear.com 

Re: Git clone fails with fatal: the remote end hung up unexpectedly

[Bryan Turner]

On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh <oofiesh@maxlinear.com> wrote:
>
> We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails with the following error using the HTTP protocol.
>
> The error on the client is:
>    fatal: the remote end hung up unexpectedly
>    fatal: early EOF
>    fatal: index-pack failed
>
> The client is writing to an NFS volume.

A further detail on this (Owen correct me if I'm wrong), but the same
clone performed to a local disk, rather than NFS, succeeds.

>
> The HTTP POST error on the server is:
>    c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection
>    c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection

Bitbucket Server developer here. I just want to clarify one thing
here, for the list. The above is logged when the cloning client
disconnects unexpectedly in the middle of a hosting operation (push or
pull). So, from the server's perspective, everything was going well
and then the client left without saying goodbye.

>
> Our git repositories are managed by Atlassian's bitbucket server v5.5.1.
>
> We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem to matter the client SW version).
>
> The Linux git server is SW v2.9.5.
>
> Per discussions about this we've tried the following:
> 1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server to 100MBs.
> 2. We set the http.postBuffer client value to 100MBs.
> Neither of these helped.
>
> Any thoughts about this please?
>
> Note:
> For reasons I will not enter into, we cannot use SSH for this.
>
> Thank you.
>
> end
>
> Regards,
> Owen Ofiesh, SCM, Software Test and Development Manager
> MaxLinear, Inc.  |  www.maxlinear.com
> 5966 La Place Court Suite 100 Carlsbad, CA 92008
> Phone: 760-517-1109
> Cell: 858-335-1690
> oofiesh@maxlinear.com
>
>

RE: Git clone fails with fatal: the remote end hung up unexpectedly

[Owen Ofiesh]

Hi Bryan, all,

> From: Bryan Turner [mailto:bturner@atlassian.com] 
> Sent: Monday, December 10, 2018 5:16 PM
> To: Owen Ofiesh <oofiesh@maxlinear.com>
> Cc: Git Users <git@vger.kernel.org>
> Subject: Re: Git clone fails with fatal: the remote end hung up unexpectedly
> 
> On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh <mailto:oofiesh@maxlinear.com> wrote:
> >
> > We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails
> > with the following error using the HTTP protocol.
> >
> > The error on the client is:
> > fatal: the remote end hung up unexpectedly
> > fatal: early EOF
> > fatal: index-pack failed
> >
> > The client is writing to an NFS volume.
> 
> A further detail on this (Owen correct me if I'm wrong), but the same
> clone performed to a local disk, rather than NFS, succeeds.

Correct. The clone to local disk succeeds.

One more data point. I tried this with SSH just now, and the behavior
is somewhat improved in that I am only seeing the failure on the NFS
volume so far with 1 in 6 clone attempts (5 successful). Whereas with
HTTP, we see the failure every time. 

> >
> > The HTTP POST error on the server is:
> > c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection
> > c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection
> 
> Bitbucket Server developer here. I just want to clarify one thing
> here, for the list. The above is logged when the cloning client
> disconnects unexpectedly in the middle of a hosting operation (push or
> pull). So, from the server's perspective, everything was going well
> and then the client left without saying goodbye.

Correct in that we see this behavior when cloning. Small updates
(push/pull) seems to work okay. We tested this previously by cloning to
local disk, then copying to NFS, and then trying very small updates.

> > Our git repositories are managed by Atlassian's bitbucket server v5.5.1.
> >
> > We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem
> > to matter the client SW version).
> >
> > The Linux git server is SW v2.9.5.
> >
> > Per discussions about this we've tried the following:
> > 1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server
> >    to 100MBs.
> > 2. We set the http.postBuffer client value to 100MBs.
> > Neither of these helped.
> >
> > Any thoughts about this please?
> >
> > Note:
> > For reasons I will not enter into, we cannot use SSH for this.
> >
> > Thank you.
> >
> > end
> >
> > Regards,
> > Owen Ofiesh, SCM, Software Test and Development Manager
> > MaxLinear, Inc. | http://www.maxlinear.com
> > 5966 La Place Court Suite 100 Carlsbad, CA 92008
> > Phone: 760-517-1109
> > Cell: 858-335-1690
> > mailto:oofiesh@maxlinear.com
> >
> >

Re: Git clone fails with fatal: the remote end hung up unexpectedly

[Jeff King]

On Tue, Dec 11, 2018 at 02:25:41AM +0000, Owen Ofiesh wrote:

> > On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh <mailto:oofiesh@maxlinear.com> wrote:
> > >
> > > We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails
> > > with the following error using the HTTP protocol.
> > >
> > > The error on the client is:
> > > fatal: the remote end hung up unexpectedly
> > > fatal: early EOF
> > > fatal: index-pack failed
> > >
> > > The client is writing to an NFS volume.
> > 
> > A further detail on this (Owen correct me if I'm wrong), but the same
> > clone performed to a local disk, rather than NFS, succeeds.
> 
> Correct. The clone to local disk succeeds.

That's weird. The messages imply that the server has started sending the
packfile, at which point we should be pumping data from git-remote-https
to fetch-pack, and from there into an index-pack process. And the
messages imply that the client saw a hangup of the network connection,
at which point index-pack complained of the truncated packfile (and then
we complained that index-pack complained).

But what's weird is that none of that should really be relevant to the
choice of local filesystem. Is it possible that using NFS stresses the
network to the point that the HTTP connection may be killed?

It's also possible there's something racy in Git's handling of the data,
and NFS is slower to write. What version of Git is this?

> One more data point. I tried this with SSH just now, and the behavior
> is somewhat improved in that I am only seeing the failure on the NFS
> volume so far with 1 in 6 clone attempts (5 successful). Whereas with
> HTTP, we see the failure every time.

If you use ssh's verbose mode, what does a failing case look like? If
you have Git v2.10 or later, you can just do:

  GIT_SSH_COMMAND='ssh -v' git clone ...

If it's older, the simplest thing is probably to put:

  Host yourserver...
  LogLevel Verbose

into your ~/.ssh/config.

I know the problem is more common with HTTP, but I suspect ssh's logging
may be better than ours. ;) You can also try:

  GIT_CURL_VERBOSE=1 git clone ...

or if you have recent enough (Git v2.10 again, I think), you can use:

  GIT_TRACE_CURL=1 git clone ...

which has a few more details.

-Peff