git clone failing over slow links?

git clone failing over slow links?

[skillzero]

People with slower connections are having problems cloning my
repository. The server seems to just drop the connection. This
repository is about 300 MB. The server is using git 1.6.1. This is
using an ssh URL. They've tried many times and it usually dies at
different places in the clone (5%, 80%, etc., with one successful
clone so far for them).

remote: Counting objects: 121290, done.
remote: Compressing objects: 100% (82778/82778), done.
fatal: The remote end hung up unexpectedly62.14 MiB | 147 KiB/s
fatal: early EOF
fatal: index-pack failed

Are there any known issues with git 1.6.1 that might cause this? I can
try to get the server updated to 1.6.2.3, but I was curious if anyone
knew of any specific issues like this. The server is running Mac OS X,
if that matters.

Re: git clone failing over slow links?

[Robin H. Johnson]

On Tue, Jun 09, 2009 at 08:52:10AM -0700, skillzero@gmail.com wrote:
> People with slower connections are having problems cloning my
> repository. The server seems to just drop the connection. This
> repository is about 300 MB. The server is using git 1.6.1. This is
> using an ssh URL. They've tried many times and it usually dies at
> different places in the clone (5%, 80%, etc., with one successful
> clone so far for them).
For Gentoo, we saw what I think was same problem early on with our
conversion experiments. In our case however, it's going to be a
non-problem as we intend on disabling initial clone and requiring use of
a bundle.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: git clone failing over slow links?

[Nicolas Pitre]

On Tue, 9 Jun 2009, Robin H. Johnson wrote:

> On Tue, Jun 09, 2009 at 08:52:10AM -0700, skillzero@gmail.com wrote:
> > People with slower connections are having problems cloning my
> > repository. The server seems to just drop the connection. This
> > repository is about 300 MB. The server is using git 1.6.1. This is
> > using an ssh URL. They've tried many times and it usually dies at
> > different places in the clone (5%, 80%, etc., with one successful
> > clone so far for them).
> For Gentoo, we saw what I think was same problem early on with our
> conversion experiments. In our case however, it's going to be a
> non-problem as we intend on disabling initial clone and requiring use of
> a bundle.

In the Gentoo case, I think your problem was rather about the memory 
usage impact of an initial clone and nothing to do with the speed of the 
connection.


Nicolas

Re: git clone failing over slow links?

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

On Tue, Jun 09, 2009 at 02:42:46PM -0400, Nicolas Pitre wrote:
> > For Gentoo, we saw what I think was same problem early on with our
> > conversion experiments. In our case however, it's going to be a
> > non-problem as we intend on disabling initial clone and requiring use of
> > a bundle.
> In the Gentoo case, I think your problem was rather about the memory 
> usage impact of an initial clone and nothing to do with the speed of the 
> connection.
That was true early on, but it still persists for users on slow
connections.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: git clone failing over slow links?

[Nicolas Pitre]

On Tue, 9 Jun 2009, skillzero@gmail.com wrote:

> People with slower connections are having problems cloning my
> repository. The server seems to just drop the connection. This
> repository is about 300 MB. The server is using git 1.6.1. This is
> using an ssh URL. They've tried many times and it usually dies at
> different places in the clone (5%, 80%, etc., with one successful
> clone so far for them).

Do you clearly have a set of people for whom the clone works and another 
set with issues?

> remote: Counting objects: 121290, done.
> remote: Compressing objects: 100% (82778/82778), done.
> fatal: The remote end hung up unexpectedly62.14 MiB | 147 KiB/s

147 KiB/s is not particularly slow either.  What is fast enough for the 
clone to usually succeed?

Also, since this is through an ssh connection, do you have anything in 
your syslog from sshd (usually in /var/log/secure) about anything 
unusual for the connection to go down?

> Are there any known issues with git 1.6.1 that might cause this? I can
> try to get the server updated to 1.6.2.3, but I was curious if anyone
> knew of any specific issues like this. The server is running Mac OS X,
> if that matters.

Maybe it does.  Major sites using git often use Linux and we would have 
heard from them if anything like your problem was seen there.


Nicolas

Re: git clone failing over slow links?

[skillzero]

On Tue, Jun 9, 2009 at 11:55 AM, Nicolas Pitre<nico@cam.org> wrote:
>
> Do you clearly have a set of people for whom the clone works and another
> set with issues?

Yes. Several us are able to reliably clone every time. A few users are
able to clone once or twice (maybe 1 out of every 10 times), but
otherwise, it fails when the server drops the connection.

From what they said, it looked like read_in_full() (in wrapper.c)
isn't getting a full buffer. In two cases, 4077 or 4082 out of 4097
bytes. xread is returning 0 as you'd expect for an EOF.

> 147 KiB/s is not particularly slow either.  What is fast enough for the
> clone to usually succeed?

Mine is 500 KB/sec where it always succeeds. I can try using dummynet
to simulate a slow link though.

> Also, since this is through an ssh connection, do you have anything in
> your syslog from sshd (usually in /var/log/secure) about anything
> unusual for the connection to go down?

I'm trying to get that from the server admin now.