Remote hung up during `git fetch`

Remote hung up during `git fetch`

[Yichao Yu]

Hi everyone,

I want to build packages for snap shoot of different branches from
different remote git repositories in the same local directory (so that
I don't need to recompile everything everytime.) and I am using a
combination of `git clone/checkout/reset/fetch` to do that. However,
during git-fetch, the remote sometimes stop responding or simply reset
the connection. This happens occasionally at least for both ssh and
git protocol (not sure about http/https) on github, bitbucket and also
kernel.org so I think it is probably not due to a weird behavior of a
certain host. Does anyone know the reason or is there anything I have
done wrong? And is there a better way to set the local tree to a
certain branch at a certain url? THX

My git version is ArchLinux package 1.8.0-1. (timezone
America/New_York in case the time stamp somehow matters)

Here is a script that always triggers the issue (at least now) and
it's output. (No I am not trying to merge git and the kernel... These
are just random public repos on kernel.org that can trigger the issue.
Although I am pulling from two repos from different project here, the
same thing can also happen on other hosts when the two repos are
actually the same project)

Yichao Yu

------------------------------------------------------------------

#!/bin/bash

repo_name=git
# remote1='git://github.com/torvalds/linux.git'
remote1='git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git'
branch1='master'
# remote2='git://github.com/git/git.git'
remote2='git://git.kernel.org/pub/scm/git/git.git'
branch2='next'

git clone --depth 1 --single-branch --branch "$branch1" "$remote1" "$repo_name"
cd "$repo_name"
git fetch -vvv "$remote2" # "$branch2:$branch2"

-----------------------------------------------

Cloning into 'git'...
remote: Counting objects: 43215, done.
remote: Compressing objects: 100% (41422/41422), done.
remote: Total 43215 (delta 3079), reused 22032 (delta 1247)
Receiving objects: 100% (43215/43215), 119.06 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (3079/3079), done.
Checking out files: 100% (40905/40905), done.
fatal: destination path 'git' already exists and is not an empty directory.
Server supports multi_ack_detailed
Server supports side-band-64k
Server supports ofs-delta
want 2d242fb3fc19fc9ba046accdd9210be8b9913f64 (HEAD)
have ef6c5be658f6a70c1256fbd18e18ee0dc24c3386
have db9d8c60266a5010e905829e10cd722519e14777
done
fatal: The remote end hung up unexpectedly

Fwd: Remote hung up during `git fetch`

[Yichao Yu]

Hi everyone,

I sent this email yesterday to the git mailing list but I cannot find
it in any archive so I decide to send it again.
Does anyone know what has happened to the mailing list? I haven't
receive any email from several kernel related busy mailing lists for
several hours....

Yichao Yu

---------- Forwarded message ----------
From: Yichao Yu <yyc1992@gmail.com>
Date: Wed, Nov 21, 2012 at 11:18 PM
Subject: Remote hung up during `git fetch`
To: git@vger.kernel.org


Hi everyone,

I want to build packages for snap shoot of different branches from
different remote git repositories in the same local directory (so that
I don't need to recompile everything everytime.) and I am using a
combination of `git clone/checkout/reset/fetch` to do that. However,
during git-fetch, the remote sometimes stop responding or simply reset
the connection. This happens occasionally at least for both ssh and
git protocol (not sure about http/https) on github, bitbucket and also
kernel.org so I think it is probably not due to a weird behavior of a
certain host. Does anyone know the reason or is there anything I have
done wrong? And is there a better way to set the local tree to a
certain branch at a certain url? THX

My git version is ArchLinux package 1.8.0-1. (timezone
America/New_York in case the time stamp somehow matters)

Here is a script that always triggers the issue (at least now) and
it's output. (No I am not trying to merge git and the kernel... These
are just random public repos on kernel.org that can trigger the issue.
Although I am pulling from two repos from different project here, the
same thing can also happen on other hosts when the two repos are
actually the same project)

Yichao Yu

------------------------------------------------------------------

#!/bin/bash

repo_name=git
# remote1='git://github.com/torvalds/linux.git'
remote1='git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git'
branch1='master'
# remote2='git://github.com/git/git.git'
remote2='git://git.kernel.org/pub/scm/git/git.git'
branch2='next'

git clone --depth 1 --single-branch --branch "$branch1" "$remote1" "$repo_name"
cd "$repo_name"
git fetch -vvv "$remote2" # "$branch2:$branch2"

-----------------------------------------------

Cloning into 'git'...
remote: Counting objects: 43215, done.
remote: Compressing objects: 100% (41422/41422), done.
remote: Total 43215 (delta 3079), reused 22032 (delta 1247)
Receiving objects: 100% (43215/43215), 119.06 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (3079/3079), done.
Checking out files: 100% (40905/40905), done.
fatal: destination path 'git' already exists and is not an empty directory.
Server supports multi_ack_detailed
Server supports side-band-64k
Server supports ofs-delta
want 2d242fb3fc19fc9ba046accdd9210be8b9913f64 (HEAD)
have ef6c5be658f6a70c1256fbd18e18ee0dc24c3386
have db9d8c60266a5010e905829e10cd722519e14777
done
fatal: The remote end hung up unexpectedly

Re: Remote hung up during `git fetch`

[Shawn Pearce]

On Thu, Nov 22, 2012 at 10:39 AM, Yichao Yu <yyc1992@gmail.com> wrote:
> I sent this email yesterday to the git mailing list but I cannot find
> it in any archive so I decide to send it again.

If it was HTML formatted it would have been silently dropped by the list.

> Does anyone know what has happened to the mailing list? I haven't
> receive any email from several kernel related busy mailing lists for
> several hours....

US holiday today? The list traffic tends to be down during holidays.

> I want to build packages for snap shoot of different branches from
> different remote git repositories in the same local directory (so that
> I don't need to recompile everything everytime.) and I am using a
> combination of `git clone/checkout/reset/fetch` to do that. However,
> during git-fetch, the remote sometimes stop responding or simply reset
> the connection. This happens occasionally at least for both ssh and
> git protocol (not sure about http/https) on github, bitbucket and also
> kernel.org so I think it is probably not due to a weird behavior of a
> certain host. Does anyone know the reason or is there anything I have
> done wrong? And is there a better way to set the local tree to a
> certain branch at a certain url? THX

If the remote server is really busy it might be OOM'ing the server
process which would disconnect the client. Or maybe its your ISP
sending a rogue RST packet to kill the connection because they don't
like traffic that leaves their network and costs them money on a
peering agreement. Or... ?

> #!/bin/bash
>
> repo_name=git
> # remote1='git://github.com/torvalds/linux.git'
> remote1='git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git'
> branch1='master'
> # remote2='git://github.com/git/git.git'
> remote2='git://git.kernel.org/pub/scm/git/git.git'
> branch2='next'
>
> git clone --depth 1 --single-branch --branch "$branch1" "$remote1" "$repo_name"
> cd "$repo_name"
> git fetch -vvv "$remote2" # "$branch2:$branch2"
>
> -----------------------------------------------
>
> Cloning into 'git'...
> remote: Counting objects: 43215, done.
> remote: Compressing objects: 100% (41422/41422), done.
> remote: Total 43215 (delta 3079), reused 22032 (delta 1247)
> Receiving objects: 100% (43215/43215), 119.06 MiB | 1.60 MiB/s, done.
> Resolving deltas: 100% (3079/3079), done.
> Checking out files: 100% (40905/40905), done.
> fatal: destination path 'git' already exists and is not an empty directory.

Why does this error come up? It looks like git already exists locally.
Git should have aborted much earlier in clone when the directory
exists.

> Server supports multi_ack_detailed
> Server supports side-band-64k
> Server supports ofs-delta
> want 2d242fb3fc19fc9ba046accdd9210be8b9913f64 (HEAD)
> have ef6c5be658f6a70c1256fbd18e18ee0dc24c3386
> have db9d8c60266a5010e905829e10cd722519e14777
> done
> fatal: The remote end hung up unexpectedly

This looks like its from the fetch command. Since the server doesn't
report any errors to the client, its hard to say why the server just
gave up right there. I wonder if this is related to the fact that you
did a shallow clone initially. The shallow clone may have confused the
server when fetch ran because it only sent 2 have lines and gave up.

Try exporting GIT_TRACE_PACKET=1 and seeing if you can get more
detailed information from the protocol on the client side.

FYI, https://kernel.googlesource.com/ mirrors git://git.kernel.org/ so
you can also try pulling from that server (e.g.
https://kernel.googlesource.com/pub/scm/git/git.git).

Re: Remote hung up during `git fetch`

[Yichao Yu]

On Thu, Nov 22, 2012 at 2:01 PM, Shawn Pearce <spearce@spearce.org> wrote:
> On Thu, Nov 22, 2012 at 10:39 AM, Yichao Yu <yyc1992@gmail.com> wrote:
>> I sent this email yesterday to the git mailing list but I cannot find
>> it in any archive so I decide to send it again.
>
> If it was HTML formatted it would have been silently dropped by the list.
Well, I simply forward the original email and both of them are plain
text I believe....


>
>> Does anyone know what has happened to the mailing list? I haven't
>> receive any email from several kernel related busy mailing lists for
>> several hours....
>
> US holiday today? The list traffic tends to be down during holidays.
This silent?.... 0 email from the kernel mailing list for 10+ hours?..
anyway.... nvm...

>
>> I want to build packages for snap shoot of different branches from
>> different remote git repositories in the same local directory (so that
>> I don't need to recompile everything everytime.) and I am using a
>> combination of `git clone/checkout/reset/fetch` to do that. However,
>> during git-fetch, the remote sometimes stop responding or simply reset
>> the connection. This happens occasionally at least for both ssh and
>> git protocol (not sure about http/https) on github, bitbucket and also
>> kernel.org so I think it is probably not due to a weird behavior of a
>> certain host. Does anyone know the reason or is there anything I have
>> done wrong? And is there a better way to set the local tree to a
>> certain branch at a certain url? THX
>
> If the remote server is really busy it might be OOM'ing the server
Well, clone succeeded but fetch didn't?...

> process which would disconnect the client. Or maybe its your ISP
> sending a rogue RST packet to kill the connection because they don't
> like traffic that leaves their network and costs them money on a
> peering agreement. Or... ?
Well, yes I have only tested here but I don't really think MIT is
doing that... No?

>
>> #!/bin/bash
>>
>> repo_name=git
>> # remote1='git://github.com/torvalds/linux.git'
>> remote1='git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git'
>> branch1='master'
>> # remote2='git://github.com/git/git.git'
>> remote2='git://git.kernel.org/pub/scm/git/git.git'
>> branch2='next'
>>
>> git clone --depth 1 --single-branch --branch "$branch1" "$remote1" "$repo_name"
>> cd "$repo_name"
>> git fetch -vvv "$remote2" # "$branch2:$branch2"
>>
>> -----------------------------------------------
>>
>> Cloning into 'git'...
>> remote: Counting objects: 43215, done.
>> remote: Compressing objects: 100% (41422/41422), done.
>> remote: Total 43215 (delta 3079), reused 22032 (delta 1247)
>> Receiving objects: 100% (43215/43215), 119.06 MiB | 1.60 MiB/s, done.
>> Resolving deltas: 100% (3079/3079), done.
>> Checking out files: 100% (40905/40905), done.
>> fatal: destination path 'git' already exists and is not an empty directory.
>
> Why does this error come up? It looks like git already exists locally.
> Git should have aborted much earlier in clone when the directory
> exists.
Ahh, sorry. I killed the previous script to use LANG=C but forgot to
remove the directory which is normally deleted automatically by the
last few lines of my script that I didn't paste here (cd and rm...).
So here is the right output.

---------------------------------------

Cloning into 'git'...
remote: Counting objects: 43215, done.
remote: Compressing objects: 100% (41071/41071), done.
remote: Total 43215 (delta 3699), reused 13345 (delta 1598)
Receiving objects: 100% (43215/43215), 116.91 MiB | 4.51 MiB/s, done.
Resolving deltas: 100% (3699/3699), done.
Checking out files: 100% (40905/40905), done.
Server supports multi_ack_detailed
Server supports side-band-64k
Server supports ofs-delta
want 2d242fb3fc19fc9ba046accdd9210be8b9913f64 (HEAD)
have ef6c5be658f6a70c1256fbd18e18ee0dc24c3386
have db9d8c60266a5010e905829e10cd722519e14777
done
fatal: read error: Connection reset by peer

----------------------------------------------------

>
>> Server supports multi_ack_detailed
>> Server supports side-band-64k
>> Server supports ofs-delta
>> want 2d242fb3fc19fc9ba046accdd9210be8b9913f64 (HEAD)
>> have ef6c5be658f6a70c1256fbd18e18ee0dc24c3386
>> have db9d8c60266a5010e905829e10cd722519e14777
>> done
>> fatal: The remote end hung up unexpectedly
>
> This looks like its from the fetch command. Since the server doesn't
yes, plz simply ignore the previous one....

> report any errors to the client, its hard to say why the server just
> gave up right there. I wonder if this is related to the fact that you
> did a shallow clone initially. The shallow clone may have confused the
> server when fetch ran because it only sent 2 have lines and gave up.
Probably (related to shallow clone...) but hopefully this is not sth I
shouldn't do?
And a shallow clone is really what I want in my case. I really don't
need the full history to compile...

>
> Try exporting GIT_TRACE_PACKET=1 and seeing if you can get more
> detailed information from the protocol on the client side.

So here it is, I have removed lots of tags fetching (hopefully they
are irrelevant). Is there anything that looks strange (Looks like it's
actually .... done???)? Any more information needed?

----------------------------------------------

Cloning into 'git'...
remote: Counting objects: 43385, done.
remote: Compressing objects: 100% (41117/41117), done.
Receiving objects: 100% (43385/43385), 117.05 MiB | 4.73 MiB/s, done.
remote: Total 43385 (delta 3862), reused 13596 (delta 1722)
Resolving deltas: 100% (3862/3862), done.
Checking out files: 100% (40905/40905), done.
packet:        fetch> git-upload-pack
/pub/scm/git/git.git\0host=git.kernel.org\0
packet:        fetch< 2d242fb3fc19fc9ba046accdd9210be8b9913f64
HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow
no-progress include-tag multi_ack_detailed
packet:        fetch< 1c03999bef32edabd3db6479279b3ef90cc635ca refs/heads/maint
packet:        fetch< 2d242fb3fc19fc9ba046accdd9210be8b9913f64 refs/heads/master
packet:        fetch< ce6fbe5b9dc5e1e8ce1499bd8f7fb982e0c68289 refs/heads/next
packet:        fetch< 7b4a70c62f3a83fbd8b44bf712141754a5f64205 refs/heads/pu
packet:        fetch< 2c806864e99fc1252cea652ab0fc837067e726ba refs/heads/todo
packet:        fetch< d5aef6e4d58cfe1549adef5b436f3ace984e8c86
refs/tags/gitgui-0.10.0
packet:        fetch< 3d654be48f65545c4d3e35f5d3bbed5489820930
refs/tags/gitgui-0.10.0^{}
packet:        fetch< 33682a5e98adfd8ba4ce0e21363c443bd273eb77
refs/tags/gitgui-0.10.1
packet:        fetch< 729ffa50f75a025935623bfc58d0932c65f7de2f
refs/tags/gitgui-0.10.1^{}

... tags... and tags... and tags.....

packet:        fetch< c15295d7477ccec489953299bd03a8e62f86e611
refs/tags/v1.8.0-rc2
packet:        fetch< cd46259ebf2e624bcee2aaae05c36663d414e1a2
refs/tags/v1.8.0-rc2^{}
packet:        fetch< 22ed067acc84eac8a0a72d20478a18aee4e25571
refs/tags/v1.8.0-rc3
packet:        fetch< 87a5461fa7b30f7b7baf27204f10219d61500fbf
refs/tags/v1.8.0-rc3^{}
packet:        fetch< 0000
Server supports multi_ack_detailed
Server supports side-band-64k
Server supports ofs-delta
want 2d242fb3fc19fc9ba046accdd9210be8b9913f64 (HEAD)
packet:        fetch> want 2d242fb3fc19fc9ba046accdd9210be8b9913f64
multi_ack_detailed side-band-64k thin-pack ofs-delta
packet:        fetch> shallow 65546ab097b023886a60df4cbc931d4cc362b98e
packet:        fetch> shallow b80d60e1c3854460a1f01d4110da5ae98f30f9b2
packet:        fetch> 0000
packet:        fetch> have 5a903166dd31a42e39283b075cc6b9a9c079d1af
have 5a903166dd31a42e39283b075cc6b9a9c079d1af
packet:        fetch> have b80d60e1c3854460a1f01d4110da5ae98f30f9b2
have b80d60e1c3854460a1f01d4110da5ae98f30f9b2
packet:        fetch> have 65546ab097b023886a60df4cbc931d4cc362b98e
have 65546ab097b023886a60df4cbc931d4cc362b98e
packet:        fetch> done
done
fatal: read error: Connection reset by peer

----------------------------------------------------------

>
> FYI, https://kernel.googlesource.com/ mirrors git://git.kernel.org/ so
> you can also try pulling from that server (e.g.
> https://kernel.googlesource.com/pub/scm/git/git.git).
THX. I have no problem cloning/pulling either the git or the kernel
tree from anywhere. It only happens when I want to do this kind of
fetching. I think git.kernel.org should be the best place to
demonstrate this kind of error since you may know exactly what is
running on the server. (And you probably don't want to try reproducing
sth on my private ssh git server?.) =P

Re: Remote hung up during `git fetch`

[Shawn Pearce]

On Thu, Nov 22, 2012 at 11:44 AM, Yichao Yu <yyc1992@gmail.com> wrote:
>> US holiday today? The list traffic tends to be down during holidays.
> This silent?.... 0 email from the kernel mailing list for 10+ hours?..
> anyway.... nvm...

Check your spam filters? I am having no trouble getting email for the
Git list. Traffic is down, but there have been several messages within
the past 4 hours. E.g. this thread among them.

> packet:        fetch> want 2d242fb3fc19fc9ba046accdd9210be8b9913f64
> multi_ack_detailed side-band-64k thin-pack ofs-delta
> packet:        fetch> shallow 65546ab097b023886a60df4cbc931d4cc362b98e
> packet:        fetch> shallow b80d60e1c3854460a1f01d4110da5ae98f30f9b2
> packet:        fetch> 0000

I think this is the problem. Your client told the sever it has the
Linux kernel shallow cloned, but its talking to a repository that
hosts git.git. The remote server doesn't know these two SHA-1s
mentioned on the shallow line, as they are from the Linux kernel
repository, so the server just hung up on you.

Basically this is an unsupported use case. A shallow repository can
only fetch from the place it cloned from. Anything else working is
pure luck. It _may_ be able to fetch from a clone of that same
repository at another server, if the clone has at least all of the
commits the client already has. If the remote clone is missing commits
(as in this case where it has none!) then it doesn't work.

Re: Remote hung up during `git fetch`

[Yichao Yu]

On Thu, Nov 22, 2012 at 2:52 PM, Shawn Pearce <spearce@spearce.org> wrote:
> On Thu, Nov 22, 2012 at 11:44 AM, Yichao Yu <yyc1992@gmail.com> wrote:
>>> US holiday today? The list traffic tends to be down during holidays.
>> This silent?.... 0 email from the kernel mailing list for 10+ hours?..
>> anyway.... nvm...
>
> Check your spam filters? I am having no trouble getting email for the
> Git list. Traffic is down, but there have been several messages within
> the past 4 hours. E.g. this thread among them.

My spam filter is fine (they are not in spam...)... probably gmail
just failed to send to / receive from vger.kernel.org for the last
several hours (or sth similar...)....

>
>> packet:        fetch> want 2d242fb3fc19fc9ba046accdd9210be8b9913f64
>> multi_ack_detailed side-band-64k thin-pack ofs-delta
>> packet:        fetch> shallow 65546ab097b023886a60df4cbc931d4cc362b98e
>> packet:        fetch> shallow b80d60e1c3854460a1f01d4110da5ae98f30f9b2
>> packet:        fetch> 0000
>
> I think this is the problem. Your client told the sever it has the
> Linux kernel shallow cloned, but its talking to a repository that
> hosts git.git. The remote server doesn't know these two SHA-1s
> mentioned on the shallow line, as they are from the Linux kernel
> repository, so the server just hung up on you.
I C. So in my real case it is probably because the different server I
am pulling from are on different branches.... (for a shallow clone, it
may look the same with commits from different projects?...)

>
> Basically this is an unsupported use case. A shallow repository can
> only fetch from the place it cloned from. Anything else working is
> pure luck. It _may_ be able to fetch from a clone of that same
> repository at another server, if the clone has at least all of the
> commits the client already has. If the remote clone is missing commits
> (as in this case where it has none!) then it doesn't work.

So is there a way to ask for a certain commit from a certain server
and update local files that has changed accordingly? For the server,
it shouldn't be much different from another shallow clone (although it
would be better if locally existing objects are not transferred.). But
I am wondering what client side command/script do I need to use.


THX.

Re: Remote hung up during `git fetch`

[Andrew Ardill]

On 23 November 2012 05:39, Yichao Yu <yyc1992@gmail.com> wrote:
> Hi everyone,
>
> I sent this email yesterday to the git mailing list but I cannot find
> it in any archive so I decide to send it again.
> Does anyone know what has happened to the mailing list? I haven't
> receive any email from several kernel related busy mailing lists for
> several hours....
>
> Yichao Yu

Your original message just came through to me (I'm on GMail) so
obviously there was a delay somewhere in routing your message/s. Looks
like it was delayed by around 18 hours somewhere...
Looking at your delayed message more closely I can see that there was
a big delay just before vger.kernel.org got it.

From: Yichao Yu <yyc1992@gmail.com>
Date: Wed, 21 Nov 2012 23:18:34 -0500
Received: by 10.64.15.165 with HTTP; Wed, 21 Nov 2012 20:18:34 -0800 (PST)
Received: by 10.50.12.165 with SMTP id
z5mr1895031igb.17.1353557934382; Wed,  21 Nov 2012 20:18:54 -0800
(PST)
Received: by mail-ie0-f174.google.com with SMTP id k11so2625936iea.19
for <git@vger.kernel.org>; Thu, 22 Nov 2012 15:00:04 -0800 (PST)

Not sure if anyone else saw issues, but it is likely an issue with
your service provider (either gmail or gmail's SMTP routers). My other
gmail traffic has been fine over the period. Maybe somebody else knows
what happened :)

Regards,

Andrew Ardill