* Cloning does not work on available download bandwidth changes
@ 2024-05-22 15:02 Paweł Bogusławski
2024-05-22 21:52 ` brian m. carlson
0 siblings, 1 reply; 6+ messages in thread
From: Paweł Bogusławski @ 2024-05-22 15:02 UTC (permalink / raw)
To: git
Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to
20mbps down)...
$ git clone https://github.com/googleapis/google-api-go-client
Cloning into 'google-api-go-client'...
remote: Enumerating objects: 644446, done.
remote: Counting objects: 100% (6922/6922), done.
remote: Compressing objects: 100% (2904/2904), done.
Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s
...and then started to watch a VOD movie on same link; when VOD buffers
data, eats almost all available down bandwidth and leaves only about 100
kB/s for git...
Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s
...and when VOD stops buffering and whole bandwith is available for git
again, git transfer starts to grow...
Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s
...but finally git throws an error
error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed
or sometimes:
error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:
CANCEL (err 8)
error: 6109 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
No such problems when downloading bigger file (i.e. linux kernel source)
with wget or curl instead of git clone (wget/curl transfer drops to
about 100 kB/s when VOD buffers and increases to full speed when VOD is
not transferring and transfer finishes successfully).
Sounds like a bug in git; should not throw an error on available
download bandwidth changes as wget and curl do and should not require
any params tuning (to stop users flooding bugtrackers).
git versions tested:
// Debian 11, amd64
$ dpkg -s git | grep Version
Version: 1:2.30.2-1+deb11u2
// Debian 12, amd64
$ dpkg -s git | grep Version
Version: 1:2.39.2-1.1
Similar reports:
https://lore.kernel.org/git/71f2b28b-3e27-4773-8408-2f4c13757b73@gmail.com/
https://lore.kernel.org/git/0d741b90-8307-40cf-b0b3-163203651a57@gmail.com/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes
2024-05-22 15:02 Cloning does not work on available download bandwidth changes Paweł Bogusławski
@ 2024-05-22 21:52 ` brian m. carlson
2024-05-22 22:58 ` Paweł Bogusławski
0 siblings, 1 reply; 6+ messages in thread
From: brian m. carlson @ 2024-05-22 21:52 UTC (permalink / raw)
To: Paweł Bogusławski; +Cc: git
[-- Attachment #1: Type: text/plain, Size: 2937 bytes --]
On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote:
> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to
> 20mbps down)...
>
> $ git clone https://github.com/googleapis/google-api-go-client
> Cloning into 'google-api-go-client'...
> remote: Enumerating objects: 644446, done.
> remote: Counting objects: 100% (6922/6922), done.
> remote: Compressing objects: 100% (2904/2904), done.
> Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s
>
> ...and then started to watch a VOD movie on same link; when VOD buffers
> data, eats almost all available down bandwidth and leaves only about 100
> kB/s for git...
>
> Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s
>
> ...and when VOD stops buffering and whole bandwith is available for git
> again, git transfer starts to grow...
>
> Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s
>
> ...but finally git throws an error
>
> error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s
> fetch-pack: unexpected disconnect while reading sideband packet
> fatal: early EOF
> fatal: index-pack failed
>
> or sometimes:
>
> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:
> CANCEL (err 8)
> error: 6109 bytes of body are still expected
> fetch-pack: unexpected disconnect while reading sideband packet
> fatal: early EOF
> fatal: fetch-pack: invalid index-pack output
>
> No such problems when downloading bigger file (i.e. linux kernel source)
> with wget or curl instead of git clone (wget/curl transfer drops to about
> 100 kB/s when VOD buffers and increases to full speed when VOD is not
> transferring and transfer finishes successfully).
>
> Sounds like a bug in git; should not throw an error on available download
> bandwidth changes as wget and curl do and should not require any params
> tuning (to stop users flooding bugtrackers).
I don't believe this is a bug in Git. The problem here is a network
issue of some sort that's causing the connection to be interrupted or
dropped. This is very common, but the possible causes are many.
A lot of times we see this it's some sort of proxy, and that can be a
non-default antivirus or firewall or TLS MITM device, but that's usually
on Windows, and you're using Debian. It can also be just a bad
connection, poor traffic management by your ISP, or a flaky wireless or
wired network card.
My _guess_ about what's happening here is poor traffic management:
because video is typically flagged as low-latency in DSCP, some device
on the path is prioritizing it to the exclusion of all other traffic,
which is causing the Git connection to be dropped. I have no proof of
this, though.
--
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes
2024-05-22 21:52 ` brian m. carlson
@ 2024-05-22 22:58 ` Paweł Bogusławski
2024-05-23 9:04 ` Jeff King
2024-05-23 12:07 ` rsbecker
0 siblings, 2 replies; 6+ messages in thread
From: Paweł Bogusławski @ 2024-05-22 22:58 UTC (permalink / raw)
To: brian m. carlson, git
W dniu 22.05.2024 o 23:52, brian m. carlson pisze:
> On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote:
>> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to
>> 20mbps down)...
>>
>> $ git clone https://github.com/googleapis/google-api-go-client
>> Cloning into 'google-api-go-client'...
>> remote: Enumerating objects: 644446, done.
>> remote: Counting objects: 100% (6922/6922), done.
>> remote: Compressing objects: 100% (2904/2904), done.
>> Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s
>>
>> ...and then started to watch a VOD movie on same link; when VOD buffers
>> data, eats almost all available down bandwidth and leaves only about 100
>> kB/s for git...
>>
>> Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s
>>
>> ...and when VOD stops buffering and whole bandwith is available for git
>> again, git transfer starts to grow...
>>
>> Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s
>>
>> ...but finally git throws an error
>>
>> error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s
>> fetch-pack: unexpected disconnect while reading sideband packet
>> fatal: early EOF
>> fatal: index-pack failed
>>
>> or sometimes:
>>
>> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:
>> CANCEL (err 8)
>> error: 6109 bytes of body are still expected
>> fetch-pack: unexpected disconnect while reading sideband packet
>> fatal: early EOF
>> fatal: fetch-pack: invalid index-pack output
>>
>> No such problems when downloading bigger file (i.e. linux kernel source)
>> with wget or curl instead of git clone (wget/curl transfer drops to about
>> 100 kB/s when VOD buffers and increases to full speed when VOD is not
>> transferring and transfer finishes successfully).
>>
>> Sounds like a bug in git; should not throw an error on available download
>> bandwidth changes as wget and curl do and should not require any params
>> tuning (to stop users flooding bugtrackers).
> I don't believe this is a bug in Git. The problem here is a network
> issue of some sort that's causing the connection to be interrupted or
> dropped. This is very common, but the possible causes are many.
>
> A lot of times we see this it's some sort of proxy, and that can be a
> non-default antivirus or firewall or TLS MITM device, but that's usually
> on Windows, and you're using Debian. It can also be just a bad
> connection, poor traffic management by your ISP, or a flaky wireless or
> wired network card.
>
> My _guess_ about what's happening here is poor traffic management:
> because video is typically flagged as low-latency in DSCP, some device
> on the path is prioritizing it to the exclusion of all other traffic,
> which is causing the Git connection to be dropped. I have no proof of
> this, though.
Try to run
git clone https://github.com/googleapis/google-api-go-client
and limit interface speed somehow i.e.
tc qdisc add dev eth0 root handle 1: htb default 12
tc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbps ceil 20kbps
tc qdisc add dev eth0 parent 1:12 netem delay 1000ms
and after a while restore speed with i.e.
tc qdisc del dev eth0 parent 1:12 netem delay 1000ms
I tried this in Debian 11 and git 2.30.2 died with:
error: 5723 bytes of body are still expected MiB | 920.00 KiB/s
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes
2024-05-22 22:58 ` Paweł Bogusławski
@ 2024-05-23 9:04 ` Jeff King
2024-05-23 11:15 ` Paweł Bogusławski
2024-05-23 12:07 ` rsbecker
1 sibling, 1 reply; 6+ messages in thread
From: Jeff King @ 2024-05-23 9:04 UTC (permalink / raw)
To: Paweł Bogusławski; +Cc: brian m. carlson, git
On Thu, May 23, 2024 at 12:58:27AM +0200, Paweł Bogusławski wrote:
> Try to run
>
> git clone https://github.com/googleapis/google-api-go-client
>
> and limit interface speed somehow i.e.
>
> tc qdisc add dev eth0 root handle 1: htb default 12
> tc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbps ceil 20kbps
> tc qdisc add dev eth0 parent 1:12 netem delay 1000ms
>
> and after a while restore speed with i.e.
>
> tc qdisc del dev eth0 parent 1:12 netem delay 1000ms
>
> I tried this in Debian 11 and git 2.30.2 died with:
>
> error: 5723 bytes of body are still expected MiB | 920.00 KiB/s
> fetch-pack: unexpected disconnect while reading sideband packet
> fatal: early EOF
> fatal: index-pack failed
I did run it a few times. It usually succeeds, but sometimes fails. When
it does fail, it does so in "slow" mode, without speeding back up. And
I see one message from before your "error:" line, which is a report that
curl reported that the connection dropped (curl is shuttling data back
from the server in the form of "packet lines", each of which is prefixed
with its expected size; the "N bytes of body are still expected" is just
that parser seeing a truncated response from curl).
But I don't think any of this is due to what Git is doing on the client
side. From the client view, the server sends a TCP RST (which I observed
with wireshark), and the connection drops. I'd suspect the drop is not
even from the application level on the server side, but rather that
something in the network path was mad about the delayed packets (or
possibly the spam of duplicated TCP acks that I sometimes saw).
So I really think this is a network issue, and not a Git issue in
particular. It's possible that the networking stack that github.com uses
is more picky about the things I mentioned above. But again, not a Git
issue.
-Peff
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes
2024-05-23 9:04 ` Jeff King
@ 2024-05-23 11:15 ` Paweł Bogusławski
0 siblings, 0 replies; 6+ messages in thread
From: Paweł Bogusławski @ 2024-05-23 11:15 UTC (permalink / raw)
To: Jeff King; +Cc: brian m. carlson, git
W dniu 23.05.2024 o 11:04, Jeff King pisze:
> I did run it a few times. It usually succeeds, but sometimes fails. When
> it does fail, it does so in "slow" mode, without speeding back up. And
> I see one message from before your "error:" line, which is a report that
> curl reported that the connection dropped (curl is shuttling data back
> from the server in the form of "packet lines", each of which is prefixed
> with its expected size; the "N bytes of body are still expected" is just
> that parser seeing a truncated response from curl).
>
> But I don't think any of this is due to what Git is doing on the client
> side. From the client view, the server sends a TCP RST (which I observed
> with wireshark), and the connection drops. I'd suspect the drop is not
> even from the application level on the server side, but rather that
> something in the network path was mad about the delayed packets (or
> possibly the spam of duplicated TCP acks that I sometimes saw).
>
> So I really think this is a network issue, and not a Git issue in
> particular. It's possible that the networking stack that github.com uses
> is more picky about the things I mentioned above. But again, not a Git
> issue.
Checked once again with...
$ export GIT_TRACE_PACKET=1
$ export GIT_TRACE=1
$ export GIT_CURL_VERBOSE=1
...and git dies with...
[...]
remote: Counting objects: 100% (6960/6960), done.
remote: Compressing objects: 100% (2926/2926), done.
12:55:29.776311 http.c:756 == Info: Connection #0 to host github.com left intact
error: 3805 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed
...a few secs after interface speed is restored (git transfer increases a few secs).
In tcpdump of above test I can see FIN sent by client (192.168.100.100) first:
# tcpdump -i eth0 -nnvvXSs 1514 -w test.pcap 'tcp and src 140.82.121.4 or dst 140.82.121.4'
[...]
21346 2024-05-23 12:55:29,776138 140.82.121.4 192.168.100.100 TLSv1.3 959 Continuation Data
21347 2024-05-23 12:55:29,776151 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [ACK] Seq=6192 Ack=19611541 Win=65535 Len=0
21348 2024-05-23 12:55:29,776702 192.168.100.100 140.82.121.4 TLSv1.3 78 Application Data
>>> here git client sends FIN but transfer is not completed <<<<
21349 2024-05-23 12:55:29,778374 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [FIN, ACK] Seq=6216 Ack=19611541 Win=65535 Len=0
21350 2024-05-23 12:55:29,836364 140.82.121.4 192.168.100.100 TLSv1.3 78 Application Data
21351 2024-05-23 12:55:29,836364 140.82.121.4 192.168.100.100 TCP 54 443 → 40624 [FIN, ACK] Seq=19611565 Ack=6216 Win=65535 Len=0
21352 2024-05-23 12:55:29,836421 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [RST] Seq=6216 Win=0 Len=0
21353 2024-05-23 12:55:29,836441 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [RST] Seq=6216 Win=0 Len=0
21354 2024-05-23 12:55:29,838428 140.82.121.4 192.168.100.100 TCP 54 443 → 40624 [ACK] Seq=19611566 Ack=6217 Win=65535 Len=0
21355 2024-05-23 12:55:29,838448 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [RST] Seq=6217 Win=0 Len=0
Cloning
$ git clone https://gitlab.com/gitlab-org/gitlab
$ git clone https://codeberg.org/forgejo/forgejo
does not generate such problem after interface speed changes so maybe it's something specific for git <---> github.com communication.
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Cloning does not work on available download bandwidth changes
2024-05-22 22:58 ` Paweł Bogusławski
2024-05-23 9:04 ` Jeff King
@ 2024-05-23 12:07 ` rsbecker
1 sibling, 0 replies; 6+ messages in thread
From: rsbecker @ 2024-05-23 12:07 UTC (permalink / raw)
To: 'Paweł Bogusławski', 'brian m. carlson',
git
On Wednesday, May 22, 2024 6:58 PM, Paweł Bogusławski wrote:
>W dniu 22.05.2024 o 23:52, brian m. carlson pisze:
>
>> On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote:
>>> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up
>>> to 20mbps down)...
>>>
>>> $ git clone https://github.com/googleapis/google-api-go-client
>>> Cloning into 'google-api-go-client'...
>>> remote: Enumerating objects: 644446, done.
>>> remote: Counting objects: 100% (6922/6922), done.
>>> remote: Compressing objects: 100% (2904/2904), done.
>>> Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s
>>>
>>> ...and then started to watch a VOD movie on same link; when VOD
>>> buffers data, eats almost all available down bandwidth and leaves
>>> only about 100 kB/s for git...
>>>
>>> Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s
>>>
>>> ...and when VOD stops buffering and whole bandwith is available for
>>> git again, git transfer starts to grow...
>>>
>>> Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s
>>>
>>> ...but finally git throws an error
>>>
>>> error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s
>>> fetch-pack: unexpected disconnect while reading sideband packet
>>> fatal: early EOF
>>> fatal: index-pack failed
>>>
>>> or sometimes:
>>>
>>> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:
>>> CANCEL (err 8)
>>> error: 6109 bytes of body are still expected
>>> fetch-pack: unexpected disconnect while reading sideband packet
>>> fatal: early EOF
>>> fatal: fetch-pack: invalid index-pack output
>>>
>>> No such problems when downloading bigger file (i.e. linux kernel
>>> source) with wget or curl instead of git clone (wget/curl transfer
>>> drops to about
>>> 100 kB/s when VOD buffers and increases to full speed when VOD is not
>>> transferring and transfer finishes successfully).
>>>
>>> Sounds like a bug in git; should not throw an error on available
>>> download bandwidth changes as wget and curl do and should not require
>>> any params tuning (to stop users flooding bugtrackers).
>> I don't believe this is a bug in Git. The problem here is a network
>> issue of some sort that's causing the connection to be interrupted or
>> dropped. This is very common, but the possible causes are many.
>>
>> A lot of times we see this it's some sort of proxy, and that can be a
>> non-default antivirus or firewall or TLS MITM device, but that's
>> usually on Windows, and you're using Debian. It can also be just a
>> bad connection, poor traffic management by your ISP, or a flaky
>> wireless or wired network card.
>>
>> My _guess_ about what's happening here is poor traffic management:
>> because video is typically flagged as low-latency in DSCP, some device
>> on the path is prioritizing it to the exclusion of all other traffic,
>> which is causing the Git connection to be dropped. I have no proof of
>> this, though.
>
>Try to run
>
>git clone https://github.com/googleapis/google-api-go-client
>
>and limit interface speed somehow i.e.
>
>tc qdisc add dev eth0 root handle 1: htb default 12 tc class add dev eth0 parent 1:1
>classid 1:12 htb rate 20kbps ceil 20kbps tc qdisc add dev eth0 parent 1:12 netem
>delay 1000ms
>
>and after a while restore speed with i.e.
>
>tc qdisc del dev eth0 parent 1:12 netem delay 1000ms
>
>I tried this in Debian 11 and git 2.30.2 died with:
>
>error: 5723 bytes of body are still expected MiB | 920.00 KiB/s
>fetch-pack: unexpected disconnect while reading sideband packet
>fatal: early EOF
>fatal: index-pack failed
I have tried this from my home ISP. It does a little throttling but sustained between 5-6 MiB/s for most of the transfer. We have seen related issues around ISP throttling that causes connections to drop for very large repositories. However, in the test I just ran, both git and https are happy to complete.
$ git clone https://github.com/googleapis/google-api-go-client
Cloning into 'google-api-go-client'...
remote: Enumerating objects: 644446, done.
remote: Counting objects: 100% (6922/6922), done.
remote: Compressing objects: 100% (2904/2904), done.
remote: Total 644446 (delta 3722), reused 6567 (delta 3445), pack-reused 637524
Receiving objects: 100% (644446/644446), 1.46 GiB | 7.45 MiB/s, done.
Resolving deltas: 100% (392971/392971), done.
Updating files: 100% (1372/1372), done.
$ git version
git version 2.43.0
Have you tried using SSH to do the clone? Assuming you have your public key registered on GitHub: git clone ssh://git@github.com/googleapis/google-api-go-client
This runs about slightly slower.
$ git clone ssh://git@github.com/googleapis/google-api-go-client
Cloning into 'google-api-go-client'...
remote: Enumerating objects: 644446, done.
remote: Counting objects: 100% (6922/6922), done.
remote: Compressing objects: 100% (2911/2911), done.
remote: Total 644446 (delta 3720), reused 6563 (delta 3438), pack-reused 637524
Receiving objects: 100% (644446/644446), 1.46 GiB | 6.31 MiB/s, done.
Resolving deltas: 100% (392969/392969), done.
Updating files: 100% (1372/1372), done.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-05-23 12:15 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-22 15:02 Cloning does not work on available download bandwidth changes Paweł Bogusławski
2024-05-22 21:52 ` brian m. carlson
2024-05-22 22:58 ` Paweł Bogusławski
2024-05-23 9:04 ` Jeff King
2024-05-23 11:15 ` Paweł Bogusławski
2024-05-23 12:07 ` rsbecker
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).