* Cloning does not work on available download bandwidth changes @ 2024-05-22 15:02 Paweł Bogusławski 2024-05-22 21:52 ` brian m. carlson 0 siblings, 1 reply; 6+ messages in thread From: Paweł Bogusławski @ 2024-05-22 15:02 UTC (permalink / raw) To: git Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to 20mbps down)... $ git clone https://github.com/googleapis/google-api-go-client Cloning into 'google-api-go-client'... remote: Enumerating objects: 644446, done. remote: Counting objects: 100% (6922/6922), done. remote: Compressing objects: 100% (2904/2904), done. Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s ...and then started to watch a VOD movie on same link; when VOD buffers data, eats almost all available down bandwidth and leaves only about 100 kB/s for git... Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s ...and when VOD stops buffering and whole bandwith is available for git again, git transfer starts to grow... Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s ...but finally git throws an error error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed or sometimes: error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) error: 6109 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output No such problems when downloading bigger file (i.e. linux kernel source) with wget or curl instead of git clone (wget/curl transfer drops to about 100 kB/s when VOD buffers and increases to full speed when VOD is not transferring and transfer finishes successfully). Sounds like a bug in git; should not throw an error on available download bandwidth changes as wget and curl do and should not require any params tuning (to stop users flooding bugtrackers). git versions tested: // Debian 11, amd64 $ dpkg -s git | grep Version Version: 1:2.30.2-1+deb11u2 // Debian 12, amd64 $ dpkg -s git | grep Version Version: 1:2.39.2-1.1 Similar reports: https://lore.kernel.org/git/71f2b28b-3e27-4773-8408-2f4c13757b73@gmail.com/ https://lore.kernel.org/git/0d741b90-8307-40cf-b0b3-163203651a57@gmail.com/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes 2024-05-22 15:02 Cloning does not work on available download bandwidth changes Paweł Bogusławski @ 2024-05-22 21:52 ` brian m. carlson 2024-05-22 22:58 ` Paweł Bogusławski 0 siblings, 1 reply; 6+ messages in thread From: brian m. carlson @ 2024-05-22 21:52 UTC (permalink / raw) To: Paweł Bogusławski; +Cc: git [-- Attachment #1: Type: text/plain, Size: 2937 bytes --] On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote: > Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to > 20mbps down)... > > $ git clone https://github.com/googleapis/google-api-go-client > Cloning into 'google-api-go-client'... > remote: Enumerating objects: 644446, done. > remote: Counting objects: 100% (6922/6922), done. > remote: Compressing objects: 100% (2904/2904), done. > Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s > > ...and then started to watch a VOD movie on same link; when VOD buffers > data, eats almost all available down bandwidth and leaves only about 100 > kB/s for git... > > Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s > > ...and when VOD stops buffering and whole bandwith is available for git > again, git transfer starts to grow... > > Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s > > ...but finally git throws an error > > error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s > fetch-pack: unexpected disconnect while reading sideband packet > fatal: early EOF > fatal: index-pack failed > > or sometimes: > > error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: > CANCEL (err 8) > error: 6109 bytes of body are still expected > fetch-pack: unexpected disconnect while reading sideband packet > fatal: early EOF > fatal: fetch-pack: invalid index-pack output > > No such problems when downloading bigger file (i.e. linux kernel source) > with wget or curl instead of git clone (wget/curl transfer drops to about > 100 kB/s when VOD buffers and increases to full speed when VOD is not > transferring and transfer finishes successfully). > > Sounds like a bug in git; should not throw an error on available download > bandwidth changes as wget and curl do and should not require any params > tuning (to stop users flooding bugtrackers). I don't believe this is a bug in Git. The problem here is a network issue of some sort that's causing the connection to be interrupted or dropped. This is very common, but the possible causes are many. A lot of times we see this it's some sort of proxy, and that can be a non-default antivirus or firewall or TLS MITM device, but that's usually on Windows, and you're using Debian. It can also be just a bad connection, poor traffic management by your ISP, or a flaky wireless or wired network card. My _guess_ about what's happening here is poor traffic management: because video is typically flagged as low-latency in DSCP, some device on the path is prioritizing it to the exclusion of all other traffic, which is causing the Git connection to be dropped. I have no proof of this, though. -- brian m. carlson (they/them or he/him) Toronto, Ontario, CA [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 262 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes 2024-05-22 21:52 ` brian m. carlson @ 2024-05-22 22:58 ` Paweł Bogusławski 2024-05-23 9:04 ` Jeff King 2024-05-23 12:07 ` rsbecker 0 siblings, 2 replies; 6+ messages in thread From: Paweł Bogusławski @ 2024-05-22 22:58 UTC (permalink / raw) To: brian m. carlson, git W dniu 22.05.2024 o 23:52, brian m. carlson pisze: > On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote: >> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to >> 20mbps down)... >> >> $ git clone https://github.com/googleapis/google-api-go-client >> Cloning into 'google-api-go-client'... >> remote: Enumerating objects: 644446, done. >> remote: Counting objects: 100% (6922/6922), done. >> remote: Compressing objects: 100% (2904/2904), done. >> Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s >> >> ...and then started to watch a VOD movie on same link; when VOD buffers >> data, eats almost all available down bandwidth and leaves only about 100 >> kB/s for git... >> >> Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s >> >> ...and when VOD stops buffering and whole bandwith is available for git >> again, git transfer starts to grow... >> >> Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s >> >> ...but finally git throws an error >> >> error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s >> fetch-pack: unexpected disconnect while reading sideband packet >> fatal: early EOF >> fatal: index-pack failed >> >> or sometimes: >> >> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: >> CANCEL (err 8) >> error: 6109 bytes of body are still expected >> fetch-pack: unexpected disconnect while reading sideband packet >> fatal: early EOF >> fatal: fetch-pack: invalid index-pack output >> >> No such problems when downloading bigger file (i.e. linux kernel source) >> with wget or curl instead of git clone (wget/curl transfer drops to about >> 100 kB/s when VOD buffers and increases to full speed when VOD is not >> transferring and transfer finishes successfully). >> >> Sounds like a bug in git; should not throw an error on available download >> bandwidth changes as wget and curl do and should not require any params >> tuning (to stop users flooding bugtrackers). > I don't believe this is a bug in Git. The problem here is a network > issue of some sort that's causing the connection to be interrupted or > dropped. This is very common, but the possible causes are many. > > A lot of times we see this it's some sort of proxy, and that can be a > non-default antivirus or firewall or TLS MITM device, but that's usually > on Windows, and you're using Debian. It can also be just a bad > connection, poor traffic management by your ISP, or a flaky wireless or > wired network card. > > My _guess_ about what's happening here is poor traffic management: > because video is typically flagged as low-latency in DSCP, some device > on the path is prioritizing it to the exclusion of all other traffic, > which is causing the Git connection to be dropped. I have no proof of > this, though. Try to run git clone https://github.com/googleapis/google-api-go-client and limit interface speed somehow i.e. tc qdisc add dev eth0 root handle 1: htb default 12 tc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbps ceil 20kbps tc qdisc add dev eth0 parent 1:12 netem delay 1000ms and after a while restore speed with i.e. tc qdisc del dev eth0 parent 1:12 netem delay 1000ms I tried this in Debian 11 and git 2.30.2 died with: error: 5723 bytes of body are still expected MiB | 920.00 KiB/s fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes 2024-05-22 22:58 ` Paweł Bogusławski @ 2024-05-23 9:04 ` Jeff King 2024-05-23 11:15 ` Paweł Bogusławski 2024-05-23 12:07 ` rsbecker 1 sibling, 1 reply; 6+ messages in thread From: Jeff King @ 2024-05-23 9:04 UTC (permalink / raw) To: Paweł Bogusławski; +Cc: brian m. carlson, git On Thu, May 23, 2024 at 12:58:27AM +0200, Paweł Bogusławski wrote: > Try to run > > git clone https://github.com/googleapis/google-api-go-client > > and limit interface speed somehow i.e. > > tc qdisc add dev eth0 root handle 1: htb default 12 > tc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbps ceil 20kbps > tc qdisc add dev eth0 parent 1:12 netem delay 1000ms > > and after a while restore speed with i.e. > > tc qdisc del dev eth0 parent 1:12 netem delay 1000ms > > I tried this in Debian 11 and git 2.30.2 died with: > > error: 5723 bytes of body are still expected MiB | 920.00 KiB/s > fetch-pack: unexpected disconnect while reading sideband packet > fatal: early EOF > fatal: index-pack failed I did run it a few times. It usually succeeds, but sometimes fails. When it does fail, it does so in "slow" mode, without speeding back up. And I see one message from before your "error:" line, which is a report that curl reported that the connection dropped (curl is shuttling data back from the server in the form of "packet lines", each of which is prefixed with its expected size; the "N bytes of body are still expected" is just that parser seeing a truncated response from curl). But I don't think any of this is due to what Git is doing on the client side. From the client view, the server sends a TCP RST (which I observed with wireshark), and the connection drops. I'd suspect the drop is not even from the application level on the server side, but rather that something in the network path was mad about the delayed packets (or possibly the spam of duplicated TCP acks that I sometimes saw). So I really think this is a network issue, and not a Git issue in particular. It's possible that the networking stack that github.com uses is more picky about the things I mentioned above. But again, not a Git issue. -Peff ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Cloning does not work on available download bandwidth changes 2024-05-23 9:04 ` Jeff King @ 2024-05-23 11:15 ` Paweł Bogusławski 0 siblings, 0 replies; 6+ messages in thread From: Paweł Bogusławski @ 2024-05-23 11:15 UTC (permalink / raw) To: Jeff King; +Cc: brian m. carlson, git W dniu 23.05.2024 o 11:04, Jeff King pisze: > I did run it a few times. It usually succeeds, but sometimes fails. When > it does fail, it does so in "slow" mode, without speeding back up. And > I see one message from before your "error:" line, which is a report that > curl reported that the connection dropped (curl is shuttling data back > from the server in the form of "packet lines", each of which is prefixed > with its expected size; the "N bytes of body are still expected" is just > that parser seeing a truncated response from curl). > > But I don't think any of this is due to what Git is doing on the client > side. From the client view, the server sends a TCP RST (which I observed > with wireshark), and the connection drops. I'd suspect the drop is not > even from the application level on the server side, but rather that > something in the network path was mad about the delayed packets (or > possibly the spam of duplicated TCP acks that I sometimes saw). > > So I really think this is a network issue, and not a Git issue in > particular. It's possible that the networking stack that github.com uses > is more picky about the things I mentioned above. But again, not a Git > issue. Checked once again with... $ export GIT_TRACE_PACKET=1 $ export GIT_TRACE=1 $ export GIT_CURL_VERBOSE=1 ...and git dies with... [...] remote: Counting objects: 100% (6960/6960), done. remote: Compressing objects: 100% (2926/2926), done. 12:55:29.776311 http.c:756 == Info: Connection #0 to host github.com left intact error: 3805 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed ...a few secs after interface speed is restored (git transfer increases a few secs). In tcpdump of above test I can see FIN sent by client (192.168.100.100) first: # tcpdump -i eth0 -nnvvXSs 1514 -w test.pcap 'tcp and src 140.82.121.4 or dst 140.82.121.4' [...] 21346 2024-05-23 12:55:29,776138 140.82.121.4 192.168.100.100 TLSv1.3 959 Continuation Data 21347 2024-05-23 12:55:29,776151 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [ACK] Seq=6192 Ack=19611541 Win=65535 Len=0 21348 2024-05-23 12:55:29,776702 192.168.100.100 140.82.121.4 TLSv1.3 78 Application Data >>> here git client sends FIN but transfer is not completed <<<< 21349 2024-05-23 12:55:29,778374 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [FIN, ACK] Seq=6216 Ack=19611541 Win=65535 Len=0 21350 2024-05-23 12:55:29,836364 140.82.121.4 192.168.100.100 TLSv1.3 78 Application Data 21351 2024-05-23 12:55:29,836364 140.82.121.4 192.168.100.100 TCP 54 443 → 40624 [FIN, ACK] Seq=19611565 Ack=6216 Win=65535 Len=0 21352 2024-05-23 12:55:29,836421 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [RST] Seq=6216 Win=0 Len=0 21353 2024-05-23 12:55:29,836441 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [RST] Seq=6216 Win=0 Len=0 21354 2024-05-23 12:55:29,838428 140.82.121.4 192.168.100.100 TCP 54 443 → 40624 [ACK] Seq=19611566 Ack=6217 Win=65535 Len=0 21355 2024-05-23 12:55:29,838448 192.168.100.100 140.82.121.4 TCP 54 40624 → 443 [RST] Seq=6217 Win=0 Len=0 Cloning $ git clone https://gitlab.com/gitlab-org/gitlab $ git clone https://codeberg.org/forgejo/forgejo does not generate such problem after interface speed changes so maybe it's something specific for git <---> github.com communication. ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Cloning does not work on available download bandwidth changes 2024-05-22 22:58 ` Paweł Bogusławski 2024-05-23 9:04 ` Jeff King @ 2024-05-23 12:07 ` rsbecker 1 sibling, 0 replies; 6+ messages in thread From: rsbecker @ 2024-05-23 12:07 UTC (permalink / raw) To: 'Paweł Bogusławski', 'brian m. carlson', git On Wednesday, May 22, 2024 6:58 PM, Paweł Bogusławski wrote: >W dniu 22.05.2024 o 23:52, brian m. carlson pisze: > >> On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote: >>> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up >>> to 20mbps down)... >>> >>> $ git clone https://github.com/googleapis/google-api-go-client >>> Cloning into 'google-api-go-client'... >>> remote: Enumerating objects: 644446, done. >>> remote: Counting objects: 100% (6922/6922), done. >>> remote: Compressing objects: 100% (2904/2904), done. >>> Receiving objects: 0% (3859/644446), 20.82 MiB | 1.01 MiB/s >>> >>> ...and then started to watch a VOD movie on same link; when VOD >>> buffers data, eats almost all available down bandwidth and leaves >>> only about 100 kB/s for git... >>> >>> Receiving objects: 1% (7111/644446), 44.49 MiB | 130.00 KiB/s >>> >>> ...and when VOD stops buffering and whole bandwith is available for >>> git again, git transfer starts to grow... >>> >>> Receiving objects: 1% (7660/644446), 50.56 MiB | 575.00 KiB/s >>> >>> ...but finally git throws an error >>> >>> error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s >>> fetch-pack: unexpected disconnect while reading sideband packet >>> fatal: early EOF >>> fatal: index-pack failed >>> >>> or sometimes: >>> >>> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: >>> CANCEL (err 8) >>> error: 6109 bytes of body are still expected >>> fetch-pack: unexpected disconnect while reading sideband packet >>> fatal: early EOF >>> fatal: fetch-pack: invalid index-pack output >>> >>> No such problems when downloading bigger file (i.e. linux kernel >>> source) with wget or curl instead of git clone (wget/curl transfer >>> drops to about >>> 100 kB/s when VOD buffers and increases to full speed when VOD is not >>> transferring and transfer finishes successfully). >>> >>> Sounds like a bug in git; should not throw an error on available >>> download bandwidth changes as wget and curl do and should not require >>> any params tuning (to stop users flooding bugtrackers). >> I don't believe this is a bug in Git. The problem here is a network >> issue of some sort that's causing the connection to be interrupted or >> dropped. This is very common, but the possible causes are many. >> >> A lot of times we see this it's some sort of proxy, and that can be a >> non-default antivirus or firewall or TLS MITM device, but that's >> usually on Windows, and you're using Debian. It can also be just a >> bad connection, poor traffic management by your ISP, or a flaky >> wireless or wired network card. >> >> My _guess_ about what's happening here is poor traffic management: >> because video is typically flagged as low-latency in DSCP, some device >> on the path is prioritizing it to the exclusion of all other traffic, >> which is causing the Git connection to be dropped. I have no proof of >> this, though. > >Try to run > >git clone https://github.com/googleapis/google-api-go-client > >and limit interface speed somehow i.e. > >tc qdisc add dev eth0 root handle 1: htb default 12 tc class add dev eth0 parent 1:1 >classid 1:12 htb rate 20kbps ceil 20kbps tc qdisc add dev eth0 parent 1:12 netem >delay 1000ms > >and after a while restore speed with i.e. > >tc qdisc del dev eth0 parent 1:12 netem delay 1000ms > >I tried this in Debian 11 and git 2.30.2 died with: > >error: 5723 bytes of body are still expected MiB | 920.00 KiB/s >fetch-pack: unexpected disconnect while reading sideband packet >fatal: early EOF >fatal: index-pack failed I have tried this from my home ISP. It does a little throttling but sustained between 5-6 MiB/s for most of the transfer. We have seen related issues around ISP throttling that causes connections to drop for very large repositories. However, in the test I just ran, both git and https are happy to complete. $ git clone https://github.com/googleapis/google-api-go-client Cloning into 'google-api-go-client'... remote: Enumerating objects: 644446, done. remote: Counting objects: 100% (6922/6922), done. remote: Compressing objects: 100% (2904/2904), done. remote: Total 644446 (delta 3722), reused 6567 (delta 3445), pack-reused 637524 Receiving objects: 100% (644446/644446), 1.46 GiB | 7.45 MiB/s, done. Resolving deltas: 100% (392971/392971), done. Updating files: 100% (1372/1372), done. $ git version git version 2.43.0 Have you tried using SSH to do the clone? Assuming you have your public key registered on GitHub: git clone ssh://git@github.com/googleapis/google-api-go-client This runs about slightly slower. $ git clone ssh://git@github.com/googleapis/google-api-go-client Cloning into 'google-api-go-client'... remote: Enumerating objects: 644446, done. remote: Counting objects: 100% (6922/6922), done. remote: Compressing objects: 100% (2911/2911), done. remote: Total 644446 (delta 3720), reused 6563 (delta 3438), pack-reused 637524 Receiving objects: 100% (644446/644446), 1.46 GiB | 6.31 MiB/s, done. Resolving deltas: 100% (392969/392969), done. Updating files: 100% (1372/1372), done. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-05-23 12:15 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-05-22 15:02 Cloning does not work on available download bandwidth changes Paweł Bogusławski 2024-05-22 21:52 ` brian m. carlson 2024-05-22 22:58 ` Paweł Bogusławski 2024-05-23 9:04 ` Jeff King 2024-05-23 11:15 ` Paweł Bogusławski 2024-05-23 12:07 ` rsbecker
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).