git push --mirror hung indefinitely

git push --mirror hung indefinitely

[Akash S]

Hi,

We are attempting to push a bare repository on our local disk to Azure DevOps using the command "git push -mirror <URL>". 

Git version being used: 2.48.1 and no global git configurations exist in the machine.

Issue: command progresses for 5-10 mins and then the git processes get hung after we see the output "Storing index (done)" and remains hung indefinitely. 

Steps to reproduce:
i) Bare clone open source repo (elasticsearch or node) to local disk.
ii) Run "git lfs push -mirror" to any repository in an Azure DevOps organization using PAT authentication.
iii) Issue happens intermittently, so alternating the push operation between the local elasticsearch and node repo to the same remote repository should help recreate the issue.

Workaround steps taken by us to not face this issue:

i) Since our repository is quite large (1.5 GB), we set http.postBuffer to 50 MB, it did not help in resolving the issue.
ii) We have set GIT_CURL_TIMEOUT=180 along with  http.lowSpeedLimit=1  http.lowSpeedTime=180. But nothing seems to make the process exit after it gets hung.
iii) We set the http.maxRequests=1 to avoid concurrent requests and that did not help either.
iv) We used the -no-thin option, did not work either, process still hung.
v) We have tried with different values as well for the above git config options and nothing seemed to bring us out from the hung state.


GIT_TRACE=2 GIT_CURL_VERBOSE=2 outputs:
remote: Storing packfile... done (43758 ms)
08:26:52.444122 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
08:26:57.457679 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
08:26:58.309172 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
remote: Storing index... done (5865 ms)
08:28:09.549750 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
08:28:09.550215 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
08:28:09.550751 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
08:28:09.551245 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
08:28:09.551768 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
08:28:09.551850 http.c:878              == Info: Connection #0 to host dev.azure.com left intact

Processes launched after running git push -mirror:
[root@dop02108c1eu06 ~]# ps -aef | grep Token
root     1043353 1043352  0 08:17 pts/10   00:00:00 /usr/local/bin/git -c http.maxRequests=1 -c http.postBuffer=50000000 -c http.lowSpeedLimit=1 -c http.lowSpeedTime=30 push --mirror --no-thin https://Token:%3cTOKEN%3e@dev.azure.com/TestADOPATPolicy/cv-restores/_git/cv-restores
root     1043354 1043353  0 08:17 pts/10   00:00:00 /usr/local/libexec/git-core/git remote-https https://Token:%3cTOKEN%3e@dev.azure.com/TestADOPATPolicy/cv-restores/_git/cv-restores https://Token:%3cTOKEN%3e@dev.azure.com/TestADOPATPolicy/cv-restores/_git/cv-restores
root     1043355 1043354  0 08:17 pts/10   00:00:05 /usr/local/libexec/git-core/git-remote-https https://Token:%3cTOKEN%3e@dev.azure.com/TestADOPATPolicy/cv-restores/_git/cv-restores https://Token:%3cTOKEN%3e@dev.azure.com/TestADOPATPolicy/cv-restores/_git/cv-restores
root     1043377 1043355  0 08:17 pts/10   00:00:03 /usr/local/libexec/git-core/git send-pack --stateless-rpc --helper-status --thin --progress https://Token:%3cTOKEN%3e@dev.azure.com/TestADOPATPolicy/cv-restores/_git/cv-restores/ --stdin
root     1052699 1049394  0 08:32 pts/14   00:00:00 grep --color=auto Token

pstack of all above processes :
[root@dop02108c1eu06 ~]# pstack 1043353
#0  0x00007f7face375a5 in read () from /lib64/libc.so.6
#1  0x00007f7facdc7418 in __GI__IO_file_underflow () from /lib64/libc.so.6
#2  0x00007f7facdbbdc8 in getdelim () from /lib64/libc.so.6
#3  0x0000000000635eb8 in strbuf_getwholeline (sb=sb@entry=0x7ffe7067cbf0, fp=0x1cf02c0, term=term@entry=10) at strbuf.c:645
#4  0x00000000006360c0 in strbuf_getdelim_strip_crlf (sb=0x7ffe7067cbf0, fp=<optimized out>, term=10) at strbuf.c:727
#5  0x000000000064dacc in recvline_fh (helper=0x1cf02c0, buffer=0x7ffe7067cbf0) at transport-helper.c:76
#6  0x000000000064ec6e in recvline (helper=0x1cf8570, buffer=0x7ffe7067cbf0) at transport-helper.c:923
#7  push_update_refs_status (data=0x1cf8570, remote_refs=0x1d2b3a0, flags=10) at transport-helper.c:923
#8  0x0000000000650c3f in push_refs_with_push (flags=10, remote_refs=0x1d2b3a0, transport=0x1cf82c0) at transport-helper.c:1088
#9  push_refs (transport=0x1cf82c0, remote_refs=0x1d2b3a0, flags=10) at transport-helper.c:1189
#10 0x0000000000653faf in transport_push (r=0x993960 <the_repo>, transport=transport@entry=0x1cf82c0, rs=rs@entry=0x98f1f0 <rs>, flags=flags@entry=10, reject_reasons=reject_reasons@entry=0x7ffe7067cfa0) at transport.c:1502
#11 0x0000000000491cef in push_with_options (flags=10, rs=0x98f1f0 <rs>, transport=0x1cf82c0) at builtin/push.c:387
#12 do_push (remote=0x1cf7b50, push_options=0x97aba0 <push_options_config>, flags=10) at builtin/push.c:442
#13 cmd_push (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>, repository=<optimized out>) at builtin/push.c:664
#14 0x0000000000404eae in run_builtin (repo=0x993960 <the_repo>, argv=<optimized out>, argc=<optimized out>, p=0x971fe8 <commands+2184>) at git.c:480
#15 handle_builtin (args=args@entry=0x7ffe7067dbe0) at git.c:740
#16 0x0000000000405f1f in run_argv (args=args@entry=0x7ffe7067dbe0) at git.c:807
#17 0x00000000004068d6 in cmd_main (argc=<optimized out>, argc@entry=13, argv=<optimized out>, argv@entry=0x7ffe7067ddb8) at git.c:947
#18 0x0000000000404a67 in main (argc=13, argv=0x7ffe7067ddb8) at common-main.c:64
[root@dop02108c1eu06 ~]#

[root@dop02108c1eu06 ~]# pstack 1043354
#0  0x00007f64b602b468 in waitpid () from /lib64/libpthread.so.0
#1  0x000000000060fc0b in wait_or_whine (pid=1043355, argv0=0x298f5f0 "git-remote-https", in_signal=in_signal@entry=0) at run-command.c:559
#2  0x00000000006115c9 in finish_command (cmd=cmd@entry=0x7ffe5685e600) at run-command.c:989
#3  0x0000000000611679 in run_command (cmd=cmd@entry=0x7ffe5685e600) at run-command.c:1015
#4  0x0000000000405fe7 in execv_dashed_external (argv=<optimized out>) at git.c:777
#5  run_argv (args=args@entry=0x7ffe5685e6d0) at git.c:844
#6  0x00000000004068d6 in cmd_main (argc=<optimized out>, argc@entry=4, argv=<optimized out>, argv@entry=0x7ffe5685e8a8) at git.c:947
#7  0x0000000000404a67 in main (argc=4, argv=0x7ffe5685e8a8) at common-main.c:64
[root@dop02108c1eu06 ~]#

[root@dop02108c1eu06 ~]# pstack 1043355 #No Pstack strace
[root@dop02108c1eu06 ~]# pstack 1043377
Thread 2 (Thread 0x7fa246a18700 (LWP 1043384)):
#0  0x00007fa24ce568e4 in read () from /lib64/libpthread.so.0
#1  0x000000000066a07e in xread (fd=fd@entry=0, buf=buf@entry=0x7fa246a07d7c, len=len@entry=4) at wrapper.c:234
#2  0x000000000066a1eb in read_in_full (fd=0, buf=buf@entry=0x7fa246a07d7c, count=count@entry=4) at wrapper.c:292
#3  0x00000000005c4d32 in get_packet_data (fd=fd@entry=0, src_buf=src_buf@entry=0x0, src_size=src_size@entry=0x0, dst=dst@entry=0x7fa246a07d7c, size=size@entry=4, options=options@entry=1) at pkt-line.c:354
#4  0x00000000005c5827 in packet_read_with_status (fd=fd@entry=0, src_buffer=src_buffer@entry=0x0, src_len=src_len@entry=0x0, buffer=buffer@entry=0x7fa246a07e20 "\001\060\060\061aok refs/tags/v0.10.45\n", size=size@entry=65520, pktlen=pktlen@entry=0x7fa246a07df8, options=1) at pkt-line.c:422
#5  0x00000000005c5fa9 in recv_sideband (me=me@entry=0x6e35ea "send-pack", in_stream=0, out=out@entry=5) at pkt-line.c:586
#6  0x0000000000612bf3 in sideband_demux (in=<optimized out>, out=5, data=0x7ffcfb7a3b08) at send-pack.c:283
#7  0x000000000060fd8e in run_thread (data=0x7ffcfb7a3930) at run-command.c:1040
#8  0x00007fa24ce4d14a in start_thread () from /lib64/libpthread.so.0
#9  0x00007fa24c974dc3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa24d49cb80 (LWP 1043377)):
#0  0x00007fa24ce568e4 in read () from /lib64/libpthread.so.0
#1  0x000000000066a07e in xread (fd=fd@entry=4, buf=buf@entry=0x7ffcfb7a36cc, len=len@entry=4) at wrapper.c:234
#2  0x000000000066a1eb in read_in_full (fd=4, buf=buf@entry=0x7ffcfb7a36cc, count=count@entry=4) at wrapper.c:292
#3  0x00000000005c4d32 in get_packet_data (fd=fd@entry=4, src_buf=src_buf@entry=0x7ffcfb7a3968, src_size=src_size@entry=0x7ffcfb7a3970, dst=dst@entry=0x7ffcfb7a36cc, size=size@entry=4, options=options@entry=6) at pkt-line.c:354
#4  0x00000000005c5827 in packet_read_with_status (fd=4, src_buffer=src_buffer@entry=0x7ffcfb7a3968, src_len=src_len@entry=0x7ffcfb7a3970, buffer=0x99af00 <packet_buffer> "ok", size=65520, pktlen=pktlen@entry=0x7ffcfb7a398c, options=6) at pkt-line.c:422
#5  0x00000000005c5d10 in packet_reader_read (reader=reader@entry=0x7ffcfb7a3960) at pkt-line.c:640
#6  0x00000000005c60ac in packet_reader_read (reader=reader@entry=0x7ffcfb7a3960) at pkt-line.c:626
#7  0x0000000000612d98 in receive_status (reader=reader@entry=0x7ffcfb7a3960, refs=refs@entry=0x1583c90) at send-pack.c:183
#8  0x0000000000613bbb in send_pack (args=args@entry=0x98f5c0 <args>, fd=fd@entry=0x7ffcfb7a3b08, conn=conn@entry=0x0, remote_refs=<optimized out>, extra_have=extra_have@entry=0x7ffcfb7a3b70) at send-pack.c:746
#9  0x00000000004b64de in cmd_send_pack (argc=<optimized out>, argv=0x153e660, prefix=<optimized out>, repo=<optimized out>) at builtin/send-pack.c:320
#10 0x0000000000404eae in run_builtin (repo=0x993960 <the_repo>, argv=<optimized out>, argc=<optimized out>, p=0x9721c8 <commands+2664>) at git.c:480
#11 handle_builtin (args=args@entry=0x7ffcfb7a4580) at git.c:740
#12 0x0000000000405f1f in run_argv (args=args@entry=0x7ffcfb7a4580) at git.c:807
#13 0x00000000004068d6 in cmd_main (argc=<optimized out>, argc@entry=8, argv=<optimized out>, argv@entry=0x7ffcfb7a4758) at git.c:947
#14 0x0000000000404a67 in main (argc=8, argv=0x7ffcfb7a4758) at common-main.c:64

strace of all above processes:

[root@dop02108c1eu06 ~]# strace -p 1043353
strace: Process 1043353 attached
read(3,


[root@dop02108c1eu06 workingDir]# strace -p 1043354
strace: Process 1043354 attached
wait4(1043355,

[root@dop02108c1eu06 workingDir]# strace -p 1043355
rt_sigaction(SIGPIPE, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0
select(4, [3], [], [], {tv_sec=0, tv_usec=795000}) = 1 (in [3], left {tv_sec=0, tv_usec=794998})
rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, 8) = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])
read(3, "\27\3\3\0006", 5)              = 5
read(3, "\216v\301[\367\322\25\326\25\266p6\322\211%\2237\305$W\21\r.oK\241\207f\306\232~\23"..., 54) = 54
write(2, "08:28:09.551245 http.c:878      "..., 95) = 95
write(5, "001f\001001aok refs/tags/v0.10.45\n", 31) = 31
rt_sigaction(SIGPIPE, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0
select(4, [3], [], [], {tv_sec=0, tv_usec=795000}) = 1 (in [3], left {tv_sec=0, tv_usec=794998})
rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, 8) = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])
read(3, "\27\3\3\0\26", 5)              = 5
read(3, "\346\302h\301\6g\216s^:\301\352\274\341G\6\245\3AT+7", 22) = 22
write(2, "08:28:09.551768 http.c:878      "..., 95) = 95
write(2, "08:28:09.551850 http.c:878      "..., 97) = 97
rt_sigaction(SIGPIPE, {sa_handler=SIG_DFL, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f337607eb20}, NULL, 8) = 0
read(6,

[root@dop02108c1eu06 ~]# strace -p 1043377
strace: Process 1043377 attached
read(4,

Thanks,
Akash

Re: git push --mirror hung indefinitely

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3530 bytes --]

On 2025-03-28 at 10:37:43, Akash S wrote:
> Hi,
> 
> We are attempting to push a bare repository on our local disk to Azure DevOps using the command "git push -mirror <URL>".
> 
> Git version being used: 2.48.1 and no global git configurations exist in the machine.
> 
> Issue: command progresses for 5-10 mins and then the git processes get hung after we see the output "Storing index (done)" and remains hung indefinitely.
> 
> Steps to reproduce:
> i) Bare clone open source repo (elasticsearch or node) to local disk.
> ii) Run "git lfs push -mirror" to any repository in an Azure DevOps organization using PAT authentication.
> iii) Issue happens intermittently, so alternating the push operation between the local elasticsearch and node repo to the same remote repository should help recreate the issue.

Does this happen if you push it to a different server?  From what it
looks like here, it might be the server not sending anything back in
time.

Also, do you have anything intercepting TLS, like a non-default
antivirus or firewall or any sort of proxy?  Sometimes such tools buffer
the output of the request, which can break things[0].  If you do, you
should either uninstall the software completely and reboot (since
disabling it is often not sufficient) or change networks if there's a
proxy or TLS MITM device on the network and see if it's fixed.

> Workaround steps taken by us to not face this issue:
> 
> i) Since our repository is quite large (1.5 GB), we set http.postBuffer to 50 MB, it did not help in resolving the issue.

`http.postBuffer` simply sets the value below which the entire body is
sent with a content-length instead of using chunked transfer encoding.
As outlined in the Git FAQ[1], this option doesn't need to be set unless
you have a server or proxy which doesn't handle HTTP/1.1 properly, and
in general those are rare (because a lot of stuff would be broken).  I'm
sure Azure DevOps is not broken in that way.

Since it almost certainly isn't needed, I'd remove it, since it just
tends to waste a lot of memory if you don't need it.

> Thread 2 (Thread 0x7fa246a18700 (LWP 1043384)):
> #0  0x00007fa24ce568e4 in read () from /lib64/libpthread.so.0
> #1  0x000000000066a07e in xread (fd=fd@entry=0, buf=buf@entry=0x7fa246a07d7c, len=len@entry=4) at wrapper.c:234
> #2  0x000000000066a1eb in read_in_full (fd=0, buf=buf@entry=0x7fa246a07d7c, count=count@entry=4) at wrapper.c:292
> #3  0x00000000005c4d32 in get_packet_data (fd=fd@entry=0, src_buf=src_buf@entry=0x0, src_size=src_size@entry=0x0, dst=dst@entry=0x7fa246a07d7c, size=size@entry=4, options=options@entry=1) at pkt-line.c:354
> #4  0x00000000005c5827 in packet_read_with_status (fd=fd@entry=0, src_buffer=src_buffer@entry=0x0, src_len=src_len@entry=0x0, buffer=buffer@entry=0x7fa246a07e20 "\001\060\060\061aok refs/tags/v0.10.45\n", size=size@entry=65520, pktlen=pktlen@entry=0x7fa246a07df8, options=1) at pkt-line.c:422
> #5  0x00000000005c5fa9 in recv_sideband (me=me@entry=0x6e35ea "send-pack", in_stream=0, out=out@entry=5) at pkt-line.c:586

This looks like it's waiting to read a pkt-line header, which should be
sent periodically by the server, so my guess is that this is particular
to the server that you're using, or, as I mentioned above, that you have
something in the middle that's intercepting the data.

[0] https://git-scm.com/docs/gitfaq#proxy
[1] https://git-scm.com/docs/gitfaq#http-postbuffer
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

RE: git push --mirror hung indefinitely

[Akash S]

Hi Brian,

i) We have raised a ticket with Microsoft Azure DevOps regarding this "git push getting hung" issue and they are claiming that it is due to requests from the user facing rate-limiting issue. We have tried to use the options GIT_CURL_TIMEOUT=180 and http.lowSpeedLimit=1 and http.lowSpeedTime=180, none of them made the git process exit after getting hung.

	Are you aware of any other config options which can help the git process exit when it is hung indefinitely or for a specified amount of time?

ii) Do you know if Azure DevOps is maybe sending the rate limit headers (X-RateLimit-Reset, Retry-After, etc) or some response code to indicate closure of the connection itself which is not being handled by git on your side and hence causing the git to stay hung indefinitely?

	You can also try and validate this using the following steps to reproduce the issue:
		a) Bare clone open-source repos (elasticsearch or node, or any huge repo with millions of git objects to easily reproduce issue) to local disk.
		b) Run "git push -mirror" to any repository in an Azure DevOps organization using PAT authentication.
			git push --mirror "https://Token:<PAT>@dev.azure.com/ORGNAME/PRJNAME/_git/REPONAME.git
		c) Issue happens intermittently, so alternating the push operation between the local elasticsearch and node repo to the same remote repository should help recreate the issue.
	
Thanks,
Akash

-----Original Message-----
From: brian m. carlson <sandals@crustytoothpaste.net> 
Sent: 29 March 2025 01:09
To: Akash S <akashs@commvault.com>
Cc: git@vger.kernel.org; Adithya Urugudige <aurugudige@commvault.com>; Abhishek Dalmia <adalmia@commvault.com>
Subject: Re: git push --mirror hung indefinitely

On 2025-03-28 at 10:37:43, Akash S wrote:
> Hi,
> 
> We are attempting to push a bare repository on our local disk to Azure DevOps using the command "git push -mirror <URL>".
> 
> Git version being used: 2.48.1 and no global git configurations exist in the machine.
> 
> Issue: command progresses for 5-10 mins and then the git processes get hung after we see the output "Storing index (done)" and remains hung indefinitely.
> 
> Steps to reproduce:
> i) Bare clone open source repo (elasticsearch or node) to local disk.
> ii) Run "git lfs push -mirror" to any repository in an Azure DevOps organization using PAT authentication.
> iii) Issue happens intermittently, so alternating the push operation between the local elasticsearch and node repo to the same remote repository should help recreate the issue.

Does this happen if you push it to a different server?  From what it looks like here, it might be the server not sending anything back in time.

Also, do you have anything intercepting TLS, like a non-default antivirus or firewall or any sort of proxy?  Sometimes such tools buffer the output of the request, which can break things[0].  If you do, you should either uninstall the software completely and reboot (since disabling it is often not sufficient) or change networks if there's a proxy or TLS MITM device on the network and see if it's fixed.

> Workaround steps taken by us to not face this issue:
> 
> i) Since our repository is quite large (1.5 GB), we set http.postBuffer to 50 MB, it did not help in resolving the issue.

`http.postBuffer` simply sets the value below which the entire body is sent with a content-length instead of using chunked transfer encoding.
As outlined in the Git FAQ[1], this option doesn't need to be set unless you have a server or proxy which doesn't handle HTTP/1.1 properly, and in general those are rare (because a lot of stuff would be broken).  I'm sure Azure DevOps is not broken in that way.

Since it almost certainly isn't needed, I'd remove it, since it just tends to waste a lot of memory if you don't need it.

> Thread 2 (Thread 0x7fa246a18700 (LWP 1043384)):
> #0  0x00007fa24ce568e4 in read () from /lib64/libpthread.so.0
> #1  0x000000000066a07e in xread (fd=fd@entry=0, 
> buf=buf@entry=0x7fa246a07d7c, len=len@entry=4) at wrapper.c:234
> #2  0x000000000066a1eb in read_in_full (fd=0, 
> buf=buf@entry=0x7fa246a07d7c, count=count@entry=4) at wrapper.c:292
> #3  0x00000000005c4d32 in get_packet_data (fd=fd@entry=0, 
> src_buf=src_buf@entry=0x0, src_size=src_size@entry=0x0, 
> dst=dst@entry=0x7fa246a07d7c, size=size@entry=4, 
> options=options@entry=1) at pkt-line.c:354
> #4  0x00000000005c5827 in packet_read_with_status (fd=fd@entry=0, 
> src_buffer=src_buffer@entry=0x0, src_len=src_len@entry=0x0, 
> buffer=buffer@entry=0x7fa246a07e20 "\001\060\060\061aok 
> refs/tags/v0.10.45\n", size=size@entry=65520, 
> pktlen=pktlen@entry=0x7fa246a07df8, options=1) at pkt-line.c:422
> #5  0x00000000005c5fa9 in recv_sideband (me=me@entry=0x6e35ea 
> "send-pack", in_stream=0, out=out@entry=5) at pkt-line.c:586

This looks like it's waiting to read a pkt-line header, which should be sent periodically by the server, so my guess is that this is particular to the server that you're using, or, as I mentioned above, that you have something in the middle that's intercepting the data.

[0] https://git-scm.com/docs/gitfaq#proxy
[1] https://git-scm.com/docs/gitfaq#http-postbuffer
--
brian m. carlson (they/them)
Toronto, Ontario, CA

RE: git push --mirror hung indefinitely

[Akash S]

Hi Brian,

We compared the GIT_TRACE_PACKET=1 output of successful and unsuccessful pushes and noticed that the "git flush message 0000" is not being received from the Azure DevOps Server during unsuccessful push. 

-------------------------------------------------------------------------------------------------------------------------------------------------------

Successful git push:
11:29:09.032050 pkt-line.c:85           packet:          git< ok refs/tags/v3.9.1
11:29:09.034601 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:29:09.034630 pkt-line.c:85           packet:     sideband< \10000
11:29:09.034643 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:29:09.034664 pkt-line.c:85           packet:     sideband< 0000
11:29:09.034664 pkt-line.c:85           packet:          git< 0000
11:29:09.034699 pkt-line.c:85           packet:          git> 0000
11:29:09.035667 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:29:09.035701 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
To https://dev.azure.com/ORGNAME/PRJNAME/_git/REPONAME

-------------------------------------------------------------------------------------------------------------------------------------------------------

Unsuccessful git push:
11:33:58.759005 pkt-line.c:85           packet:     sideband< \10018ok refs/tags/v3.1.7
11:33:58.759035 pkt-line.c:85           packet:          git< ok refs/tags/v3.1.7
11:33:58.760165 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:33:58.760197 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
(process hung here)

-------------------------------------------------------------------------------------------------------------------------------------------------------

We noticed that "git push mirror" ultimately spawns "git send-pack" with options "--stdin" and "--stateless-rpc" and from the git send-pack documentation we confirmed that the line must end with a flush packet which is ultimately not happening in the hung cases. pstack of send-pack process also shows it is waiting on reading packets as you mentioned earlier.

Can you please confirm if the hung issue is due to the missing message or packet from the Azure DevOps Server?

Thanks,
Akash
 
-----Original Message-----
From: Akash S 
Sent: 29 March 2025 12:02
To: brian m. carlson <sandals@crustytoothpaste.net>
Cc: git@vger.kernel.org; Adithya Urugudige <aurugudige@commvault.com>; Abhishek Dalmia <adalmia@commvault.com>
Subject: RE: git push --mirror hung indefinitely

Hi Brian,

i) We have raised a ticket with Microsoft Azure DevOps regarding this "git push getting hung" issue and they are claiming that it is due to requests from the user facing rate-limiting issue. We have tried to use the options GIT_CURL_TIMEOUT=180 and http.lowSpeedLimit=1 and http.lowSpeedTime=180, none of them made the git process exit after getting hung.

	Are you aware of any other config options which can help the git process exit when it is hung indefinitely or for a specified amount of time?

ii) Do you know if Azure DevOps is maybe sending the rate limit headers (X-RateLimit-Reset, Retry-After, etc) or some response code to indicate closure of the connection itself which is not being handled by git on your side and hence causing the git to stay hung indefinitely?

	You can also try and validate this using the following steps to reproduce the issue:
		a) Bare clone open-source repos (elasticsearch or node, or any huge repo with millions of git objects to easily reproduce issue) to local disk.
		b) Run "git push -mirror" to any repository in an Azure DevOps organization using PAT authentication.
			git push --mirror "https://Token:<PAT>@dev.azure.com/ORGNAME/PRJNAME/_git/REPONAME.git
		c) Issue happens intermittently, so alternating the push operation between the local elasticsearch and node repo to the same remote repository should help recreate the issue.
	
Thanks,
Akash

-----Original Message-----
From: brian m. carlson <sandals@crustytoothpaste.net>
Sent: 29 March 2025 01:09
To: Akash S <akashs@commvault.com>
Cc: git@vger.kernel.org; Adithya Urugudige <aurugudige@commvault.com>; Abhishek Dalmia <adalmia@commvault.com>
Subject: Re: git push --mirror hung indefinitely

On 2025-03-28 at 10:37:43, Akash S wrote:
> Hi,
> 
> We are attempting to push a bare repository on our local disk to Azure DevOps using the command "git push -mirror <URL>".
> 
> Git version being used: 2.48.1 and no global git configurations exist in the machine.
> 
> Issue: command progresses for 5-10 mins and then the git processes get hung after we see the output "Storing index (done)" and remains hung indefinitely.
> 
> Steps to reproduce:
> i) Bare clone open source repo (elasticsearch or node) to local disk.
> ii) Run "git lfs push -mirror" to any repository in an Azure DevOps organization using PAT authentication.
> iii) Issue happens intermittently, so alternating the push operation between the local elasticsearch and node repo to the same remote repository should help recreate the issue.

Does this happen if you push it to a different server?  From what it looks like here, it might be the server not sending anything back in time.

Also, do you have anything intercepting TLS, like a non-default antivirus or firewall or any sort of proxy?  Sometimes such tools buffer the output of the request, which can break things[0].  If you do, you should either uninstall the software completely and reboot (since disabling it is often not sufficient) or change networks if there's a proxy or TLS MITM device on the network and see if it's fixed.

> Workaround steps taken by us to not face this issue:
> 
> i) Since our repository is quite large (1.5 GB), we set http.postBuffer to 50 MB, it did not help in resolving the issue.

`http.postBuffer` simply sets the value below which the entire body is sent with a content-length instead of using chunked transfer encoding.
As outlined in the Git FAQ[1], this option doesn't need to be set unless you have a server or proxy which doesn't handle HTTP/1.1 properly, and in general those are rare (because a lot of stuff would be broken).  I'm sure Azure DevOps is not broken in that way.

Since it almost certainly isn't needed, I'd remove it, since it just tends to waste a lot of memory if you don't need it.

> Thread 2 (Thread 0x7fa246a18700 (LWP 1043384)):
> #0  0x00007fa24ce568e4 in read () from /lib64/libpthread.so.0
> #1  0x000000000066a07e in xread (fd=fd@entry=0, 
> buf=buf@entry=0x7fa246a07d7c, len=len@entry=4) at wrapper.c:234
> #2  0x000000000066a1eb in read_in_full (fd=0, 
> buf=buf@entry=0x7fa246a07d7c, count=count@entry=4) at wrapper.c:292
> #3  0x00000000005c4d32 in get_packet_data (fd=fd@entry=0, 
> src_buf=src_buf@entry=0x0, src_size=src_size@entry=0x0, 
> dst=dst@entry=0x7fa246a07d7c, size=size@entry=4,
> options=options@entry=1) at pkt-line.c:354
> #4  0x00000000005c5827 in packet_read_with_status (fd=fd@entry=0, 
> src_buffer=src_buffer@entry=0x0, src_len=src_len@entry=0x0,
> buffer=buffer@entry=0x7fa246a07e20 "\001\060\060\061aok 
> refs/tags/v0.10.45\n", size=size@entry=65520, 
> pktlen=pktlen@entry=0x7fa246a07df8, options=1) at pkt-line.c:422
> #5  0x00000000005c5fa9 in recv_sideband (me=me@entry=0x6e35ea 
> "send-pack", in_stream=0, out=out@entry=5) at pkt-line.c:586

This looks like it's waiting to read a pkt-line header, which should be sent periodically by the server, so my guess is that this is particular to the server that you're using, or, as I mentioned above, that you have something in the middle that's intercepting the data.

[0] https://git-scm.com/docs/gitfaq#proxy
[1] https://git-scm.com/docs/gitfaq#http-postbuffer
--
brian m. carlson (they/them)
Toronto, Ontario, CA

Re: git push --mirror hung indefinitely

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3428 bytes --]

On 2025-03-29 at 14:07:17, Akash S wrote:
> We compared the GIT_TRACE_PACKET=1 output of successful and
> unsuccessful pushes and noticed that the "git flush message 0000" is
> not being received from the Azure DevOps Server during unsuccessful
> push. 
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> Successful git push:
> 11:29:09.032050 pkt-line.c:85           packet:          git< ok refs/tags/v3.9.1
> 11:29:09.034601 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:29:09.034630 pkt-line.c:85           packet:     sideband< \10000
> 11:29:09.034643 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:29:09.034664 pkt-line.c:85           packet:     sideband< 0000
> 11:29:09.034664 pkt-line.c:85           packet:          git< 0000
> 11:29:09.034699 pkt-line.c:85           packet:          git> 0000
> 11:29:09.035667 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:29:09.035701 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
> To https://dev.azure.com/ORGNAME/PRJNAME/_git/REPONAME
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> Unsuccessful git push:
> 11:33:58.759005 pkt-line.c:85           packet:     sideband< \10018ok refs/tags/v3.1.7
> 11:33:58.759035 pkt-line.c:85           packet:          git< ok refs/tags/v3.1.7
> 11:33:58.760165 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:33:58.760197 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
> (process hung here)
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> We noticed that "git push mirror" ultimately spawns "git send-pack"
> with options "--stdin" and "--stateless-rpc" and from the git
> send-pack documentation we confirmed that the line must end with a
> flush packet which is ultimately not happening in the hung cases.
> pstack of send-pack process also shows it is waiting on reading
> packets as you mentioned earlier.

`--stateless-rpc` is used for HTTPS, so that's expected.

> Can you please confirm if the hung issue is due to the missing message or packet from the Azure DevOps Server?

We don't have a full trace output here, so it's hard to say completely
for certain, but from what I know of the protocol off the top of my
head, a flush packet is always required at the end of the section of
message.  So I would say that it's very likely that's the cause here.

Note that it could be that you're _also_ missing one or more `ok`
responses from the ref updates you're doing here, and that it's hanging
_before_ it would normally send the flush packet.  That might point to a
different cause on the server side, but the effect would be the same.

If you're talking to the Azure DevOps people, I'd provide them a full
trace output of both working and non-working pushes so that they can
inspect it themselves, which will help them figure out where it might be
going wrong.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]