[RFH] Why do osx CI jobs so unreliable?

[RFH] Why do osx CI jobs so unreliable?

[Junio C Hamano]

I've been observing that in recent push-out to 'master' and 'next',
osx-* jobs in GitHub Actions CI keep running for 6 hours and get
killed.

What is troubling is that this seems to be very flaky.  For example,
https://github.com/git/git/actions/runs/27778820659 is testing
95e20213 (Hopefully final batch before -rc2, 2026-06-17) which got
killed after wasting 6 hours in osx-clang and osx-gcc jobs.

https://github.com/git/git/actions/runs/27790036076 is testing
the same 'master', with a patch to .github/workflows/main.yml to
remove everything except for config and osx-* jobs, which succeeded
within 30 minutes.

Stumped...

Re: [RFH] Why do osx CI jobs so unreliable?

[Patrick Steinhardt]

On Thu, Jun 18, 2026 at 05:35:23PM -0700, Junio C Hamano wrote:
> I've been observing that in recent push-out to 'master' and 'next',
> osx-* jobs in GitHub Actions CI keep running for 6 hours and get
> killed.
> 
> What is troubling is that this seems to be very flaky.  For example,
> https://github.com/git/git/actions/runs/27778820659 is testing
> 95e20213 (Hopefully final batch before -rc2, 2026-06-17) which got
> killed after wasting 6 hours in osx-clang and osx-gcc jobs.
> 
> https://github.com/git/git/actions/runs/27790036076 is testing
> the same 'master', with a patch to .github/workflows/main.yml to
> remove everything except for config and osx-* jobs, which succeeded
> within 30 minutes.
> 
> Stumped...

So the raw logs have the following trailer:

  2026-06-18T23:53:33.2996180Z Cleaning up orphan processes
  2026-06-18T23:53:33.7900380Z Terminate orphan process: pid (34022) (git-remote-http)
  2026-06-18T23:53:33.9848670Z Terminate orphan process: pid (15488) (httpd)
  2026-06-18T23:53:34.0321490Z Terminate orphan process: pid (13146) (httpd)
  2026-06-18T23:53:34.0808280Z Terminate orphan process: pid (13145) (httpd)
  2026-06-18T23:53:34.1212760Z Terminate orphan process: pid (13144) (httpd)
  2026-06-18T23:53:34.1570160Z Terminate orphan process: pid (13141) (httpd)
  2026-06-18T23:53:34.1924140Z Terminate orphan process: pid (12553) (bash)
  2026-06-18T23:53:34.2472970Z Terminate orphan process: pid (12552) (tee)
  2026-06-18T23:53:34.6547890Z Terminate orphan process: pid (21209) (bash)

So I strongly suspect that it most be one of the t555* tests.
Furthermore, the t5551 and t5559 (both of which are actually the same
test) are the only test suites that use lib-httpd.sh and which are
missing in the job logs.

I have not been able to reproduce this hang on my macOS virtual machine
though, and on GitLab I didn't notice a similar hang recently. Maybe
this is something that's specific to GitHub's environment...? No idea.

Patrick

Re: [RFH] Why do osx CI jobs so unreliable?

[Michael Montalbo]

Patrick Steinhardt <ps@pks.im> writes:
> So I strongly suspect that it most be one of the t555* tests.
> [...]
> Maybe this is something that's specific to GitHub's environment...

I think you're right it's t5551/t5559. The runs Junio linked:

  osx-clang     cancelled  360min
  osx-gcc       cancelled  360min
  osx-reftable  success     35min
  osx-meson     success     61min

All four run the same t5551/t5559 under EXPENSIVE. The two that
finished differ in just two ways, which look like the levers:
osx-reftable generates the 100k-ref advertisement in ~24ms vs ~1.2s
for loose refs on macOS (so much less time mid-response), and
osx-meson runs tests at nproc while the prove jobs hardcode --jobs=10
on a 3-core runner (over recent master/next the prove jobs hang ~40%,
meson ~10%).

When it is wedged the whole chain sits at 0% CPU. upload-pack is
blocked in write() on the ls-refs advertisement, curl blocked in
select(). So it looks like an HTTP/2 flow-control stall on the
response side. The same stall resets itself after ~60-85s on my Linux
box and on a bare-metal Mac, but not on the GitHub runner; I haven't
pinned down why yet.

On the chance those two levers are the fix, a branch off master:

  https://github.com/mmontalbo/git/tree/mm/macos-ci-hang-fix

  - pack the refs in t5551's enormous-ref-negotiation test (doesn't
    change what it checks on the wire, just avoids re-reading 100k loose
    files to advertise them, like reftable already does)
  - use the core count for $JOBS on the GitHub macOS path, matching the
    GitLab branch in the same ci/lib.sh and what meson does

I ran the two macOS jobs under EXPENSIVE about eight times with these
and they all finished in ~30-44min instead of hanging. Happy to send
out a patch if it's helpful.

Re: [RFH] Why do osx CI jobs so unreliable?

[Jeff King]

On Sat, Jun 20, 2026 at 08:33:13AM -0700, Michael Montalbo wrote:

> Patrick Steinhardt <ps@pks.im> writes:
> > So I strongly suspect that it most be one of the t555* tests.
> > [...]
> > Maybe this is something that's specific to GitHub's environment...
> 
> I think you're right it's t5551/t5559. The runs Junio linked:
> 
>   osx-clang     cancelled  360min
>   osx-gcc       cancelled  360min
>   osx-reftable  success     35min
>   osx-meson     success     61min
> 
> All four run the same t5551/t5559 under EXPENSIVE. The two that
> finished differ in just two ways, which look like the levers:
> osx-reftable generates the 100k-ref advertisement in ~24ms vs ~1.2s
> for loose refs on macOS (so much less time mid-response), and
> osx-meson runs tests at nproc while the prove jobs hardcode --jobs=10
> on a 3-core runner (over recent master/next the prove jobs hang ~40%,
> meson ~10%).

If the problem is a racy deadlock, there is a reasonable chance that
some jobs may simply be lucky. Even if things like packing refs help, I
suspect the problem may still be lurking. Maybe I'm just a pessimist,
though. ;)

> When it is wedged the whole chain sits at 0% CPU. upload-pack is
> blocked in write() on the ls-refs advertisement, curl blocked in
> select(). So it looks like an HTTP/2 flow-control stall on the
> response side. The same stall resets itself after ~60-85s on my Linux
> box and on a bare-metal Mac, but not on the GitHub runner; I haven't
> pinned down why yet.

We had some HTTP/2 stalls/deadlocks in the past, and they were dependent
on libcurl and apache (actually h2_mod) versions. IIRC some of the
non-TLS code paths for HTTP/2 were not well tested, which led to
8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course
after that commit those cleartext code paths should not be a problem, so
that is probably not exactly the issue now.

But it might be worth checking the versions you're running locally
versus what's in the GitHub runner.

-Peff

Re: [RFH] Why do osx CI jobs so unreliable?

[Patrick Steinhardt]

On Sun, Jun 21, 2026 at 05:34:07PM -0400, Jeff King wrote:
> On Sat, Jun 20, 2026 at 08:33:13AM -0700, Michael Montalbo wrote:
> 
> > Patrick Steinhardt <ps@pks.im> writes:
> > > So I strongly suspect that it most be one of the t555* tests.
> > > [...]
> > > Maybe this is something that's specific to GitHub's environment...
> > 
> > I think you're right it's t5551/t5559. The runs Junio linked:
> > 
> >   osx-clang     cancelled  360min
> >   osx-gcc       cancelled  360min
> >   osx-reftable  success     35min
> >   osx-meson     success     61min
> > 
> > All four run the same t5551/t5559 under EXPENSIVE. The two that
> > finished differ in just two ways, which look like the levers:
> > osx-reftable generates the 100k-ref advertisement in ~24ms vs ~1.2s
> > for loose refs on macOS (so much less time mid-response), and
> > osx-meson runs tests at nproc while the prove jobs hardcode --jobs=10
> > on a 3-core runner (over recent master/next the prove jobs hang ~40%,
> > meson ~10%).
> 
> If the problem is a racy deadlock, there is a reasonable chance that
> some jobs may simply be lucky. Even if things like packing refs help, I
> suspect the problem may still be lurking. Maybe I'm just a pessimist,
> though. ;)

I had the same thought.

> > When it is wedged the whole chain sits at 0% CPU. upload-pack is
> > blocked in write() on the ls-refs advertisement, curl blocked in
> > select(). So it looks like an HTTP/2 flow-control stall on the
> > response side. The same stall resets itself after ~60-85s on my Linux
> > box and on a bare-metal Mac, but not on the GitHub runner; I haven't
> > pinned down why yet.
> 
> We had some HTTP/2 stalls/deadlocks in the past, and they were dependent
> on libcurl and apache (actually h2_mod) versions. IIRC some of the
> non-TLS code paths for HTTP/2 were not well tested, which led to
> 8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course
> after that commit those cleartext code paths should not be a problem, so
> that is probably not exactly the issue now.
> 
> But it might be worth checking the versions you're running locally
> versus what's in the GitHub runner.

I didn't observe any similar hangs in GitLab's CI systems, so I wonder
whether this is because of different versions of curl. And indeed we use
different versions:

  - On GitHub we use 8.6.0.

  - On GitLab we use 8.7.1.

Now this of course doesn't mean that updating the curl version is the
fix to this whole issue, as there's a ton of other factors that could
play a role in whether or not the test hangs. So while we could just
upgrade parts of the stack and cross our fingers, but that feels rather
unsatisfactory. Still, one place to start could be to update our build
images to macOS 15.

But the big question to me is whether the hang is because of a bug in
Git with how we drive curl, a bug in curl itself, or a bug in Apache.

Patrick

Re: [RFH] Why do osx CI jobs so unreliable?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> If the problem is a racy deadlock, there is a reasonable chance that
> some jobs may simply be lucky. Even if things like packing refs help, I
> suspect the problem may still be lurking. Maybe I'm just a pessimist,
> though. ;)

I share the pessimism X-<.

> We had some HTTP/2 stalls/deadlocks in the past, and they were dependent
> on libcurl and apache (actually h2_mod) versions. IIRC some of the
> non-TLS code paths for HTTP/2 were not well tested, which led to
> 8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course
> after that commit those cleartext code paths should not be a problem, so
> that is probably not exactly the issue now.
>
> But it might be worth checking the versions you're running locally
> versus what's in the GitHub runner.

True.

Re: [RFH] Why do osx CI jobs so unreliable?

[Patrick Steinhardt]

On Mon, Jun 22, 2026 at 06:42:24AM +0200, Patrick Steinhardt wrote:
> On Sun, Jun 21, 2026 at 05:34:07PM -0400, Jeff King wrote:
> > On Sat, Jun 20, 2026 at 08:33:13AM -0700, Michael Montalbo wrote:
[snip]
> > > When it is wedged the whole chain sits at 0% CPU. upload-pack is
> > > blocked in write() on the ls-refs advertisement, curl blocked in
> > > select(). So it looks like an HTTP/2 flow-control stall on the
> > > response side. The same stall resets itself after ~60-85s on my Linux
> > > box and on a bare-metal Mac, but not on the GitHub runner; I haven't
> > > pinned down why yet.
> > 
> > We had some HTTP/2 stalls/deadlocks in the past, and they were dependent
> > on libcurl and apache (actually h2_mod) versions. IIRC some of the
> > non-TLS code paths for HTTP/2 were not well tested, which led to
> > 8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course
> > after that commit those cleartext code paths should not be a problem, so
> > that is probably not exactly the issue now.
> > 
> > But it might be worth checking the versions you're running locally
> > versus what's in the GitHub runner.
> 
> I didn't observe any similar hangs in GitLab's CI systems, so I wonder
> whether this is because of different versions of curl. And indeed we use
> different versions:
> 
>   - On GitHub we use 8.6.0.
> 
>   - On GitLab we use 8.7.1.
> 
> Now this of course doesn't mean that updating the curl version is the
> fix to this whole issue, as there's a ton of other factors that could
> play a role in whether or not the test hangs. So while we could just
> upgrade parts of the stack and cross our fingers, but that feels rather
> unsatisfactory. Still, one place to start could be to update our build
> images to macOS 15.
> 
> But the big question to me is whether the hang is because of a bug in
> Git with how we drive curl, a bug in curl itself, or a bug in Apache.

I noticed that a osx-clang job failed today in t5551 [1]. This time it
didn't hang, but produced an actual error:

    2026-06-22T09:25:45.1984230Z ++ git -C too-many-refs fetch -q --tags
    2026-06-22T09:25:45.1984420Z error: RPC failed; curl 18 transfer closed with outstanding read data remaining
    2026-06-22T09:25:45.1984520Z fatal: expected flush after ref listing
    2026-06-22T09:25:45.1984610Z error: last command exited with $?=128
    2026-06-22T09:25:45.1984660Z ++ rm -f tags
    2026-06-22T09:25:45.1984710Z ++ :
    2026-06-22T09:25:45.1984830Z not ok 35 - http can handle enormous ref negotiation

There was a second test failing similarly.

Patrick

[1]: https://github.com/git/git/actions/runs/27940620478/job/82672854726

Re: [RFH] Why do osx CI jobs so unreliable?

[Patrick Steinhardt]

On Mon, Jun 22, 2026 at 11:48:01AM +0200, Patrick Steinhardt wrote:
> On Mon, Jun 22, 2026 at 06:42:24AM +0200, Patrick Steinhardt wrote:
> > On Sun, Jun 21, 2026 at 05:34:07PM -0400, Jeff King wrote:
> > > On Sat, Jun 20, 2026 at 08:33:13AM -0700, Michael Montalbo wrote:
> [snip]
> > > > When it is wedged the whole chain sits at 0% CPU. upload-pack is
> > > > blocked in write() on the ls-refs advertisement, curl blocked in
> > > > select(). So it looks like an HTTP/2 flow-control stall on the
> > > > response side. The same stall resets itself after ~60-85s on my Linux
> > > > box and on a bare-metal Mac, but not on the GitHub runner; I haven't
> > > > pinned down why yet.
> > > 
> > > We had some HTTP/2 stalls/deadlocks in the past, and they were dependent
> > > on libcurl and apache (actually h2_mod) versions. IIRC some of the
> > > non-TLS code paths for HTTP/2 were not well tested, which led to
> > > 8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course
> > > after that commit those cleartext code paths should not be a problem, so
> > > that is probably not exactly the issue now.
> > > 
> > > But it might be worth checking the versions you're running locally
> > > versus what's in the GitHub runner.
> > 
> > I didn't observe any similar hangs in GitLab's CI systems, so I wonder
> > whether this is because of different versions of curl. And indeed we use
> > different versions:
> > 
> >   - On GitHub we use 8.6.0.
> > 
> >   - On GitLab we use 8.7.1.
> > 
> > Now this of course doesn't mean that updating the curl version is the
> > fix to this whole issue, as there's a ton of other factors that could
> > play a role in whether or not the test hangs. So while we could just
> > upgrade parts of the stack and cross our fingers, but that feels rather
> > unsatisfactory. Still, one place to start could be to update our build
> > images to macOS 15.
> > 
> > But the big question to me is whether the hang is because of a bug in
> > Git with how we drive curl, a bug in curl itself, or a bug in Apache.
> 
> I noticed that a osx-clang job failed today in t5551 [1]. This time it
> didn't hang, but produced an actual error:
> 
>     2026-06-22T09:25:45.1984230Z ++ git -C too-many-refs fetch -q --tags
>     2026-06-22T09:25:45.1984420Z error: RPC failed; curl 18 transfer closed with outstanding read data remaining
>     2026-06-22T09:25:45.1984520Z fatal: expected flush after ref listing
>     2026-06-22T09:25:45.1984610Z error: last command exited with $?=128
>     2026-06-22T09:25:45.1984660Z ++ rm -f tags
>     2026-06-22T09:25:45.1984710Z ++ :
>     2026-06-22T09:25:45.1984830Z not ok 35 - http can handle enormous ref negotiation
> 
> There was a second test failing similarly.

Oh, and Linux is also failing in the same test suite [1], even though
the job logs are truncated, so it's hard to say whether it's the same
failure or not.

There certainly seems to be a deeper issue here. We could of course just
disable the test again, but by now I do wonder whether this would paper
over an actual bug.

Patrick

[1]: https://github.com/git/git/actions/runs/27940620478/job/82672854864

Re: [RFH] Why do osx CI jobs so unreliable?

[Patrick Steinhardt]

On Mon, Jun 22, 2026 at 11:55:31AM +0200, Patrick Steinhardt wrote:
> On Mon, Jun 22, 2026 at 11:48:01AM +0200, Patrick Steinhardt wrote:
> > On Mon, Jun 22, 2026 at 06:42:24AM +0200, Patrick Steinhardt wrote:
> > > On Sun, Jun 21, 2026 at 05:34:07PM -0400, Jeff King wrote:
> > > > On Sat, Jun 20, 2026 at 08:33:13AM -0700, Michael Montalbo wrote:
> > [snip]
> > > > > When it is wedged the whole chain sits at 0% CPU. upload-pack is
> > > > > blocked in write() on the ls-refs advertisement, curl blocked in
> > > > > select(). So it looks like an HTTP/2 flow-control stall on the
> > > > > response side. The same stall resets itself after ~60-85s on my Linux
> > > > > box and on a bare-metal Mac, but not on the GitHub runner; I haven't
> > > > > pinned down why yet.
> > > > 
> > > > We had some HTTP/2 stalls/deadlocks in the past, and they were dependent
> > > > on libcurl and apache (actually h2_mod) versions. IIRC some of the
> > > > non-TLS code paths for HTTP/2 were not well tested, which led to
> > > > 8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course
> > > > after that commit those cleartext code paths should not be a problem, so
> > > > that is probably not exactly the issue now.
> > > > 
> > > > But it might be worth checking the versions you're running locally
> > > > versus what's in the GitHub runner.
> > > 
> > > I didn't observe any similar hangs in GitLab's CI systems, so I wonder
> > > whether this is because of different versions of curl. And indeed we use
> > > different versions:
> > > 
> > >   - On GitHub we use 8.6.0.
> > > 
> > >   - On GitLab we use 8.7.1.
> > > 
> > > Now this of course doesn't mean that updating the curl version is the
> > > fix to this whole issue, as there's a ton of other factors that could
> > > play a role in whether or not the test hangs. So while we could just
> > > upgrade parts of the stack and cross our fingers, but that feels rather
> > > unsatisfactory. Still, one place to start could be to update our build
> > > images to macOS 15.
> > > 
> > > But the big question to me is whether the hang is because of a bug in
> > > Git with how we drive curl, a bug in curl itself, or a bug in Apache.
> > 
> > I noticed that a osx-clang job failed today in t5551 [1]. This time it
> > didn't hang, but produced an actual error:
> > 
> >     2026-06-22T09:25:45.1984230Z ++ git -C too-many-refs fetch -q --tags
> >     2026-06-22T09:25:45.1984420Z error: RPC failed; curl 18 transfer closed with outstanding read data remaining
> >     2026-06-22T09:25:45.1984520Z fatal: expected flush after ref listing
> >     2026-06-22T09:25:45.1984610Z error: last command exited with $?=128
> >     2026-06-22T09:25:45.1984660Z ++ rm -f tags
> >     2026-06-22T09:25:45.1984710Z ++ :
> >     2026-06-22T09:25:45.1984830Z not ok 35 - http can handle enormous ref negotiation
> > 
> > There was a second test failing similarly.
> 
> Oh, and Linux is also failing in the same test suite [1], even though
> the job logs are truncated, so it's hard to say whether it's the same
> failure or not.
> 
> There certainly seems to be a deeper issue here. We could of course just
> disable the test again, but by now I do wonder whether this would paper
> over an actual bug.
> 
> Patrick
> 
> [1]: https://github.com/git/git/actions/runs/27940620478/job/82672854864

Sorry for the repeated spam.

I think the issue is rather simple: we're hitting timeouts in Apache. If
you apply the following diff:

diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 40a690b0bb..4054fe008f 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -302,3 +302,5 @@ RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]
 		SVNPath "${LIB_HTTPD_SVNPATH}"
 	</Location>
 </IfDefine>
+
+Timeout 1

Then you'll see the same errors locally:

    $ GIT_TEST_LONG=Yes meson test t5551-http-fetch-smart --test-args=-ix -i
    Failed to clone 'sub'. Retry scheduled
    Cloning into '/home/pks/Development/git/build/test-output/trash directory.t5551-http-fetch-smart/sub'...
    error: RPC failed; curl 18 transfer closed with outstanding read data remaining
    fatal: early EOF
    fatal: fetch-pack: invalid index-pack output
    fatal: clone of 'http://127.0.0.1:5551/smart_headers/repo.git' into submodule path '/home/pks/Development/git/build/test-output/trash directory.t5551-http-fetch-smart/sub' failed
    Failed to clone 'sub' a second time, aborting
    error: last command exited with $?=1
    not ok 36 - custom http headers
    #	
    #		test_must_fail git -c http.extraheader="x-magic-two: cadabra" \
    #			fetch "$HTTPD_URL/smart_headers/repo.git" &&
    #		git -c http.extraheader="x-magic-one: abra" \
    #		    -c http.extraheader="x-magic-two: cadabra" \
    #		    fetch "$HTTPD_URL/smart_headers/repo.git" &&
    #		git update-index --add --cacheinfo 160000,$(git rev-parse HEAD),sub &&
    #		git config -f .gitmodules submodule.sub.path sub &&
    #		git config -f .gitmodules submodule.sub.url \
    #			"$HTTPD_URL/smart_headers/repo.git" &&
    #		git submodule init sub &&
    #		test_must_fail git submodule update sub &&
    #		git -c http.extraheader="x-magic-one: abra" \
    #		    -c http.extraheader="x-magic-two: cadabra" \
    #			submodule update sub
    #	
    1..36

And Apache also logs this as a timeout:

    [Mon Jun 22 10:26:52.115717 2026] [cgi:warn] [pid 3686957:tid 3686957] [client 127.0.0.1:55114] AH01220: Timeout waiting for output from CGI script /home/pks/Development/git/build/git-http-backend
    [Mon Jun 22 10:26:52.115748 2026] [core:error] [pid 3686957:tid 3686957] (70007)The timeout specified has expired: [client 127.0.0.1:55114] AH00574: ap_content_length_filter: apr_bucket_read() failed
    [Mon Jun 22 10:27:01.567533 2026] [cgi:warn] [pid 3686958:tid 3686958] [client 127.0.0.1:54384] AH01220: Timeout waiting for output from CGI script /home/pks/Development/git/build/git-http-backend
    [Mon Jun 22 10:27:01.567559 2026] [core:error] [pid 3686958:tid 3686958] (70007)The timeout specified has expired: [client 127.0.0.1:54384] AH00574: ap_content_length_filter: apr_bucket_read() failed

This is because our keepalive mechanisms aren't helping:

  - The TCP-level keepalives don't help with Apache.

  - The application-level sideband keepalives don't apply to the
    "ls-refs" endpoint.

Whether that's the same issue like we see in macOS sometimes is a
different question.

Patrick