macOS builds broken in CI

macOS builds broken in CI

[Stefan Hajnoczi]

Hi Alex,
tests/tcg/multiarch/system/Makefile.softmmu-target's
run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
days ago it still passed, but now rerunning the pipeline on the same
commit ID fails:

- Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
- Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018

I've tried many CI runs with different pull requests applied on top of
qemu.git/master and they fail in the same way. Maybe it's an
environmental thing with the macOS runners?

It's unclear where the SIGKILL (Error 137) comes from. Could it be an
out-of-memory condition on macOS runners?

I am ignoring the macOS builds for the time being and continuing to
merge pull requests.

TEST memory-sve on aarch64
TEST softmmu gdbstub support on aarch64
qemu-system-aarch64: -gdb
unix:path=/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmporwqcbrdqemu-gdbstub/gdbstub.socket,server=on:
info: QEMU waiting for connection on:
disconnected:unix:/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmporwqcbrdqemu-gdbstub/gdbstub.socket,server=on
qemu-system-aarch64: QEMU: Terminated via GDBstub
TEST softmmu gdbstub support on aarch64
qemu-system-aarch64: -gdb
unix:path=/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmpkfurbf2mqemu-gdbstub/gdbstub.socket,server=on:
info: QEMU waiting for connection on:
disconnected:unix:/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmpkfurbf2mqemu-gdbstub/gdbstub.socket,server=on
gmake[1]: *** [/Users/gitlab/builds/qemu-project/qemu/tests/tcg/multiarch/system/Makefile.softmmu-target:35:
run-gdbstub-interrupt] Error 137
gmake: *** [/Users/gitlab/builds/qemu-project/qemu/tests/Makefile.include:75:
run-tcg-tests-aarch64-softmmu] Error 2
WARNING: step_script could not run to completion because the timeout
was exceeded. For more control over job and script timeouts see:
https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts

https://gitlab.com/qemu-project/qemu/-/jobs/15038503463

Thanks,
Stefan

Re: macOS builds broken in CI

[Mohamed Mediouni]

> On 25. Jun 2026, at 20:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
> Hi Alex,
> tests/tcg/multiarch/system/Makefile.softmmu-target's
> run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
> days ago it still passed, but now rerunning the pipeline on the same
> commit ID fails:
> 
> - Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
> - Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018
> 
> I've tried many CI runs with different pull requests applied on top of
> qemu.git/master and they fail in the same way. Maybe it's an
> environmental thing with the macOS runners?
> 
> It's unclear where the SIGKILL (Error 137) comes from. Could it be an
> out-of-memory condition on macOS runners?

Hi,

SIGKILL is the known symptom for failure during signature verification
by AMFI.

Now that we don’t have separate -unsigned binaries anymore but relying
on in-place ad-hoc (w/o certificate) signing with entitlements we might
have issues linked to that.

In the past file modification had issues with verification tied to the
inode and the workaround being a reboot or a cat binary > new_binary;
chmod +x new_binary so that the old (now out of sync) signing state is lost

> 
> I am ignoring the macOS builds for the time being and continuing to
> merge pull requests.
> 
> TEST memory-sve on aarch64
> TEST softmmu gdbstub support on aarch64
> qemu-system-aarch64: -gdb
> unix:path=/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmporwqcbrdqemu-gdbstub/gdbstub.socket,server=on:
> info: QEMU waiting for connection on:
> disconnected:unix:/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmporwqcbrdqemu-gdbstub/gdbstub.socket,server=on
> qemu-system-aarch64: QEMU: Terminated via GDBstub
> TEST softmmu gdbstub support on aarch64
> qemu-system-aarch64: -gdb
> unix:path=/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmpkfurbf2mqemu-gdbstub/gdbstub.socket,server=on:
> info: QEMU waiting for connection on:
> disconnected:unix:/var/folders/fc/8_hzw5tx3n7c9c04tzd6tv_40000gn/T/tmpkfurbf2mqemu-gdbstub/gdbstub.socket,server=on
> gmake[1]: *** [/Users/gitlab/builds/qemu-project/qemu/tests/tcg/multiarch/system/Makefile.softmmu-target:35:
> run-gdbstub-interrupt] Error 137
> gmake: *** [/Users/gitlab/builds/qemu-project/qemu/tests/Makefile.include:75:
> run-tcg-tests-aarch64-softmmu] Error 2
> WARNING: step_script could not run to completion because the timeout
> was exceeded. For more control over job and script timeouts see:
> https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/15038503463
> 
> Thanks,
> Stefan
> 

Re: macOS builds broken in CI

[Mohamed Mediouni]

> On 25. Jun 2026, at 21:11, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
> 
> 
> 
>> On 25. Jun 2026, at 20:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> 
>> Hi Alex,
>> tests/tcg/multiarch/system/Makefile.softmmu-target's
>> run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
>> days ago it still passed, but now rerunning the pipeline on the same
>> commit ID fails:
>> 
>> - Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
>> - Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018
>> 
>> I've tried many CI runs with different pull requests applied on top of
>> qemu.git/master and they fail in the same way. Maybe it's an
>> environmental thing with the macOS runners?
>> 
>> It's unclear where the SIGKILL (Error 137) comes from. Could it be an
>> out-of-memory condition on macOS runners?
> 
> Hi,
> 
> SIGKILL is the known symptom for failure during signature verification
> by AMFI.
> 
> Now that we don’t have separate -unsigned binaries anymore but relying
> on in-place ad-hoc (w/o certificate) signing with entitlements we might
> have issues linked to that.
> 
> In the past file modification had issues with verification tied to the
> inode and the workaround being a reboot or a cat binary > new_binary;
> chmod +x new_binary so that the old (now out of sync) signing state is lost
> 
This is highlighted in the software update context in: https://developer.apple.com/documentation/security/updating-mac-software

> This code is incorrect because it modifies the command-line tool’s executable file in place. macOS caches information about the code’s signature in the kernel. It doesn’t flush that cache when you modify the file’s contents. Modifying the file in place yields a mismatch between the file’s contents and the in-kernel cache, which can cause a hard-to-reproduce code-signing crash the next time you run the tool.
> While this example uses a command-line tool to demonstrate the issue, updating any file that contains signed code might trigger this code-signing crash. That includes executables, frameworks, dynamic libraries, and bundles.
> To update a file that contains signed code without risking this crash, write the updated code to a temporary file and replace the existing file with that temporary one:

Re: macOS builds broken in CI

[Stefan Hajnoczi]

On Thu, Jun 25, 2026 at 3:15 PM Mohamed Mediouni
<mohamed@unpredictable.fr> wrote:
>
>
>
> > On 25. Jun 2026, at 21:11, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
> >
> >
> >
> >> On 25. Jun 2026, at 20:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >>
> >> Hi Alex,
> >> tests/tcg/multiarch/system/Makefile.softmmu-target's
> >> run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
> >> days ago it still passed, but now rerunning the pipeline on the same
> >> commit ID fails:
> >>
> >> - Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
> >> - Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018
> >>
> >> I've tried many CI runs with different pull requests applied on top of
> >> qemu.git/master and they fail in the same way. Maybe it's an
> >> environmental thing with the macOS runners?
> >>
> >> It's unclear where the SIGKILL (Error 137) comes from. Could it be an
> >> out-of-memory condition on macOS runners?
> >
> > Hi,
> >
> > SIGKILL is the known symptom for failure during signature verification
> > by AMFI.
> >
> > Now that we don’t have separate -unsigned binaries anymore but relying
> > on in-place ad-hoc (w/o certificate) signing with entitlements we might
> > have issues linked to that.
> >
> > In the past file modification had issues with verification tied to the
> > inode and the workaround being a reboot or a cat binary > new_binary;
> > chmod +x new_binary so that the old (now out of sync) signing state is lost
> >
> This is highlighted in the software update context in: https://developer.apple.com/documentation/security/updating-mac-software
>
> > This code is incorrect because it modifies the command-line tool’s executable file in place. macOS caches information about the code’s signature in the kernel. It doesn’t flush that cache when you modify the file’s contents. Modifying the file in place yields a mismatch between the file’s contents and the in-kernel cache, which can cause a hard-to-reproduce code-signing crash the next time you run the tool.
> > While this example uses a command-line tool to demonstrate the issue, updating any file that contains signed code might trigger this code-signing crash. That includes executables, frameworks, dynamic libraries, and bundles.
> > To update a file that contains signed code without risking this crash, write the updated code to a temporary file and replace the existing file with that temporary one:
>

Nice, thanks for sharing this! Do you want to send a patch to fix this?

Stefan

Re: macOS builds broken in CI

[Stefan Hajnoczi]

On Thu, Jun 25, 2026 at 9:53 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Thu, Jun 25, 2026 at 3:15 PM Mohamed Mediouni
> <mohamed@unpredictable.fr> wrote:
> >
> >
> >
> > > On 25. Jun 2026, at 21:11, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
> > >
> > >
> > >
> > >> On 25. Jun 2026, at 20:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >>
> > >> Hi Alex,
> > >> tests/tcg/multiarch/system/Makefile.softmmu-target's
> > >> run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
> > >> days ago it still passed, but now rerunning the pipeline on the same
> > >> commit ID fails:
> > >>
> > >> - Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
> > >> - Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018
> > >>
> > >> I've tried many CI runs with different pull requests applied on top of
> > >> qemu.git/master and they fail in the same way. Maybe it's an
> > >> environmental thing with the macOS runners?
> > >>
> > >> It's unclear where the SIGKILL (Error 137) comes from. Could it be an
> > >> out-of-memory condition on macOS runners?
> > >
> > > Hi,
> > >
> > > SIGKILL is the known symptom for failure during signature verification
> > > by AMFI.
> > >
> > > Now that we don’t have separate -unsigned binaries anymore but relying
> > > on in-place ad-hoc (w/o certificate) signing with entitlements we might
> > > have issues linked to that.
> > >
> > > In the past file modification had issues with verification tied to the
> > > inode and the workaround being a reboot or a cat binary > new_binary;
> > > chmod +x new_binary so that the old (now out of sync) signing state is lost
> > >
> > This is highlighted in the software update context in: https://developer.apple.com/documentation/security/updating-mac-software
> >
> > > This code is incorrect because it modifies the command-line tool’s executable file in place. macOS caches information about the code’s signature in the kernel. It doesn’t flush that cache when you modify the file’s contents. Modifying the file in place yields a mismatch between the file’s contents and the in-kernel cache, which can cause a hard-to-reproduce code-signing crash the next time you run the tool.
> > > While this example uses a command-line tool to demonstrate the issue, updating any file that contains signed code might trigger this code-signing crash. That includes executables, frameworks, dynamic libraries, and bundles.
> > > To update a file that contains signed code without risking this crash, write the updated code to a temporary file and replace the existing file with that temporary one:
> >
>
> Nice, thanks for sharing this! Do you want to send a patch to fix this?

The macOS CI jobs have been failing for a week. The QEMU 11.1 soft
freeze is on July 7th and I'm concerned that we will enter the 11.1
freeze without working macOS builds in the CI system.

Does anyone want to fix this macOS host issue?

Stefan

Re: macOS builds broken in CI

[Stefan Hajnoczi]

(Update Philippe's email address.)

On Wed, Jul 1, 2026 at 7:23 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Thu, Jun 25, 2026 at 9:53 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > On Thu, Jun 25, 2026 at 3:15 PM Mohamed Mediouni
> > <mohamed@unpredictable.fr> wrote:
> > >
> > >
> > >
> > > > On 25. Jun 2026, at 21:11, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
> > > >
> > > >
> > > >
> > > >> On 25. Jun 2026, at 20:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > > >>
> > > >> Hi Alex,
> > > >> tests/tcg/multiarch/system/Makefile.softmmu-target's
> > > >> run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
> > > >> days ago it still passed, but now rerunning the pipeline on the same
> > > >> commit ID fails:
> > > >>
> > > >> - Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
> > > >> - Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018
> > > >>
> > > >> I've tried many CI runs with different pull requests applied on top of
> > > >> qemu.git/master and they fail in the same way. Maybe it's an
> > > >> environmental thing with the macOS runners?
> > > >>
> > > >> It's unclear where the SIGKILL (Error 137) comes from. Could it be an
> > > >> out-of-memory condition on macOS runners?
> > > >
> > > > Hi,
> > > >
> > > > SIGKILL is the known symptom for failure during signature verification
> > > > by AMFI.
> > > >
> > > > Now that we don’t have separate -unsigned binaries anymore but relying
> > > > on in-place ad-hoc (w/o certificate) signing with entitlements we might
> > > > have issues linked to that.
> > > >
> > > > In the past file modification had issues with verification tied to the
> > > > inode and the workaround being a reboot or a cat binary > new_binary;
> > > > chmod +x new_binary so that the old (now out of sync) signing state is lost
> > > >
> > > This is highlighted in the software update context in: https://developer.apple.com/documentation/security/updating-mac-software
> > >
> > > > This code is incorrect because it modifies the command-line tool’s executable file in place. macOS caches information about the code’s signature in the kernel. It doesn’t flush that cache when you modify the file’s contents. Modifying the file in place yields a mismatch between the file’s contents and the in-kernel cache, which can cause a hard-to-reproduce code-signing crash the next time you run the tool.
> > > > While this example uses a command-line tool to demonstrate the issue, updating any file that contains signed code might trigger this code-signing crash. That includes executables, frameworks, dynamic libraries, and bundles.
> > > > To update a file that contains signed code without risking this crash, write the updated code to a temporary file and replace the existing file with that temporary one:
> > >
> >
> > Nice, thanks for sharing this! Do you want to send a patch to fix this?
>
> The macOS CI jobs have been failing for a week. The QEMU 11.1 soft
> freeze is on July 7th and I'm concerned that we will enter the 11.1
> freeze without working macOS builds in the CI system.
>
> Does anyone want to fix this macOS host issue?
>
> Stefan

Re: macOS builds broken in CI

[Alex Bennée]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> (Update Philippe's email address.)
>
> On Wed, Jul 1, 2026 at 7:23 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>
>> On Thu, Jun 25, 2026 at 9:53 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> >
>> > On Thu, Jun 25, 2026 at 3:15 PM Mohamed Mediouni
>> > <mohamed@unpredictable.fr> wrote:
>> > >
>> > >
>> > >
>> > > > On 25. Jun 2026, at 21:11, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
>> > > >
>> > > >
>> > > >
>> > > >> On 25. Jun 2026, at 20:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> > > >>
>> > > >> Hi Alex,
>> > > >> tests/tcg/multiarch/system/Makefile.softmmu-target's
>> > > >> run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
>> > > >> days ago it still passed, but now rerunning the pipeline on the same
>> > > >> commit ID fails:
>> > > >>
>> > > >> - Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
>> > > >> - Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018
>> > > >>
>> > > >> I've tried many CI runs with different pull requests applied on top of
>> > > >> qemu.git/master and they fail in the same way. Maybe it's an
>> > > >> environmental thing with the macOS runners?
>> > > >>
>> > > >> It's unclear where the SIGKILL (Error 137) comes from. Could it be an
>> > > >> out-of-memory condition on macOS runners?
>> > > >
>> > > > Hi,
>> > > >
>> > > > SIGKILL is the known symptom for failure during signature verification
>> > > > by AMFI.
>> > > >
>> > > > Now that we don’t have separate -unsigned binaries anymore but relying
>> > > > on in-place ad-hoc (w/o certificate) signing with entitlements we might
>> > > > have issues linked to that.
>> > > >
>> > > > In the past file modification had issues with verification tied to the
>> > > > inode and the workaround being a reboot or a cat binary > new_binary;
>> > > > chmod +x new_binary so that the old (now out of sync) signing state is lost
>> > > >
>> > > This is highlighted in the software update context in: https://developer.apple.com/documentation/security/updating-mac-software
>> > >
>> > > > This code is incorrect because it modifies the command-line
>> > > > tool’s executable file in place. macOS caches information
>> > > > about the code’s signature in the kernel. It doesn’t flush
>> > > > that cache when you modify the file’s contents. Modifying the
>> > > > file in place yields a mismatch between the file’s contents
>> > > > and the in-kernel cache, which can cause a hard-to-reproduce
>> > > > code-signing crash the next time you run the tool.
>> > > > While this example uses a command-line tool to demonstrate the
>> > > > issue, updating any file that contains signed code might
>> > > > trigger this code-signing crash. That includes executables,
>> > > > frameworks, dynamic libraries, and bundles.
>> > > > To update a file that contains signed code without risking this crash, write the updated code to a temporary file and replace the existing file with that temporary one:
>> > >
>> >
>> > Nice, thanks for sharing this! Do you want to send a patch to fix this?
>>
>> The macOS CI jobs have been failing for a week. The QEMU 11.1 soft
>> freeze is on July 7th and I'm concerned that we will enter the 11.1
>> freeze without working macOS builds in the CI system.
>>
>> Does anyone want to fix this macOS host issue?

I'll have a look tomorrow.

>>
>> Stefan

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro