Re: macOS builds broken in CI

["Alex Bennée" <alex.bennee@linaro.org>]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Mon, Jul 20, 2026 at 5:38 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Thu, 25 Jun 2026 at 20:15, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
>> >
>> >
>> >
>> > > On 25. Jun 2026, at 21:11, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
>> > >
>> > >
>> > >
>> > >> On 25. Jun 2026, at 20:38, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> > >>
>> > >> Hi Alex,
>> > >> tests/tcg/multiarch/system/Makefile.softmmu-target's
>> > >> run-gdbstub-interrupt seems to be broken on GitLab CI macOS runners. 5
>> > >> days ago it still passed, but now rerunning the pipeline on the same
>> > >> commit ID fails:
>> > >>
>> > >> - Good (5 days ago): https://gitlab.com/qemu-project/qemu/-/pipelines/2615134022
>> > >> - Bad (today): https://gitlab.com/qemu-project/qemu/-/pipelines/2629704018
>> > >>
>> > >> I've tried many CI runs with different pull requests applied on top of
>> > >> qemu.git/master and they fail in the same way. Maybe it's an
>> > >> environmental thing with the macOS runners?
>> > >>
>> > >> It's unclear where the SIGKILL (Error 137) comes from. Could it be an
>> > >> out-of-memory condition on macOS runners?
>> > >
>> > > Hi,
>> > >
>> > > SIGKILL is the known symptom for failure during signature verification
>> > > by AMFI.
>> > >
>> > > Now that we don’t have separate -unsigned binaries anymore but relying
>> > > on in-place ad-hoc (w/o certificate) signing with entitlements we might
>> > > have issues linked to that.
>> > >
>> > > In the past file modification had issues with verification tied to the
>> > > inode and the workaround being a reboot or a cat binary > new_binary;
>> > > chmod +x new_binary so that the old (now out of sync) signing state is lost
>> > >
>> > This is highlighted in the software update context in: https://developer.apple.com/documentation/security/updating-mac-software
>> >
>> > > This code is incorrect because it modifies the command-line
>> > > tool’s executable file in place. macOS caches information about
>> > > the code’s signature in the kernel. It doesn’t flush that cache
>> > > when you modify the file’s contents. Modifying the file in place
>> > > yields a mismatch between the file’s contents and the in-kernel
>> > > cache, which can cause a hard-to-reproduce code-signing crash
>> > > the next time you run the tool.
>> > > While this example uses a command-line tool to demonstrate the issue, updating any file that contains signed code might trigger this code-signing crash. That includes executables, frameworks, dynamic libraries, and bundles.
>> > > To update a file that contains signed code without risking this crash, write the updated code to a temporary file and replace the existing file with that temporary one:
>>
>> I was just looking at this again today, but looking at our
>> scripts/entitlement.sh, when meson.build calls the script SRC
>> will be qemu-system-foo-unsigned and DST will be
>> qemu-system-foo, and we take the in_place case and do:
>>
>>   cp -pPf "$SRC" "$DST.tmp"
>>   SRC="$DST.tmp"
>>
>> to copy the -unsigned binary to a .tmp file. Then we
>> run codesign on that .tmp file, and finally
>>
>>   mv -f "$SRC" "$DST"
>>
>> to rename the signed binary to its final name.
>>
>> So that looks to me like we're following the rules for
>> not signing executables in place to me...
>
> Hmm...then the next step could be reproducing the job failure locally
> on a Mac where it can be debugged or bisected.

Unfortunately I can't reproduce it on the one piece of hardware I have
access to.

We can disable the gdb tests and the rest passes:

  https://gitlab.com/qemu-project/qemu-ci-testing/-/pipelines/2690820623

>
> Please let me know if you need help on the GitLab CI side. Mac runners
> are available to open source projects, so you might not be able to
> test in a personal fork
> (https://docs.gitlab.com/ci/runners/hosted_runners/macos/). Peter, I
> think you have the same GitLab permissions as I do, so you could
> probably run experiments without my help.
>
> Stefan

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro