[RFC PATCH 0/2] buildsys: Use host meson on macOS Sonoma (14)

[RFC PATCH 0/2] buildsys: Use host meson on macOS Sonoma (14)

[Philippe Mathieu-Daudé]

macOS 14 "Sonoma" was released end of September 2023,
developers who upgraded they host can not build QEMU
anymore due to some issue with meson, described in
https://gitlab.com/qemu-project/qemu/-/issues/1939
(mkvenv ignore the host meson and forces to use the
old meson 0.63.3, which doesn't work on Sonoma).

I have no clue how to fix this meson issue, but I put
these 2 patches to keep testing Sonoma out-of-tree
during the release freeze window.

Successful build:
https://gitlab.com/philmd/qemu/-/jobs/5491256155

Based-on: <20231108162022.76189-1-philmd@linaro.org>
  .gitlab-ci.d/cirrus: Upgrade macOS to 13 (Ventura)
  https://lore.kernel.org/qemu-devel/20231108162022.76189-1-philmd@linaro.org/

Philippe Mathieu-Daudé (2):
  configure: Use distrib meson on macOS
  .gitlab-ci.d/cirrus: Add manual testing of macOS 14 (Sonoma)

 configure                         |  9 +--------
 .gitlab-ci.d/cirrus.yml           | 17 +++++++++++++++++
 .gitlab-ci.d/cirrus/macos-14.vars | 16 ++++++++++++++++
 tests/lcitool/libvirt-ci          |  2 +-
 tests/lcitool/refresh             |  1 +
 5 files changed, 36 insertions(+), 9 deletions(-)
 create mode 100644 .gitlab-ci.d/cirrus/macos-14.vars

-- 
2.41.0

[NOTFORMERGE PATCH 1/2] configure: Use distrib meson

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 configure | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/configure b/configure
index f1456f6123..93735b7661 100755
--- a/configure
+++ b/configure
@@ -955,15 +955,8 @@ if $python -c 'import sys; sys.exit(sys.version_info >= (3,11))'; then
     $mkvenv ensure --dir "${source_path}/python/wheels" \
         'tomli>=1.2.0' || exit 1
 fi
-$mkvenv ensuregroup --dir "${source_path}/python/wheels" \
-     ${source_path}/pythondeps.toml meson || exit 1
 
-# At this point, we expect Meson to be installed and available.
-# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
-# We ignore PATH completely here: we want to use the venv's Meson
-# *exclusively*.
-
-meson="$(cd pyvenv/bin; pwd)/meson"
+meson="$(which meson)"
 
 # Conditionally ensure Sphinx is installed.
 
-- 
2.41.0

Re: [NOTFORMERGE PATCH 1/2] configure: Use distrib meson

[Peter Maydell]

On Wed, 8 Nov 2023 at 18:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  configure | 9 +--------
>  1 file changed, 1 insertion(+), 8 deletions(-)
>
> diff --git a/configure b/configure
> index f1456f6123..93735b7661 100755
> --- a/configure
> +++ b/configure
> @@ -955,15 +955,8 @@ if $python -c 'import sys; sys.exit(sys.version_info >= (3,11))'; then
>      $mkvenv ensure --dir "${source_path}/python/wheels" \
>          'tomli>=1.2.0' || exit 1
>  fi
> -$mkvenv ensuregroup --dir "${source_path}/python/wheels" \
> -     ${source_path}/pythondeps.toml meson || exit 1
>
> -# At this point, we expect Meson to be installed and available.
> -# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
> -# We ignore PATH completely here: we want to use the venv's Meson
> -# *exclusively*.
> -
> -meson="$(cd pyvenv/bin; pwd)/meson"
> +meson="$(which meson)"

Presumably the other approach would be to make the pythondeps.toml
say "on macos we need at least version X of meson" ?

Do we know what the upstream meson issue is that we're running into
with the older version on this host ?

thanks
-- PMM

Re: [NOTFORMERGE PATCH 1/2] configure: Use distrib meson

[BALATON Zoltan]

[-- Attachment #1: Type: text/plain, Size: 1984 bytes --]

On Wed, 8 Nov 2023, Peter Maydell wrote:
> On Wed, 8 Nov 2023 at 18:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>  configure | 9 +--------
>>  1 file changed, 1 insertion(+), 8 deletions(-)
>>
>> diff --git a/configure b/configure
>> index f1456f6123..93735b7661 100755
>> --- a/configure
>> +++ b/configure
>> @@ -955,15 +955,8 @@ if $python -c 'import sys; sys.exit(sys.version_info >= (3,11))'; then
>>      $mkvenv ensure --dir "${source_path}/python/wheels" \
>>          'tomli>=1.2.0' || exit 1
>>  fi
>> -$mkvenv ensuregroup --dir "${source_path}/python/wheels" \
>> -     ${source_path}/pythondeps.toml meson || exit 1
>>
>> -# At this point, we expect Meson to be installed and available.
>> -# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
>> -# We ignore PATH completely here: we want to use the venv's Meson
>> -# *exclusively*.
>> -
>> -meson="$(cd pyvenv/bin; pwd)/meson"
>> +meson="$(which meson)"

Doesn't this revert 66e2c6cbace ?

> Presumably the other approach would be to make the pythondeps.toml
> say "on macos we need at least version X of meson" ?
>
> Do we know what the upstream meson issue is that we're running into
> with the older version on this host ?

Linked from the QEMU ticket in the cover letter:
https://github.com/mesonbuild/meson/issues/12399

AFAIU the problem is that macOS has system python in 
/Library/Frameworks/python.framework (or something like that) but homebrew 
does not use that but installs its own python somewhere in /opt and meson 
from homebrew uses that python but configure finds the system python which 
does not have meson so it then downloads its own meson but that's too old 
for macOS Sonoma. It should probably check for homebrew or macports 
locations too or check for meson and get the python that's using. A 
workaround apparently is to pass --python with the right path to 
configure.

Regards,
BALATON Zoltan

Re: [NOTFORMERGE PATCH 1/2] configure: Use distrib meson

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1596 bytes --]

Il mer 8 nov 2023, 20:52 BALATON Zoltan <balaton@eik.bme.hu> ha scritto:

> AFAIU the problem is that macOS has system python in
> /Library/Frameworks/python.framework (or something like that) but homebrew
> does not use that but installs its own python somewhere in /opt and meson
> from homebrew uses that python but configure finds the system python which
> does not have meson so it then downloads its own meson but that's too old
> for macOS Sonoma


Correct (except s/downloads/installs/), basically --python/$PYTHON is the
sole interpreter used to run *any* Python code during the QEMU build. The
difference is that QEMU 8.1 would indeed download Meson, while QEMU 8.2
always uses the wheel that is shipped with QEMU. This change was done
because the embedded Meson is intended to be a "known good" version that we
(and pythondeps.toml) can point users to, in case a newer version from the
distro has issues. Of course the other side of the coin is that an older
version might not support new OSes as in this case.

The workaround below is not just a workaround but the intended solution if
one wants to use the Meson and Sphinx packages from homebrew, and cannot
modify the PATH to privilege homebrew's Python interpreter. In order to
support Sonoma out of the box with the system Python, however, we can bump
the shipped version of Meson to 1.2.x (and adjust pythondeps.toml).

Paolo

It should probably check for homebrew or macports
> locations too or check for meson and get the python that's using. A
> workaround apparently is to pass --python with the right path to
> configure.
>

[-- Attachment #2: Type: text/html, Size: 2277 bytes --]

[PATCH 2/2] .gitlab-ci.d/cirrus: Add manual testing of macOS 14 (Sonoma)

[Philippe Mathieu-Daudé]

Upgrade libvirt-ci so it covers macOS 14. Add a manual entry
(QEMU_JOB_OPTIONAL: 1) to test on Sonoma release. Refresh the
lci-tool generated files.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
Docs are disabled due to:

  Collecting sphinx
    Downloading sphinx-7.2.6-py3-none-any.whl.metadata (5.9 kB)
  Collecting sphinx-rtd-theme
    Downloading sphinx_rtd_theme-1.3.0-py2.py3-none-any.whl.metadata (4.5 kB)
  ...
  Installing collected packages ...
  ...
  python determined to be '/opt/homebrew/bin/python3'
  python version: Python 3.11.6
  mkvenv: Creating non-isolated virtual environment at 'pyvenv'
  mkvenv: checking for sphinx>=1.6
  mkvenv: checking for sphinx_rtd_theme>=0.5
  ...
  Program /opt/homebrew/opt/python@3.12/bin/sphinx-build found: NO
  ../docs/meson.build:1:15: ERROR: Program '/opt/homebrew/opt/python@3.12/bin/sphinx-build' not found or not executable

¯\_(ツ)_/¯
---
 .gitlab-ci.d/cirrus.yml           | 17 +++++++++++++++++
 .gitlab-ci.d/cirrus/macos-14.vars | 16 ++++++++++++++++
 tests/lcitool/libvirt-ci          |  2 +-
 tests/lcitool/refresh             |  1 +
 4 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci.d/cirrus/macos-14.vars

diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 07dc6edae1..84ce143509 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -74,6 +74,23 @@ aarch64-macos-13-base-build:
     PKG_CONFIG_PATH: /opt/homebrew/curl/lib/pkgconfig:/opt/homebrew/ncurses/lib/pkgconfig:/opt/homebrew/readline/lib/pkgconfig
     TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
 
+aarch64-macos-14-base-build:
+  extends: .cirrus_build_job
+  variables:
+    NAME: macos-14
+    CIRRUS_VM_INSTANCE_TYPE: macos_instance
+    CIRRUS_VM_IMAGE_SELECTOR: image
+    CIRRUS_VM_IMAGE_NAME: ghcr.io/cirruslabs/macos-sonoma-base:latest
+    CIRRUS_VM_CPUS: 12
+    CIRRUS_VM_RAM: 24G
+    UPDATE_COMMAND: brew update
+    INSTALL_COMMAND: brew install
+    PATH_EXTRA: /opt/homebrew/ccache/libexec:/opt/homebrew/gettext/bin
+    PKG_CONFIG_PATH: /opt/homebrew/curl/lib/pkgconfig:/opt/homebrew/ncurses/lib/pkgconfig:/opt/homebrew/readline/lib/pkgconfig
+    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
+    QEMU_JOB_OPTIONAL: 1
+    CONFIGURE_ARGS: --disable-docs
+
 
 # The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job
 .cirrus_kvm_job:
diff --git a/.gitlab-ci.d/cirrus/macos-14.vars b/.gitlab-ci.d/cirrus/macos-14.vars
new file mode 100644
index 0000000000..43070f4a26
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/macos-14.vars
@@ -0,0 +1,16 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+#  $ lcitool variables macos-14 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci
+
+CCACHE='/opt/homebrew/bin/ccache'
+CPAN_PKGS=''
+CROSS_PKGS=''
+MAKE='/opt/homebrew/bin/gmake'
+NINJA='/opt/homebrew/bin/ninja'
+PACKAGING_COMMAND='brew'
+PIP3='/opt/homebrew/bin/pip3'
+PKGS='bash bc bison bzip2 capstone ccache cmocka ctags curl dbus diffutils dtc flex gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo json-c libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb llvm lzo make meson mtools ncurses nettle ninja pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy socat sparse spice-protocol swtpm tesseract usbredir vde vte3 xorriso zlib zstd'
+PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme tomli'
+PYTHON='/opt/homebrew/bin/python3'
diff --git a/tests/lcitool/libvirt-ci b/tests/lcitool/libvirt-ci
index 36bc517161..77c800186f 160000
--- a/tests/lcitool/libvirt-ci
+++ b/tests/lcitool/libvirt-ci
@@ -1 +1 @@
-Subproject commit 36bc517161c45ead20224d47f2dc4fa428af6724
+Subproject commit 77c800186f34b21be7660750577cc5582a914deb
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 2259f131b4..e11d0ba381 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -197,6 +197,7 @@ try:
     #
     generate_cirrus("freebsd-13")
     generate_cirrus("macos-13")
+    generate_cirrus("macos-14")
 
     #
     # VM packages lists
-- 
2.41.0

Re: [PATCH 2/2] .gitlab-ci.d/cirrus: Add manual testing of macOS 14 (Sonoma)

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 4723 bytes --]

Il mer 8 nov 2023, 19:33 Philippe Mathieu-Daudé <philmd@linaro.org> ha
scritto:

> Upgrade libvirt-ci so it covers macOS 14. Add a manual entry
> (QEMU_JOB_OPTIONAL: 1) to test on Sonoma release. Refresh the
> lci-tool generated files.
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
> Docs are disabled due to:
>
>   Collecting sphinx
>     Downloading sphinx-7.2.6-py3-none-any.whl.metadata (5.9 kB)
>   Collecting sphinx-rtd-theme
>     Downloading sphinx_rtd_theme-1.3.0-py2.py3-none-any.whl.metadata (4.5
> kB)
>

Same issue: the pip that is installing sphinx is unrelated to the Python
that is used to compile QEMU. Use /opt/homebrew/bin/pip3 instead.

Paolo

  ...
>   Installing collected packages ...
>   ...
>   python determined to be '/opt/homebrew/bin/python3'
>   python version: Python 3.11.6
>   mkvenv: Creating non-isolated virtual environment at 'pyvenv'
>   mkvenv: checking for sphinx>=1.6
>   mkvenv: checking for sphinx_rtd_theme>=0.5
>   ...
>   Program /opt/homebrew/opt/python@3.12/bin/sphinx-build found: NO
>   ../docs/meson.build:1:15: ERROR: Program '/opt/homebrew/opt/python@3.12/bin/sphinx-build'
> not found or not executable
>
> ¯\_(ツ)_/¯
> ---
>  .gitlab-ci.d/cirrus.yml           | 17 +++++++++++++++++
>  .gitlab-ci.d/cirrus/macos-14.vars | 16 ++++++++++++++++
>  tests/lcitool/libvirt-ci          |  2 +-
>  tests/lcitool/refresh             |  1 +
>  4 files changed, 35 insertions(+), 1 deletion(-)
>  create mode 100644 .gitlab-ci.d/cirrus/macos-14.vars
>
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> index 07dc6edae1..84ce143509 100644
> --- a/.gitlab-ci.d/cirrus.yml
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -74,6 +74,23 @@ aarch64-macos-13-base-build:
>      PKG_CONFIG_PATH:
> /opt/homebrew/curl/lib/pkgconfig:/opt/homebrew/ncurses/lib/pkgconfig:/opt/homebrew/readline/lib/pkgconfig
>      TEST_TARGETS: check-unit check-block check-qapi-schema
> check-softfloat check-qtest-x86_64
>
> +aarch64-macos-14-base-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: macos-14
> +    CIRRUS_VM_INSTANCE_TYPE: macos_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image
> +    CIRRUS_VM_IMAGE_NAME: ghcr.io/cirruslabs/macos-sonoma-base:latest
> +    CIRRUS_VM_CPUS: 12
> +    CIRRUS_VM_RAM: 24G
> +    UPDATE_COMMAND: brew update
> +    INSTALL_COMMAND: brew install
> +    PATH_EXTRA: /opt/homebrew/ccache/libexec:/opt/homebrew/gettext/bin
> +    PKG_CONFIG_PATH:
> /opt/homebrew/curl/lib/pkgconfig:/opt/homebrew/ncurses/lib/pkgconfig:/opt/homebrew/readline/lib/pkgconfig
> +    TEST_TARGETS: check-unit check-block check-qapi-schema
> check-softfloat check-qtest-x86_64
> +    QEMU_JOB_OPTIONAL: 1
> +    CONFIGURE_ARGS: --disable-docs
> +
>
>  # The following jobs run VM-based tests via KVM on a Linux-based
> Cirrus-CI job
>  .cirrus_kvm_job:
> diff --git a/.gitlab-ci.d/cirrus/macos-14.vars
> b/.gitlab-ci.d/cirrus/macos-14.vars
> new file mode 100644
> index 0000000000..43070f4a26
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/macos-14.vars
> @@ -0,0 +1,16 @@
> +# THIS FILE WAS AUTO-GENERATED
> +#
> +#  $ lcitool variables macos-14 qemu
> +#
> +# https://gitlab.com/libvirt/libvirt-ci
> +
> +CCACHE='/opt/homebrew/bin/ccache'
> +CPAN_PKGS=''
> +CROSS_PKGS=''
> +MAKE='/opt/homebrew/bin/gmake'
> +NINJA='/opt/homebrew/bin/ninja'
> +PACKAGING_COMMAND='brew'
> +PIP3='/opt/homebrew/bin/pip3'
> +PKGS='bash bc bison bzip2 capstone ccache cmocka ctags curl dbus
> diffutils dtc flex gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc
> jpeg-turbo json-c libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp
> libssh libtasn1 libusb llvm lzo make meson mtools ncurses nettle ninja
> pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy socat sparse
> spice-protocol swtpm tesseract usbredir vde vte3 xorriso zlib zstd'
> +PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme tomli'
> +PYTHON='/opt/homebrew/bin/python3'
> diff --git a/tests/lcitool/libvirt-ci b/tests/lcitool/libvirt-ci
> index 36bc517161..77c800186f 160000
> --- a/tests/lcitool/libvirt-ci
> +++ b/tests/lcitool/libvirt-ci
> @@ -1 +1 @@
> -Subproject commit 36bc517161c45ead20224d47f2dc4fa428af6724
> +Subproject commit 77c800186f34b21be7660750577cc5582a914deb
> diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
> index 2259f131b4..e11d0ba381 100755
> --- a/tests/lcitool/refresh
> +++ b/tests/lcitool/refresh
> @@ -197,6 +197,7 @@ try:
>      #
>      generate_cirrus("freebsd-13")
>      generate_cirrus("macos-13")
> +    generate_cirrus("macos-14")
>
>      #
>      # VM packages lists
> --
> 2.41.0
>
>

[-- Attachment #2: Type: text/html, Size: 5971 bytes --]