Re: [PATCH v2 00/27] configure: create a python venv and ensure meson, sphinx

[John Snow <jsnow@redhat.com>]

On Tue, May 16, 2023 at 6:57 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> This patch series creates a mandatory python virtual environment
> ("venv") during configure time and uses it to ensure the availability of
> meson and sphinx.
>
> See https://www.qemu.org/2023/03/24/python/ for motivations. The summary
> is that the goal of this series is to ensure that the `python` used to
> run meson is the same `python` used to run Sphinx, tests, and any
> build-time python scripting we have. As it stands, meson and sphinx (and
> their extensions) *may* run in a different python environment than the
> one configured and chosen by the user at configure/build time.
>
> The effective change of this series is that QEMU will now
> unconditionally create a venv at configure-time and will ensure that
> meson (and sphinx, if docs are enabled) are available through that venv.
>
> Some important points as a pre-emptive "FAQ":
>
> - Works for Python 3.6 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
>   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
>   No new dependencies (...for most platforms. Debian and NetBSD get an
>   asterisk, see patch 4).
>
> - The venv is unconditionally created and lives at {build_dir}/pyvenv.
>   The python interpreter used by this venv is always the one identified
>   by configure. (Which in turn is always the one specified by --python
>   or $PYTHON).  --meson and --sphinx-build disappear.
>
> - *almost* all python scripts in qemu.git executed as part of the build
>   system, meson, sphinx, avocado tests, vm tests or CI are always
>   executed within this venv.  iotests are not yet integrated.
>
> - Missing dependencies, when possible, are fetched and installed
>   on-demand automatically to make developer environments "just work".
>   However, it remains possible to build and test fully offline.  Right
>   now, online behavior is only triggered by --enable-docs, and even
>   in that case...
>
> - ... distribution packaged 'meson' and 'sphinx' are still utilized
>   whenever possible as the highest preference.
>
> - The bundled 'meson' takes the shape of a .whl file, replacing the
>   git submodule.  qemu-qmp will follow suit, while Sphinx will not be
>   bundled just like it already isn't.
>
> - about 13% of the mkvenv.py script is devoted to supporting Python
>   3.7 and Debian 10. ¯\_(ツ)_/¯
>

I like that you measured this. It feels like more than 13%, but I'm
going to trust this oddly specific number.

> Tested with "make check-minreqs" after every patch and with GitLab at
> https://gitlab.com/bonzini/qemu/-/pipelines/868627762.

For v1, I also tested with "make check-minreqs && make check-tox" and
also started checking/formatting with black --line-length=79. I
haven't added black into the linters yet because there's surely a ton
of code that needs observed before I do that, but Dan suggested it and
I liked it enough to start using it myself to take the ambiguity out
of how to handle long lines.

(Not worth a respin if black isn't happy, I didn't check. Just mentioning it.)

>
> Enjoy!
>
> John and Paolo
>
> v1->v2:
> - integrate distlib for script generation and also to look up
>   installed packages

Thanks :)

> - print a more precise error for --diagnose when running in an
>   isolated venv
> - reorganized code to avoid ensurepip, so that it is not spread
>   across multiple patches
> - made non-fatal errors a bit less scary

"Oops" => "Oopsies!!"

>
>
> John Snow (23):
>   python: update pylint configuration
>   python: add mkvenv.py
>   mkvenv: add better error message for broken or missing ensurepip
>   mkvenv: add nested venv workaround
>   mkvenv: add ensure subcommand
>   mkvenv: add --diagnose option to explain "ensure" failures
>   mkvenv: add console script entry point generation
>   mkvenv: use pip's vendored distlib as a fallback
>   mkvenv: avoid ensurepip if pip is installed
>   mkvenv: work around broken pip installations on Debian 10
>   tests/docker: add python3-venv dependency
>   tests/vm: Configure netbsd to use Python 3.10
>   tests/vm: add py310-expat to NetBSD
>   python: add vendor.py utility
>   configure: create a python venv unconditionally
>   python/wheels: add vendored meson package
>   configure: use 'mkvenv ensure meson' to bootstrap meson
>   qemu.git: drop meson git submodule
>   tests: Use configure-provided pyvenv for tests
>   configure: move --enable-docs and --disable-docs back to configure
>   configure: bootstrap sphinx with mkvenv
>   configure: add --enable-pypi and --disable-pypi
>   configure: Add courtesy hint to Python version failure message
>
> Paolo Bonzini (4):
>   python: shut up "pip install" during "make check-minreqs"
>   Python: Drop support for Python 3.6
>   mkvenv: mark command as required
>   python: bump some of the dependencies
>
>  .gitlab-ci.d/buildtest-template.yml           |   4 +-
>  .gitlab-ci.d/buildtest.yml                    |   6 +-
>  .gitmodules                                   |   3 -
>  configure                                     | 152 +--
>  docs/about/build-platforms.rst                |   2 +-
>  docs/conf.py                                  |   9 -
>  docs/devel/acpi-bits.rst                      |   6 +-
>  docs/devel/testing.rst                        |  14 +-
>  docs/meson.build                              |   2 +-
>  meson                                         |   1 -
>  meson_options.txt                             |   2 -
>  python/Makefile                               |  19 +-
>  python/scripts/mkvenv.py                      | 897 ++++++++++++++++++
>  python/scripts/vendor.py                      |  74 ++
>  python/setup.cfg                              |  46 +-
>  python/tests/flake8.sh                        |   1 +
>  python/tests/isort.sh                         |   1 +
>  python/tests/minreqs.txt                      |  19 +-
>  python/tests/mypy.sh                          |   1 +
>  python/tests/pylint.sh                        |   1 +
>  python/wheels/meson-0.61.5-py3-none-any.whl   | Bin 0 -> 862509 bytes
>  .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
>  scripts/device-crash-test                     |   2 +-
>  scripts/meson-buildoptions.sh                 |   3 -
>  scripts/qapi/mypy.ini                         |   2 +-
>  tests/Makefile.include                        |  10 +-
>  .../dockerfiles/debian-all-test-cross.docker  |   3 +-
>  .../dockerfiles/debian-hexagon-cross.docker   |   3 +-
>  .../dockerfiles/debian-riscv64-cross.docker   |   3 +-
>  .../dockerfiles/debian-tricore-cross.docker   |   3 +-
>  tests/requirements.txt                        |   7 +-
>  tests/vm/netbsd                               |   2 +
>  32 files changed, 1167 insertions(+), 135 deletions(-)
>  delete mode 160000 meson
>  create mode 100644 python/scripts/mkvenv.py
>  create mode 100755 python/scripts/vendor.py
>  create mode 100644 python/wheels/meson-0.61.5-py3-none-any.whl
>
> --
> 2.40.1
>