qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 00/27] configure: create a python venv and ensure meson, sphinx
@ 2023-05-11  3:54 John Snow
  2023-05-11  3:54 ` [PATCH 01/27] python: shut up "pip install" during "make check-minreqs" John Snow
                   ` (26 more replies)
  0 siblings, 27 replies; 37+ messages in thread
From: John Snow @ 2023-05-11  3:54 UTC (permalink / raw)
  To: qemu-devel
  Cc: Warner Losh, Peter Maydell, Daniel P. Berrangé, Ani Sinha,
	Beraldo Leal, Markus Armbruster, Ryo ONODERA, Paolo Bonzini,
	Kyle Evans, Alex Bennée, Michael Roth, Reinoud Zandijk,
	Marc-André Lureau, Cleber Rosa, Thomas Huth,
	Michael S. Tsirkin, John Snow, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta

This patch series creates a mandatory python virtual environment
("venv") during configure time and uses it to ensure the availability of
meson and sphinx.

See https://www.qemu.org/2023/03/24/python/ for motivations. The summary
is that the goal of this series is to ensure that the `python` used to
run meson is the same `python` used to run Sphinx, tests, and any
build-time python scripting we have. As it stands, meson and sphinx (and
their extensions) *may* run in a different python environment than the
one configured and chosen by the user at configure/build time.

The effective change of this series is that QEMU will now
unconditionally create a venv at configure-time and will ensure that
meson (and sphinx, if docs are enabled) are available through that venv.

Some important points as a pre-emptive "FAQ":

- This venv is unconditionally created and lives at {build_dir}/pyvenv.

- The python interpreter used by this venv is always the one identified
  by configure. (Which in turn is always the one specified by --python
  or $PYTHON)

- *almost* all python scripts in qemu.git executed as part of the build
  system, meson, sphinx, avocado tests, vm tests or CI are always
  executed within this venv.

  (iotests are not yet integrated; I plan to tackle this separately as a
  follow-up in order to have a more tightly focused scope on that
  series.)

- It remains possible to build and test fully offline.
  (In most cases, you just need meson and sphinx from your distro's repo.)

- Distribution packaged 'meson' and 'sphinx' are still utilized whenever
  possible as the highest preference.

- Vendored versions of e.g. 'meson' are always preferred to PyPI
  versions for speed, repeatability and ensuring tarball builds work
  as-is offline.

  (Sphinx will not be vendored, just like it already isn't.)

- Missing dependencies, when possible, are fetched and installed
  on-demand automatically to make developer environments "just work".

- Works for Python 3.6 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
  Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere

- No new dependencies (...for most platforms. Debian and NetBSD get an
  asterisk.)

- The meson git submodule is unused after this series and can be removed.

... I could write more, and I'm sure there's a few things that need
buffing out or changing here and there, but I think it's largely good.

So, enjoy!

~js

John Snow (22):
  python: update pylint configuration
  python: add mkvenv.py
  mkvenv: add better error message for missing pyexpat module
  mkvenv: add nested venv workaround
  mkvenv: add ensure subcommand
  mkvenv: add diagnose() method for ensure() failures
  mkvenv: add console script entry point generation
  mkvenv: work around broken pip installations on Debian 10
  tests/docker: add python3-venv dependency
  tests/vm: Configure netbsd to use Python 3.10
  tests/vm: add py310-expat to NetBSD
  python: add vendor.py utility
  configure: create a python venv unconditionally
  python/wheels: add vendored meson package
  configure: use 'mkvenv ensure meson' to bootstrap meson
  qemu.git: drop meson git submodule
  tests: Use configure-provided pyvenv for tests
  configure: move --enable-docs and --disable-docs back to configure
  configure: bootstrap sphinx with mkvenv
  configure: add --enable-pypi and --disable-pypi
  configure: Add courtesy hint to Python version failure message
  mkvenv.py: experiment; use distlib to generate script entry points

Paolo Bonzini (5):
  python: shut up "pip install" during "make check-minreqs"
  mkvenv: create pip binary in virtual environment
  Python: Drop support for Python 3.6
  mkvenv: mark command as required
  python: bump some of the dependencies

 docs/about/build-platforms.rst                |   2 +-
 docs/conf.py                                  |   9 -
 docs/devel/acpi-bits.rst                      |   6 +-
 docs/devel/testing.rst                        |  14 +-
 docs/meson.build                              |   2 +-
 configure                                     | 155 ++--
 .gitlab-ci.d/buildtest-template.yml           |   4 +-
 .gitlab-ci.d/buildtest.yml                    |   6 +-
 .gitmodules                                   |   3 -
 meson                                         |   1 -
 meson_options.txt                             |   2 -
 python/Makefile                               |  19 +-
 python/scripts/mkvenv.py                      | 858 ++++++++++++++++++
 python/scripts/vendor.py                      |  74 ++
 python/setup.cfg                              |  27 +-
 python/tests/flake8.sh                        |   1 +
 python/tests/isort.sh                         |   1 +
 python/tests/minreqs.txt                      |  16 +-
 python/tests/mypy.sh                          |   1 +
 python/tests/pylint.sh                        |   1 +
 python/wheels/meson-0.61.5-py3-none-any.whl   | Bin 0 -> 862509 bytes
 .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
 scripts/device-crash-test                     |   2 +-
 scripts/meson-buildoptions.sh                 |   3 -
 scripts/qapi/mypy.ini                         |   2 +-
 tests/Makefile.include                        |  10 +-
 .../dockerfiles/debian-all-test-cross.docker  |   3 +-
 .../dockerfiles/debian-hexagon-cross.docker   |   3 +-
 .../dockerfiles/debian-riscv64-cross.docker   |   3 +-
 .../dockerfiles/debian-tricore-cross.docker   |   3 +-
 tests/requirements.txt                        |   7 +-
 tests/vm/netbsd                               |   2 +
 32 files changed, 1108 insertions(+), 136 deletions(-)
 delete mode 160000 meson
 create mode 100644 python/scripts/mkvenv.py
 create mode 100755 python/scripts/vendor.py
 create mode 100644 python/wheels/meson-0.61.5-py3-none-any.whl

-- 
2.40.0




^ permalink raw reply	[flat|nested] 37+ messages in thread

end of thread, other threads:[~2023-05-11 16:16 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-05-11  3:54 [PATCH 00/27] configure: create a python venv and ensure meson, sphinx John Snow
2023-05-11  3:54 ` [PATCH 01/27] python: shut up "pip install" during "make check-minreqs" John Snow
2023-05-11  3:54 ` [PATCH 02/27] python: update pylint configuration John Snow
2023-05-11  3:54 ` [PATCH 03/27] python: add mkvenv.py John Snow
2023-05-11  3:54 ` [PATCH 04/27] mkvenv: add better error message for missing pyexpat module John Snow
2023-05-11  3:54 ` [PATCH 05/27] mkvenv: add nested venv workaround John Snow
2023-05-11  3:54 ` [PATCH 06/27] mkvenv: add ensure subcommand John Snow
2023-05-11  3:54 ` [PATCH 07/27] mkvenv: add diagnose() method for ensure() failures John Snow
2023-05-11  6:53   ` Paolo Bonzini
2023-05-11 15:53     ` John Snow
2023-05-11 15:56       ` Paolo Bonzini
2023-05-11 15:59         ` John Snow
2023-05-11  3:54 ` [PATCH 08/27] mkvenv: add console script entry point generation John Snow
2023-05-11  3:54 ` [PATCH 09/27] mkvenv: create pip binary in virtual environment John Snow
2023-05-11  3:54 ` [PATCH 10/27] mkvenv: work around broken pip installations on Debian 10 John Snow
2023-05-11  3:54 ` [PATCH 11/27] tests/docker: add python3-venv dependency John Snow
2023-05-11  3:54 ` [PATCH 12/27] tests/vm: Configure netbsd to use Python 3.10 John Snow
2023-05-11  3:54 ` [PATCH 13/27] tests/vm: add py310-expat to NetBSD John Snow
2023-05-11  3:54 ` [PATCH 14/27] python: add vendor.py utility John Snow
2023-05-11  3:54 ` [PATCH 15/27] configure: create a python venv unconditionally John Snow
2023-05-11  3:54 ` [PATCH 16/27] python/wheels: add vendored meson package John Snow
2023-05-11  3:54 ` [PATCH 17/27] configure: use 'mkvenv ensure meson' to bootstrap meson John Snow
2023-05-11  3:54 ` [PATCH 18/27] qemu.git: drop meson git submodule John Snow
2023-05-11  3:54 ` [PATCH 19/27] tests: Use configure-provided pyvenv for tests John Snow
2023-05-11  3:54 ` [PATCH 20/27] configure: move --enable-docs and --disable-docs back to configure John Snow
2023-05-11  3:54 ` [PATCH 21/27] configure: bootstrap sphinx with mkvenv John Snow
2023-05-11  3:54 ` [PATCH 22/27] configure: add --enable-pypi and --disable-pypi John Snow
2023-05-11  3:54 ` [PATCH 23/27] Python: Drop support for Python 3.6 John Snow
2023-05-11  3:54 ` [PATCH 24/27] configure: Add courtesy hint to Python version failure message John Snow
2023-05-11  3:54 ` [PATCH 25/27] mkvenv: mark command as required John Snow
2023-05-11  3:54 ` [PATCH 26/27] python: bump some of the dependencies John Snow
2023-05-11  3:54 ` [PATCH 27/27] mkvenv.py: experiment; use distlib to generate script entry points John Snow
2023-05-11  6:57   ` Paolo Bonzini
2023-05-11  7:02   ` Paolo Bonzini
2023-05-11 15:58     ` John Snow
2023-05-11 16:14       ` Paolo Bonzini
2023-05-11 16:16         ` John Snow

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).