* [RFC PATCH v2 01/10] python: add mkvenv.py
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 02/10] tests: add python3-venv dependency John Snow
` (8 subsequent siblings)
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
This script will be responsible for building a lightweight Python
virtual environment at configure time. It works with Python 3.7 or
newer.
It has been designed to:
- work *offline*, no PyPI required.
- work *quickly*, The fast path is only ~65ms on my machine.
- work *robustly*, with multiple fallbacks to keep things working.
- work *cooperatively*, using system packages where possible.
(You can use your distro's meson, no problem.)
Due to its unique position in the build chain, it exists outside of the
installable python packages in-tree and *must* be runnable without any
third party dependencies.
Under normal circumstances, the only dependency required to execute this
script is Python 3.7+ itself. The script is *faster* by several seconds
when setuptools and pip are installed in the host environment, which is
probably the case for a typical multi-purpose developer workstation.
In the event that pip/setuptools are missing or not usable, additional
dependencies may be required on some distributions which remove certain
Python stdlib modules to package them separately:
- Debian may require python3-venv to provide "ensurepip"
- NetBSD may require py310-expat to provide "pyexpat" *
(* Or whichever version is current for NetBSD.)
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/scripts/mkvenv.py | 379 +++++++++++++++++++++++++++++++++++++++
1 file changed, 379 insertions(+)
create mode 100644 python/scripts/mkvenv.py
diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
new file mode 100644
index 0000000000..6df95382f3
--- /dev/null
+++ b/python/scripts/mkvenv.py
@@ -0,0 +1,379 @@
+"""
+mkvenv - QEMU venv bootstrapping utility
+
+usage: TODO/FIXME
+"""
+
+# Copyright (C) 2022-2023 Red Hat, Inc.
+#
+# Authors:
+# John Snow <jsnow@redhat.com>
+# Paolo Bonzini <pbonzini@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import argparse
+from importlib.util import find_spec
+import logging
+import os
+from os import PathLike
+from pathlib import Path
+import re
+import stat
+import subprocess
+import sys
+import traceback
+from types import SimpleNamespace
+from typing import Dict, Sequence, Optional, Union, Iterator
+import venv
+
+
+# Do not add any mandatory dependencies from outside the stdlib:
+# This script *must* be usable standalone!
+
+DirType = Union[str, bytes, 'PathLike[str]', 'PathLike[bytes]']
+logger = logging.getLogger('mkvenv')
+
+
+class Ouch(RuntimeError):
+ """An Exception class we can't confuse with a builtin."""
+
+
+class QemuEnvBuilder(venv.EnvBuilder):
+ """
+ An extension of venv.EnvBuilder for building QEMU's configure-time venv.
+
+ The only functional change is that it adds the ability to regenerate
+ console_script shims for packages available via system_site
+ packages.
+
+ Parameters for base class init:
+ - system_site_packages: bool = False
+ - clear: bool = False
+ - symlinks: bool = False
+ - upgrade: bool = False
+ - with_pip: bool = False
+ - prompt: Optional[str] = None
+ - upgrade_deps: bool = False (Since 3.9)
+ """
+ def __init__(self, *args, **kwargs) -> None:
+ logger.debug("QemuEnvBuilder.__init__(...)")
+ self.script_packages = kwargs.pop('script_packages', ())
+ super().__init__(*args, **kwargs)
+
+ # The EnvBuilder class is cute and toggles this setting off
+ # before post_setup, but we need it to decide if we want to
+ # generate shims or not.
+ self._system_site_packages = self.system_site_packages
+
+ def post_setup(self, context: SimpleNamespace) -> None:
+ logger.debug("post_setup(...)")
+
+ # Generate console_script entry points for system packages:
+ if self._system_site_packages:
+ generate_console_scripts(
+ context.env_exe, context.bin_path, self.script_packages)
+
+ # print the python executable to stdout for configure.
+ print(context.env_exe)
+
+
+def need_ensurepip() -> bool:
+ """
+ Tests for the presence of setuptools and pip.
+
+ :return: `True` if we do not detect both packages.
+ """
+ # Don't try to actually import them, it's fraught with danger:
+ # https://github.com/pypa/setuptools/issues/2993
+ if find_spec("setuptools") and find_spec("pip"):
+ return False
+ return True
+
+
+def check_ensurepip(with_pip: bool) -> None:
+ """
+ Check that we have ensurepip.
+
+ Raise a fatal exception with a helpful hint if it isn't available.
+ """
+ if with_pip and not find_spec("ensurepip"):
+ msg = ("Python's ensurepip module is not found.\n"
+
+ "It's normally part of the Python standard library, "
+ "maybe your distribution packages it separately?\n"
+
+ "Either install ensurepip, or alleviate the need for it in the "
+ "first place by installing pip and setuptools for "
+ f"'{sys.executable}'.\n"
+
+ "(Hint: Debian puts ensurepip in its python3-venv package.)")
+ raise Ouch(msg)
+
+
+def make_venv( # pylint: disable=too-many-arguments
+ venv_path: Union[str, Path],
+ system_site_packages: bool = False,
+ clear: bool = True,
+ symlinks: Optional[bool] = None,
+ upgrade: bool = False,
+ with_pip: Optional[bool] = None,
+ script_packages: Sequence[str]=(),
+) -> None:
+ """
+ Create a venv using the QemuEnvBuilder class.
+
+ TODO/FIXME: write docs O:-)
+ """
+ logging.debug("%s: make_venv(venv_path=%s, system_site_packages=%s, "
+ "clear=%s, symlinks=%s, upgrade=%s, with_pip=%s, "
+ "script_packages=%s)",
+ __file__, str(venv_path), system_site_packages,
+ clear, symlinks, upgrade, with_pip, script_packages)
+
+ print(f"MKVENV {str(venv_path)}", file=sys.stderr)
+
+ # ensurepip is slow: venv creation can be very fast for cases where
+ # we allow the use of system_site_packages. Toggle ensure_pip on only
+ # in the cases where we really need it.
+ if with_pip is None:
+ with_pip = True if not system_site_packages else need_ensurepip()
+ logger.debug("with_pip unset, choosing %s", with_pip)
+
+ check_ensurepip(with_pip)
+
+ if symlinks is None:
+ # Default behavior of standard venv CLI
+ symlinks = os.name != "nt"
+
+ builder = QemuEnvBuilder(
+ system_site_packages=system_site_packages,
+ clear=clear,
+ symlinks=symlinks,
+ upgrade=upgrade,
+ with_pip=with_pip,
+ script_packages=script_packages,
+ )
+ try:
+ logger.debug("Invoking builder.create()")
+ try:
+ builder.create(str(venv_path))
+ except SystemExit as exc:
+ # Some versions of the venv module raise SystemExit; *nasty*!
+ # We want the exception that prompted it. It might be a subprocess
+ # error that has output we *really* want to see.
+ logger.debug("Intercepted SystemExit from EnvBuilder.create()")
+ raise exc.__cause__ or exc.__context__ or exc
+ logger.debug("builder.create() finished")
+ except subprocess.CalledProcessError as exc:
+ print(f"cmd: {exc.cmd}", file=sys.stderr)
+ print(f"returncode: {exc.returncode}", file=sys.stderr)
+
+ def _stringify(data: Union[str, bytes]) -> str:
+ if data and isinstance(data, bytes):
+ return data.decode()
+ return data
+
+ if exc.stdout:
+ print("========== stdout ==========",
+ _stringify(exc.stdout),
+ "============================",
+ sep='\n', file=sys.stderr)
+ if exc.stderr:
+ print("========== stderr ==========",
+ _stringify(exc.stderr),
+ "============================",
+ sep='\n', file=sys.stderr)
+ raise Ouch("VENV creation subprocess failed.") from exc
+
+
+def _gen_importlib(packages: Sequence[str]) -> Iterator[Dict[str, str]]:
+ # pylint: disable=import-outside-toplevel
+ try:
+ # First preference: Python 3.8+ stdlib
+ from importlib.metadata import (
+ PackageNotFoundError,
+ distribution,
+ )
+ except ImportError as exc:
+ logger.debug("%s", str(exc))
+ # Second preference: Commonly available PyPI backport
+ from importlib_metadata import (
+ PackageNotFoundError,
+ distribution,
+ )
+
+ # Borrowed from CPython (Lib/importlib/metadata/__init__.py)
+ pattern = re.compile(
+ r'(?P<module>[\w.]+)\s*'
+ r'(:\s*(?P<attr>[\w.]+)\s*)?'
+ r'((?P<extras>\[.*\])\s*)?$'
+ )
+
+ def _generator() -> Iterator[Dict[str, str]]:
+ for package in packages:
+ try:
+ entry_points = distribution(package).entry_points
+ except PackageNotFoundError:
+ continue
+
+ # The EntryPoints type is only available in 3.10+,
+ # treat this as a vanilla list and filter it ourselves.
+ entry_points = filter(
+ lambda ep: ep.group == 'console_scripts', entry_points)
+
+ for entry_point in entry_points:
+ # Python 3.8 doesn't have 'module' or 'attr' attributes
+ if not (hasattr(entry_point, 'module') and
+ hasattr(entry_point, 'attr')):
+ match = pattern.match(entry_point.value)
+ assert match is not None
+ module = match.group('module')
+ attr = match.group('attr')
+ else:
+ module = entry_point.module
+ attr = entry_point.attr
+ yield {
+ 'name': entry_point.name,
+ 'module': module,
+ 'import_name': attr,
+ 'func': attr,
+ }
+
+ return _generator()
+
+
+def _gen_pkg_resources(packages: Sequence[str]) -> Iterator[Dict[str, str]]:
+ # pylint: disable=import-outside-toplevel
+ # Bundled with setuptools; has a good chance of being available.
+ import pkg_resources
+
+ def _generator() -> Iterator[Dict[str, str]]:
+ for package in packages:
+ try:
+ eps = pkg_resources.get_entry_map(package, 'console_scripts')
+ except pkg_resources.DistributionNotFound:
+ continue
+
+ for entry_point in eps.values():
+ yield {
+ 'name': entry_point.name,
+ 'module': entry_point.module_name,
+ 'import_name': ".".join(entry_point.attrs),
+ 'func': ".".join(entry_point.attrs),
+ }
+
+ return _generator()
+
+
+# Borrowed/adapted from pip's vendored version of distutils:
+SCRIPT_TEMPLATE = r'''#!{python_path:s}
+# -*- coding: utf-8 -*-
+import re
+import sys
+from {module:s} import {import_name:s}
+if __name__ == '__main__':
+ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
+ sys.exit({func:s}())
+'''
+
+
+def generate_console_scripts(python_path: str, bin_path: str,
+ packages: Sequence[str]) -> None:
+ """
+ Generate script shims for console_script entry points in @packages.
+ """
+ if not packages:
+ return
+
+ def _get_entry_points() -> Iterator[Dict[str, str]]:
+ """Python 3.7 compatibility shim for iterating entry points."""
+ # Python 3.8+, or Python 3.7 with importlib_metadata installed.
+ try:
+ return _gen_importlib(packages)
+ except ImportError as exc:
+ logger.debug("%s", str(exc))
+
+ # Python 3.7 with setuptools installed.
+ try:
+ return _gen_pkg_resources(packages)
+ except ImportError as exc:
+ logger.debug("%s", str(exc))
+ raise Ouch(
+ "Neither importlib.metadata nor pkg_resources found, "
+ "can't generate console script shims.\n"
+ "Use Python 3.8+, or install importlib-metadata, or setuptools."
+ ) from exc
+
+ for entry_point in _get_entry_points():
+ script_path = os.path.join(bin_path, entry_point['name'])
+ script = SCRIPT_TEMPLATE.format(python_path=python_path, **entry_point)
+ with open(script_path, "w", encoding='UTF-8') as file:
+ file.write(script)
+ mode = os.stat(script_path).st_mode | stat.S_IEXEC
+ os.chmod(script_path, mode)
+
+ logger.debug("wrote '%s'", script_path)
+
+
+def main() -> int:
+ """CLI interface to make_qemu_venv. See module docstring."""
+ if os.environ.get('DEBUG') or os.environ.get('GITLAB_CI'):
+ # You're welcome.
+ logging.basicConfig(level=logging.DEBUG)
+ elif os.environ.get('V'):
+ logging.basicConfig(level=logging.INFO)
+
+ parser = argparse.ArgumentParser(description="Bootstrap QEMU venv.")
+ subparsers = parser.add_subparsers(
+ title="Commands",
+ description="Various actions this utility can perform",
+ prog="prog",
+ dest="command",
+ required=True,
+ metavar="command",
+ help='Description')
+
+ subparser = subparsers.add_parser('create', help='create a venv')
+ subparser.add_argument(
+ '--gen',
+ type=str,
+ action='append',
+ help="Regenerate console_scripts for given packages, if found.",
+ )
+ subparser.add_argument(
+ 'target',
+ type=str,
+ action='store',
+ help="Target directory to install virtual environment into.",
+ )
+
+ args = parser.parse_args()
+ try:
+ if args.command == 'create':
+ script_packages = []
+ for element in args.gen or ():
+ script_packages.extend(element.split(","))
+ make_venv(
+ args.target,
+ system_site_packages=True,
+ clear=True,
+ upgrade=False,
+ with_pip=None, # Autodetermine
+ script_packages=script_packages,
+ )
+ logger.debug("mkvenv.py create - exiting")
+ except Ouch as exc:
+ print("\n*** Ouch! ***\n", file=sys.stderr)
+ print(str(exc), "\n\n", file=sys.stderr)
+ return 1
+ except: # pylint: disable=bare-except
+ print("mkvenv did not complete successfully:", file=sys.stderr)
+ traceback.print_exc()
+ return 2
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 02/10] tests: add python3-venv dependency
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
2023-04-14 5:54 ` [RFC PATCH v2 01/10] python: add mkvenv.py John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 14:17 ` Philippe Mathieu-Daudé
2023-04-14 5:54 ` [RFC PATCH v2 03/10] mkvenv: Add better error message for missing pyexapt module John Snow
` (7 subsequent siblings)
9 siblings, 1 reply; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
Several debian-based tests need the python3-venv dependency as a
consequence of Debian debundling the "ensurepip" module normally
included with Python.
As mkvenv.py stands as of this commit, Debian requires EITHER:
(A) setuptools and pip, or
(B) ensurepip
mkvenv is a few seconds faster if you have setuptools and pip, so
developers should prefer the first requirement. For the purposes of CI,
the time-save is a wash; it's only a matter of who is responsible for
installing pip and when; the timing is about the same.
Arbitrarily, I chose adding ensurepip to the test configuration because
it is normally part of the Python stdlib, and always having it allows us
a more consistent cross-platform environment.
Signed-off-by: John Snow <jsnow@redhat.com>
---
tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++-
tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++-
tests/docker/dockerfiles/debian-riscv64-cross.docker | 3 ++-
tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++-
4 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/docker/dockerfiles/debian-all-test-cross.docker
index 981e9bdc7b..f9f401544a 100644
--- a/tests/docker/dockerfiles/debian-all-test-cross.docker
+++ b/tests/docker/dockerfiles/debian-all-test-cross.docker
@@ -57,7 +57,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
gcc-sh4-linux-gnu \
libc6-dev-sh4-cross \
gcc-sparc64-linux-gnu \
- libc6-dev-sparc64-cross
+ libc6-dev-sparc64-cross \
+ python3-venv
ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
index b99d99f943..c2cfb6a5d0 100644
--- a/tests/docker/dockerfiles/debian-hexagon-cross.docker
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -20,7 +20,8 @@ RUN apt-get update && \
bison \
flex \
git \
- ninja-build && \
+ ninja-build \
+ python3-venv && \
# Install QEMU build deps for use in CI
DEBIAN_FRONTEND=noninteractive eatmydata \
apt build-dep -yy --arch-only qemu
diff --git a/tests/docker/dockerfiles/debian-riscv64-cross.docker b/tests/docker/dockerfiles/debian-riscv64-cross.docker
index 803afb9573..081404e014 100644
--- a/tests/docker/dockerfiles/debian-riscv64-cross.docker
+++ b/tests/docker/dockerfiles/debian-riscv64-cross.docker
@@ -28,7 +28,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy \
libglib2.0-dev \
ninja-build \
pkg-config \
- python3
+ python3 \
+ python3-venv
# Add ports and riscv64 architecture
RUN echo "deb http://ftp.ports.debian.org/debian-ports/ sid main" >> /etc/apt/sources.list
diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index cfd2faf9a8..269bfa8d42 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -33,7 +33,8 @@ RUN apt update && \
pkgconf \
python3-pip \
python3-setuptools \
- python3-wheel
+ python3-wheel \
+ python3-venv
RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \
| tar -xzC /usr/local/
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [RFC PATCH v2 02/10] tests: add python3-venv dependency
2023-04-14 5:54 ` [RFC PATCH v2 02/10] tests: add python3-venv dependency John Snow
@ 2023-04-14 14:17 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 12+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-04-14 14:17 UTC (permalink / raw)
To: John Snow, qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, Ryo ONODERA,
Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta
On 14/4/23 07:54, John Snow wrote:
> Several debian-based tests need the python3-venv dependency as a
> consequence of Debian debundling the "ensurepip" module normally
> included with Python.
>
> As mkvenv.py stands as of this commit, Debian requires EITHER:
>
> (A) setuptools and pip, or
> (B) ensurepip
>
> mkvenv is a few seconds faster if you have setuptools and pip, so
> developers should prefer the first requirement. For the purposes of CI,
> the time-save is a wash; it's only a matter of who is responsible for
> installing pip and when; the timing is about the same.
>
> Arbitrarily, I chose adding ensurepip to the test configuration because
> it is normally part of the Python stdlib, and always having it allows us
> a more consistent cross-platform environment.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
> tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++-
> tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++-
> tests/docker/dockerfiles/debian-riscv64-cross.docker | 3 ++-
> tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++-
> 4 files changed, 8 insertions(+), 4 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 03/10] mkvenv: Add better error message for missing pyexapt module
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
2023-04-14 5:54 ` [RFC PATCH v2 01/10] python: add mkvenv.py John Snow
2023-04-14 5:54 ` [RFC PATCH v2 02/10] tests: add python3-venv dependency John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 04/10] tests/vm: Configure netbsd to use Python 3.10 John Snow
` (6 subsequent siblings)
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
NetBSD debundles pyexpat from python, but ensurepip needs pyexpat. Try
our best to offer a helpful error message instead of just failing
catastrophically.
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/scripts/mkvenv.py | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 6df95382f3..a3284e9ef1 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -98,7 +98,10 @@ def check_ensurepip(with_pip: bool) -> None:
Raise a fatal exception with a helpful hint if it isn't available.
"""
- if with_pip and not find_spec("ensurepip"):
+ if not with_pip:
+ return
+
+ if not find_spec("ensurepip"):
msg = ("Python's ensurepip module is not found.\n"
"It's normally part of the Python standard library, "
@@ -111,6 +114,20 @@ def check_ensurepip(with_pip: bool) -> None:
"(Hint: Debian puts ensurepip in its python3-venv package.)")
raise Ouch(msg)
+ # ensurepip uses pyexpat, which can also go missing on us:
+ if not find_spec("pyexpat"):
+ msg = ("Python's pyexpat module is not found.\n"
+
+ "It's normally part of the Python standard library, "
+ "maybe your distribution packages it separately?\n"
+
+ "Either install pyexpat, or alleviate the need for it in the "
+ "first place by installing pip and setuptools for "
+ f"'{sys.executable}'.\n\n"
+
+ "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)")
+ raise Ouch(msg)
+
def make_venv( # pylint: disable=too-many-arguments
venv_path: Union[str, Path],
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 04/10] tests/vm: Configure netbsd to use Python 3.10
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
` (2 preceding siblings ...)
2023-04-14 5:54 ` [RFC PATCH v2 03/10] mkvenv: Add better error message for missing pyexapt module John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 05/10] tests/vm: add py310-expat to NetBSD John Snow
` (5 subsequent siblings)
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
NetBSD removes some packages from the Python stdlib, but only
re-packages them for Python 3.10. Switch to using Python 3.10.
Signed-off-by: John Snow <jsnow@redhat.com>
---
tests/vm/netbsd | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index 0b9536ca17..13eae109c0 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -30,6 +30,7 @@ class NetBSDVM(basevm.BaseVM):
"git-base",
"pkgconf",
"xz",
+ "python310",
"ninja-build",
# gnu tools
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 05/10] tests/vm: add py310-expat to NetBSD
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
` (3 preceding siblings ...)
2023-04-14 5:54 ` [RFC PATCH v2 04/10] tests/vm: Configure netbsd to use Python 3.10 John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 06/10] mkvenv: generate console entry shims from inside the venv John Snow
` (4 subsequent siblings)
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
NetBSD cannot successfully run "ensurepip" without access to the pyexpat
module, which NetBSD debundles. Like the Debian patch, it would be
strictly faster long term to install pip/setuptools, and I recommend
developers at their workstations take that approach instead.
For the purposes of a throwaway VM, there's not really a speed
difference for who is responsible for installing pip; us (needs
py310-pip) or Python (needs py310-expat).
Signed-off-by: John Snow <jsnow@redhat.com>
---
tests/vm/netbsd | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index 13eae109c0..c7e3f1e735 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -31,6 +31,7 @@ class NetBSDVM(basevm.BaseVM):
"pkgconf",
"xz",
"python310",
+ "py310-expat",
"ninja-build",
# gnu tools
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 06/10] mkvenv: generate console entry shims from inside the venv
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
` (4 preceding siblings ...)
2023-04-14 5:54 ` [RFC PATCH v2 05/10] tests/vm: add py310-expat to NetBSD John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 07/10] mkvenv: work around broken pip installations on Debian 10 John Snow
` (3 subsequent siblings)
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
This patch is meant to ensure that console entry point scripts will
always generate on Python 3.7 installations where we may not have access
to importlib.metadata. By running it from a separate process *inside*
the venv, we can be assured to have access to setuptools and by
extension pkg_resources as a fallback.
As a coincidence, it also gives us a pretty good place to do all kinds
of other "in-venv" setup that we might want to do later; for instance
meson and sphinx bootstrapping might be good candidates.
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/scripts/mkvenv.py | 71 ++++++++++++++++++++++++++++++++--------
1 file changed, 57 insertions(+), 14 deletions(-)
diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index a3284e9ef1..317697a953 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -62,22 +62,33 @@ def __init__(self, *args, **kwargs) -> None:
self.script_packages = kwargs.pop('script_packages', ())
super().__init__(*args, **kwargs)
- # The EnvBuilder class is cute and toggles this setting off
- # before post_setup, but we need it to decide if we want to
- # generate shims or not.
- self._system_site_packages = self.system_site_packages
+ # Make the context available post-creation:
+ self._context: Optional[SimpleNamespace] = None
+
+ def ensure_directories(self, env_dir: DirType) -> SimpleNamespace:
+ logger.debug("ensure_directories(env_dir=%s)", env_dir)
+ self._context = super().ensure_directories(env_dir)
+ return self._context
+
+ def create(self, env_dir: DirType) -> None:
+ logger.debug("create(env_dir=%s)", env_dir)
+ super().create(env_dir)
+ self.post_post_setup(self._context)
def post_setup(self, context: SimpleNamespace) -> None:
logger.debug("post_setup(...)")
-
- # Generate console_script entry points for system packages:
- if self._system_site_packages:
- generate_console_scripts(
- context.env_exe, context.bin_path, self.script_packages)
-
# print the python executable to stdout for configure.
print(context.env_exe)
+ def post_post_setup(self, context: SimpleNamespace) -> None:
+ # The final, final hook; enter the venv and run any
+ # last steps we want to occur *inside* the venv.
+ args = [context.env_exe, __file__, 'post_init',
+ '--binpath', context.bin_path]
+ if self.system_site_packages:
+ args += ['--gen', ",".join(self.script_packages)]
+ subprocess.run(args, check=True)
+
def need_ensurepip() -> bool:
"""
@@ -301,6 +312,10 @@ def generate_console_scripts(python_path: str, bin_path: str,
"""
Generate script shims for console_script entry points in @packages.
"""
+ logger.debug(
+ "generate_console_scripts(python_path=%s, bin_path=%s, packages=%s)",
+ python_path, bin_path, packages)
+
if not packages:
return
@@ -334,6 +349,16 @@ def _get_entry_points() -> Iterator[Dict[str, str]]:
logger.debug("wrote '%s'", script_path)
+def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
+ """
+ This is intended to be run *inside the venv* after it is created.
+ """
+ python_path = sys.executable
+ logger.debug("post_venv_setup(bin_path=%s, packages=%s)",
+ bin_path, packages)
+ generate_console_scripts(python_path, bin_path, packages)
+
+
def main() -> int:
"""CLI interface to make_qemu_venv. See module docstring."""
if os.environ.get('DEBUG') or os.environ.get('GITLAB_CI'):
@@ -366,12 +391,28 @@ def main() -> int:
help="Target directory to install virtual environment into.",
)
+ subparser = subparsers.add_parser(
+ 'post_init', help='post-venv initialization')
+ subparser.add_argument(
+ '--gen',
+ type=str,
+ action='append',
+ help="Regenerate console_scripts for given packages, if found.",
+ )
+ subparser.add_argument(
+ '--binpath',
+ type=str,
+ action='store',
+ help="Path where console script shims should be generated",
+ )
+
args = parser.parse_args()
+ script_packages = []
+ for element in args.gen or ():
+ script_packages.extend(element.split(","))
+
try:
if args.command == 'create':
- script_packages = []
- for element in args.gen or ():
- script_packages.extend(element.split(","))
make_venv(
args.target,
system_site_packages=True,
@@ -380,7 +421,9 @@ def main() -> int:
with_pip=None, # Autodetermine
script_packages=script_packages,
)
- logger.debug("mkvenv.py create - exiting")
+ if args.command == 'post_init':
+ post_venv_setup(args.binpath, script_packages)
+ logger.debug("mkvenv.py %s: exiting", args.command)
except Ouch as exc:
print("\n*** Ouch! ***\n", file=sys.stderr)
print(str(exc), "\n\n", file=sys.stderr)
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 07/10] mkvenv: work around broken pip installations on Debian 10
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
` (5 preceding siblings ...)
2023-04-14 5:54 ` [RFC PATCH v2 06/10] mkvenv: generate console entry shims from inside the venv John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 08/10] configure: create a python venv unconditionally John Snow
` (2 subsequent siblings)
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
This is a workaround intended for Debian 10, where the debian-patched
pip does not function correctly if accessed from within a virtual
environment.
We don't support Debian 10 any longer, but it's possible that this bug
might appear on other derivative platforms and this workaround may prove
useful.
RFC, a note from Paolo:
"BTW, another way to repair Debian 10's pip is to create a symbolic link
to sys.base_prefix + '/share/python-wheels' in sys.prefix +
'/share/python-wheels'. Since this is much faster, perhaps it can be
done unconditionally [...] ?"
I was slightly apprehensive about this as it felt "hackier", but it is
indeed a lot less code and much faster. It's probably low-risk. Should
we do that instead, or should we just scrap any fix at all under the
premise that Debian 10 support is dropped anyway?
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/scripts/mkvenv.py | 61 +++++++++++++++++++++++++++++-----------
1 file changed, 44 insertions(+), 17 deletions(-)
diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 317697a953..6e1a979355 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -103,27 +103,25 @@ def need_ensurepip() -> bool:
return True
-def check_ensurepip(with_pip: bool) -> None:
+def check_ensurepip(prefix: str = '', suggest_remedy: bool = False) -> None:
"""
Check that we have ensurepip.
Raise a fatal exception with a helpful hint if it isn't available.
"""
- if not with_pip:
- return
-
if not find_spec("ensurepip"):
msg = ("Python's ensurepip module is not found.\n"
"It's normally part of the Python standard library, "
"maybe your distribution packages it separately?\n"
- "Either install ensurepip, or alleviate the need for it in the "
- "first place by installing pip and setuptools for "
- f"'{sys.executable}'.\n"
-
- "(Hint: Debian puts ensurepip in its python3-venv package.)")
- raise Ouch(msg)
+ "(Debian puts ensurepip in its python3-venv package.)\n")
+ if suggest_remedy:
+ msg += (
+ "Either install ensurepip, or alleviate the need for it in the "
+ "first place by installing pip and setuptools for "
+ f"'{sys.executable}'.\n")
+ raise Ouch(prefix + msg)
# ensurepip uses pyexpat, which can also go missing on us:
if not find_spec("pyexpat"):
@@ -132,12 +130,13 @@ def check_ensurepip(with_pip: bool) -> None:
"It's normally part of the Python standard library, "
"maybe your distribution packages it separately?\n"
- "Either install pyexpat, or alleviate the need for it in the "
- "first place by installing pip and setuptools for "
- f"'{sys.executable}'.\n\n"
-
- "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)")
- raise Ouch(msg)
+ "(NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)\n")
+ if suggest_remedy:
+ msg += (
+ "Either install pyexpat, or alleviate the need for it in the "
+ "first place by installing pip and setuptools for "
+ f"'{sys.executable}'.\n")
+ raise Ouch(prefix + msg)
def make_venv( # pylint: disable=too-many-arguments
@@ -169,7 +168,8 @@ def make_venv( # pylint: disable=too-many-arguments
with_pip = True if not system_site_packages else need_ensurepip()
logger.debug("with_pip unset, choosing %s", with_pip)
- check_ensurepip(with_pip)
+ if with_pip:
+ check_ensurepip(suggest_remedy=True)
if symlinks is None:
# Default behavior of standard venv CLI
@@ -349,6 +349,31 @@ def _get_entry_points() -> Iterator[Dict[str, str]]:
logger.debug("wrote '%s'", script_path)
+def checkpip():
+ """
+ Debian10 has a pip that's broken when used inside of a virtual environment.
+
+ We try to detect and correct that case here.
+ """
+ try:
+ import pip._internal
+ logger.debug("pip appears to be working correctly.")
+ return
+ except ModuleNotFoundError as exc:
+ if exc.name == 'pip._internal':
+ # Uh, fair enough. They did say "internal".
+ # Let's just assume it's fine.
+ return
+ logger.warning("pip appears to be malfunctioning: %s", str(exc))
+
+ check_ensurepip("pip appears to be non-functional, and ")
+
+ logging.debug("Attempting to repair pip ...")
+ subprocess.run((sys.executable, '-m', 'ensurepip'),
+ stdout=subprocess.DEVNULL, check=True)
+ logging.debug("Pip is now (hopefully) repaired!")
+
+
def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
"""
This is intended to be run *inside the venv* after it is created.
@@ -358,6 +383,8 @@ def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
bin_path, packages)
generate_console_scripts(python_path, bin_path, packages)
+ # Test for a broken pip (Debian 10 or derivative?) and fix it if needed
+ checkpip()
def main() -> int:
"""CLI interface to make_qemu_venv. See module docstring."""
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 08/10] configure: create a python venv unconditionally
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
` (6 preceding siblings ...)
2023-04-14 5:54 ` [RFC PATCH v2 07/10] mkvenv: work around broken pip installations on Debian 10 John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 09/10] configure: remove --meson=; install meson to the pyvenv John Snow
2023-04-14 5:54 ` [RFC PATCH v2 10/10] tests: Use configure-provided pyvenv for tests John Snow
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
This patch changes the configure script so that it always creates a
python virtual environment unconditionally. Meson bootstrapping is
temporarily altered to force the use of meson from git or vendored
source.
(A patch later in this series restores the use of a
distribution-provided Meson.)
Signed-off-by: John Snow <jsnow@redhat.com>
---
configure | 34 +++++++++++++++++++++++++++++-----
1 file changed, 29 insertions(+), 5 deletions(-)
diff --git a/configure b/configure
index 800b5850f4..03278fd891 100755
--- a/configure
+++ b/configure
@@ -608,7 +608,6 @@ check_py_version() {
python=
first_python=
if test -z "${PYTHON}"; then
- explicit_python=no
# A bare 'python' is traditionally python 2.x, but some distros
# have it as python 3.x, so check in both places.
for binary in python3 python python3.11 python3.10 python3.9 python3.8 python3.7 python3.6; do
@@ -627,7 +626,6 @@ else
# Same as above, but only check the environment variable.
has "${PYTHON}" || error_exit "The PYTHON environment variable does not point to an executable"
python=$(command -v "$PYTHON")
- explicit_python=yes
if check_py_version "$python"; then
# This one is good.
first_python=
@@ -719,7 +717,7 @@ for opt do
;;
--install=*)
;;
- --python=*) python="$optarg" ; explicit_python=yes
+ --python=*) python="$optarg"
;;
--skip-meson) skip_meson=yes
;;
@@ -1079,8 +1077,34 @@ if ! check_py_version "$python"; then
"Use --python=/path/to/python to specify a supported Python."
fi
-# Resolve PATH + suppress writing compiled files
-python="$(command -v "$python") -B"
+# Resolve PATH
+python="$(command -v "$python")"
+explicit_python=yes
+
+# Create a Python virtual environment using our configured python.
+# The stdout of this script will be the location of a symlink that
+# points to the configured Python.
+# Entry point scripts for pip, meson, and sphinx are generated if those
+# packages are present.
+
+# Defaults assumed for now:
+# - venv is cleared if it exists already;
+# - venv is allowed to use system packages;
+# - all setup is performed **offline**;
+# - No packages are installed by default;
+# - pip is not installed into the venv when possible,
+# but ensurepip is called as a fallback when necessary.
+
+echo "python determined to be '$python'"
+echo "python version: $($python --version)"
+
+python="$($python -B "${source_path}/python/scripts/mkvenv.py" create --gen pip,meson,sphinx pyvenv)"
+if test "$?" -ne 0 ; then
+ error_exit "python venv creation failed"
+fi
+
+# Suppress writing compiled files
+python="$python -B"
has_meson() {
local python_dir=$(dirname "$python")
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 09/10] configure: remove --meson=; install meson to the pyvenv
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
` (7 preceding siblings ...)
2023-04-14 5:54 ` [RFC PATCH v2 08/10] configure: create a python venv unconditionally John Snow
@ 2023-04-14 5:54 ` John Snow
2023-04-14 5:54 ` [RFC PATCH v2 10/10] tests: Use configure-provided pyvenv for tests John Snow
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
This patch changes how we detect and install meson.
The previous patch creates a lightweight Python virtual environment
unconditionally using the user's configured $python that inherits system
packages. If Meson is installed there and meets our minimum version
requirements, we will use that Meson.
In the event that Meson is installed but *not for the chosen Python
interpreter*, not found, or of insufficient version, we will attempt to
install Meson from source into the newly created Python virtual
environment.
At present, the source is obtained in the same manner as it has been;
preferring git submodules first and relying on vendored source as a
backup.
This patch (as of now) does *not* connect to PyPI and will work offline
for all supported build platforms.
As a result of this patch, the Python interpreter we use for both our
own build scripts *and* Meson extensions are always known to be the
exact same Python. As a further benefit, there will also be a symlink
available in the build directory that points to the correct, configured
python and can be used by e.g. manual tests to invoke the correct,
configured Python unambiguously.
Notes:
- The meson git submodule can be removed in favor of just loading meson
from PyPI; but we probably want a configure flag to toggle
online/offline behavior. (What do we want the default to be? Online is
my guess.)
- Installing meson from the source tree for vendored cases (tarball
releases) can be replaced by vendoring a .whl file instead. This will
remove some of the `--no-use-pep517` hackiness and alleviates the
requirement that users install the python3 'wheel' dependency.
- Most of this logic can be moved into mkvenv.py, *especially* if the
meson git submodule is removed.
Signed-off-by: John Snow <jsnow@redhat.com>
---
configure | 104 +++++++++++++++-------------
.gitlab-ci.d/buildtest-template.yml | 4 +-
2 files changed, 59 insertions(+), 49 deletions(-)
diff --git a/configure b/configure
index 03278fd891..c8f73d4ff7 100755
--- a/configure
+++ b/configure
@@ -721,8 +721,6 @@ for opt do
;;
--skip-meson) skip_meson=yes
;;
- --meson=*) meson="$optarg"
- ;;
--ninja=*) ninja="$optarg"
;;
--smbd=*) smbd="$optarg"
@@ -1006,7 +1004,6 @@ Advanced options (experts only):
--cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases
--make=MAKE use specified make [$make]
--python=PYTHON use specified python [$python]
- --meson=MESON use specified meson [$meson]
--ninja=NINJA use specified ninja [$ninja]
--smbd=SMBD use specified smbd [$smbd]
--with-git=GIT use specified git [$git]
@@ -1079,7 +1076,8 @@ fi
# Resolve PATH
python="$(command -v "$python")"
-explicit_python=yes
+# This variable is intended to be used only for error messages:
+target_python=$python
# Create a Python virtual environment using our configured python.
# The stdout of this script will be the location of a symlink that
@@ -1106,58 +1104,70 @@ fi
# Suppress writing compiled files
python="$python -B"
-has_meson() {
- local python_dir=$(dirname "$python")
- # PEP405: pyvenv.cfg is either adjacent to the Python executable
- # or one directory above
- if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then
- # Ensure that Meson and Python come from the same virtual environment
- test -x "$python_dir/meson" &&
- test "$(command -v meson)" -ef "$python_dir/meson"
- else
- has meson
- fi
+pip_install() {
+ $python -m pip install -v \
+ --disable-pip-version-check \
+ --no-index \
+ "$@"
}
-if test -z "$meson"; then
- if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then
- meson=meson
- elif test "$git_submodules_action" != 'ignore' ; then
- meson=git
- elif test -e "${source_path}/meson/meson.py" ; then
- meson=internal
+# OK, let's have some fun!
+
+# This install command is intended to either fail or be a NOP;
+# because we're offline, it's just a convenient version check.
+if ! pip_install 'meson>=0.61.5'; then
+ # Either we don't have Meson, or our Meson is too old.
+ # (Future revisions of this patchset can be less chatty.)
+ if test -e pyvenv/bin/meson; then
+ echo "Meson in pyvenv is too old: $(pyvenv/bin/meson --version)"
+ elif has meson ; then
+ echo "Meson was found installed on your system," \
+ "but not for the configured Python interpreter ($target_python)."
+ echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)"
else
- if test "$explicit_python" = yes; then
- error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found."
- else
- error_exit "Meson not found. Use --meson=/path/to/meson"
+ echo "Meson was not found."
+ fi
+
+ # OK, but can we fix it, though? :~)
+ if test "$git_submodules_action" != 'ignore' ; then
+ git_submodules="${git_submodules} meson"
+ echo "Attempting to install meson from git submodule ..."
+ # Stolen from later in the configure file.
+ # Is it a problem if we front-load this now and run it again later?
+ if ! (GIT="$git" "$source_path/scripts/git-submodule.sh" "$git_submodules_action" "$git_submodules"); then
+ exit 1
fi
+ elif test -e "${source_path}/meson/setup.cfg" ; then
+ echo "Attempting to install meson from vendored source ..."
+ else
+ # In the future, we could use PyPI as a source if the user allows it.
+ # For now, you're outta luck!
+ error_exit "A suitable version of Meson was not found."
fi
-else
- # Meson uses its own Python interpreter to invoke other Python scripts,
- # but the user wants to use the one they specified with --python.
+
+ # If we're here, we have the meson source and we can attempt to
+ # install it into our venv.
+
+ # We want to install meson with --no-use-pep517 if possible,
+ # because it avoids needing a 'wheel' dependency. Old versions
+ # of pip do this by default, so test for the behavior.
#
- # We do not want to override the distro Python interpreter (and sometimes
- # cannot: for example in Homebrew /usr/bin/meson is a bash script), so
- # just require --meson=git|internal together with --python.
- if test "$explicit_python" = yes; then
- case "$meson" in
- git | internal) ;;
- *) error_exit "--python requires using QEMU's embedded Meson distribution." ;;
- esac
+ # --no-build-isolation was added to pip 10.0.
+ # --no-use-pep517 was added ... sometime after 18.1?
+ pip_flags='--no-build-isolation'
+ if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>&1 ; then
+ pip_flags="${pip_flags} --no-use-pep517"
+ fi
+ if ! pip_install $pip_flags "${source_path}/meson" ; then
+ exit 1
fi
fi
-if test "$meson" = git; then
- git_submodules="${git_submodules} meson"
-fi
-
-case "$meson" in
- git | internal)
- meson="$python ${source_path}/meson/meson.py"
- ;;
- *) meson=$(command -v "$meson") ;;
-esac
+# At this point, we expect Meson to be installed and available.
+# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
+# We ignore PATH completely here: we want to use the venv's Meson
+# *exclusively*.
+meson="$(cd pyvenv/bin; pwd)/meson"
# Probe for ninja
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index a6cfe9be97..7edb50b760 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -12,12 +12,12 @@
- mkdir build
- cd build
- ../configure --enable-werror --disable-docs --enable-fdt=system
- ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"}
+ ${TARGETS:+--target-list="$TARGETS"}
$CONFIGURE_ARGS ||
{ cat config.log meson-logs/meson-log.txt && exit 1; }
- if test -n "$LD_JOBS";
then
- ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
+ pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
fi || exit 1;
- make -j"$JOBS"
- if test -n "$MAKE_CHECK_ARGS";
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [RFC PATCH v2 10/10] tests: Use configure-provided pyvenv for tests
2023-04-14 5:54 [RFC PATCH v2 00/10] configure: create a python venv and install meson John Snow
` (8 preceding siblings ...)
2023-04-14 5:54 ` [RFC PATCH v2 09/10] configure: remove --meson=; install meson to the pyvenv John Snow
@ 2023-04-14 5:54 ` John Snow
9 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2023-04-14 5:54 UTC (permalink / raw)
To: qemu-devel
Cc: Cleber Rosa, Ani Sinha, Warner Losh, Alex Bennée,
Reinoud Zandijk, Thomas Huth, Paolo Bonzini, John Snow,
Ryo ONODERA, Kyle Evans, Beraldo Leal, Michael S. Tsirkin,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use instead the
configure-provided 'pyvenv' and install optional packages into that.
Note: At the time of writing, avocado tests require avocado-framework <
90 whereas the qemu.qmp self-tests rely on avocado-framework >= 90. This
collision is avoided for now because the qemu.git/python/qemu/ code does
not need avocado at *runtime*; it does not install avocado-framework as
a necessary dependency and is skipped in this circumstance.
Nevertheless, we do want to address that discrepancy in the future so
that it will be possible to re-use the same venv for
qemu.git/python/qemu self-tests to introduce them to make check as "make
check-python".
Signed-off-by: John Snow <jsnow@redhat.com>
---
docs/devel/acpi-bits.rst | 6 +++---
docs/devel/testing.rst | 14 +++++++-------
.gitlab-ci.d/buildtest.yml | 6 +++---
scripts/ci/org.centos/stream/8/x86_64/test-avocado | 4 ++--
scripts/device-crash-test | 2 +-
tests/Makefile.include | 10 +++++-----
tests/requirements.txt | 7 +++++--
7 files changed, 26 insertions(+), 23 deletions(-)
diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst
index 9eb4b9e666..0c40359109 100644
--- a/docs/devel/acpi-bits.rst
+++ b/docs/devel/acpi-bits.rst
@@ -61,19 +61,19 @@ Under ``tests/avocado/`` as the root we have:
::
$ make check-venv (needed only the first time to create the venv)
- $ ./tests/venv/bin/avocado run -t acpi tests/avocado
+ $ ./pyvenv/bin/avocado run -t acpi tests/avocado
The above will run all acpi avocado tests including this one.
In order to run the individual tests, perform the following:
::
- $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap -
+ $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap -
The above will produce output in tap format. You can omit "--tap -" in the
end and it will produce output like the following:
::
- $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
+ $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py
Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef
JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4071e72710..50664d9eb9 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -882,9 +882,9 @@ You can run the avocado tests simply by executing:
make check-avocado
-This involves the automatic creation of Python virtual environment
-within the build tree (at ``tests/venv``) which will have all the
-right dependencies, and will save tests results also within the
+This involves the automatic installation, from PyPI, of all the
+necessary avocado-framework dependencies into the QEMU venv within the
+build tree (at ``./pyvenv``). Test results are also saved within the
build tree (at ``tests/results``).
Note: the build environment must be using a Python 3 stack, and have
@@ -941,7 +941,7 @@ may be invoked by running:
.. code::
- tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
+ pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
Note that if ``make check-avocado`` was not executed before, it is
possible to create the Python virtual environment with the dependencies
@@ -956,20 +956,20 @@ a test file. To run tests from a single file within the build tree, use:
.. code::
- tests/venv/bin/avocado run tests/avocado/$TESTFILE
+ pyvenv/bin/avocado run tests/avocado/$TESTFILE
To run a single test within a test file, use:
.. code::
- tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
+ pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
Valid test names are visible in the output from any previous execution
of Avocado or ``make check-avocado``, and can also be queried using:
.. code::
- tests/venv/bin/avocado list tests/avocado
+ pyvenv/bin/avocado list tests/avocado
Manual Installation
~~~~~~~~~~~~~~~~~~~
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index ba6f551752..53de9f23c4 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -103,7 +103,7 @@ crash-test-debian:
script:
- cd build
- make check-venv
- - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
+ - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
build-system-fedora:
extends:
@@ -146,8 +146,8 @@ crash-test-fedora:
script:
- cd build
- make check-venv
- - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
- - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
+ - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
+ - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
build-system-centos:
extends:
diff --git a/scripts/ci/org.centos/stream/8/x86_64/test-avocado b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
index d2c0e5fb4c..7bb5b317b6 100755
--- a/scripts/ci/org.centos/stream/8/x86_64/test-avocado
+++ b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
@@ -4,7 +4,7 @@
# KVM and x86_64, or tests that are generic enough to be valid for all
# targets. Such a test list can be generated with:
#
-# ./tests/venv/bin/avocado list --filter-by-tags-include-empty \
+# ./pyvenv/bin/avocado list --filter-by-tags-include-empty \
# --filter-by-tags-include-empty-key -t accel:kvm,arch:x86_64 \
# tests/avocado/
#
@@ -22,7 +22,7 @@
# - tests/avocado/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types
#
make get-vm-images
-./tests/venv/bin/avocado run \
+./pyvenv/bin/avocado run \
--job-results-dir=tests/results/ \
tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm \
tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm \
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 73bcb98693..ef6ac262fa 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -43,7 +43,7 @@ except ModuleNotFoundError as exc:
print(f"Module '{exc.name}' not found.")
print(" Try 'make check-venv' from your build directory,")
print(" and then one way to run this script is like so:")
- print(f' > $builddir/tests/venv/bin/python3 "{path}"')
+ print(f' > $builddir/pyvenv/bin/python3 "{path}"')
sys.exit(1)
logger = logging.getLogger('device-crash-test')
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 9422ddaece..3f39c607cc 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -89,7 +89,8 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES)
# Build up our target list from the filtered list of ninja targets
TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets)))
-TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_DIR=$(BUILD_DIR)/pyvenv
+TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group
TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3
@@ -111,8 +112,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
$(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
"VENVPIP","$1")
-$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
- $(call quiet-command, $(PYTHON) -m venv $@, VENV, $@)
+$(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ)
$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
$(call quiet-command, touch $@)
@@ -121,7 +121,7 @@ $(TESTS_RESULTS_DIR):
$(call quiet-command, mkdir -p $@, \
MKDIR, $@)
-check-venv: $(TESTS_VENV_DIR)
+check-venv: $(TESTS_VENV_TOKEN)
FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS)))
FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
@@ -163,7 +163,7 @@ check:
check-build: run-ninja
check-clean:
- rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
+ rm -rf $(TESTS_RESULTS_DIR)
clean: check-clean clean-tcg
distclean: distclean-tcg
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 0ba561b6bd..07e713ef5a 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,6 +1,9 @@
# Add Python module requirements, one per line, to be installed
-# in the tests/venv Python virtual environment. For more info,
+# in the qemu build_dir/pyvenv Python virtual environment. For more info,
# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-# Note that qemu.git/python/ is always implicitly installed.
+#
+# Note that qemu.git/python/ is implicitly installed to this venv when
+# 'make check-venv' is run, and will persist until configure is run
+# again.
avocado-framework==88.1
pycdlib==1.11.0
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread