[PATCH v3 04/15] python/mkvenv: add mechanism to install local package(s)

[John Snow <jsnow@redhat.com>]

Currently, we "implicitly" install the local 'qemu' python package for
'make check-venv' with some logic inside tests/Makefile.include. I would
like to make this installation explicit in pythondeps.toml instead.

The version constraint specification that Python specifies does not
support relative paths, so it is difficult (or impossible?) to specify a
path within the source tree, and we will need a workaround to do so.

By specifying a package name that starts with $SRCROOT, you can now
specify a file path to a local package for installation. This is done to
allow us to install the python packages hosted inside of the tree while
also processing dependencies; i.e. so that our "qemu" package can
specify that it needs "qemu.qmp", which soon will not be included in
qemu.git.

This also has the benefit of being able to specify in a declarative
configuration file that our pyvenv environment *will* have our local
python packages installed and available without any PYTHONPATH hacks,
which should simplify iotests, device-crash-test and functional tests
without needing to manage local inclusion paths in environment
variables.

On the downsides, installing packages through mkvenv/ensuregroup means
that there are extra steps we need to take in order to install a local
package *offline*; namely we must disable build isolation (so we have
access to setuptools) and we must also include python3-wheel in QEMU's
build dependencies in order for "make check" to run successfully when in
an offline, isolated environment.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 38 ++++++++++++++++++++++++++++++++------
 1 file changed, 32 insertions(+), 6 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index a22e3ee3394..608b8faa9b2 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -662,6 +662,7 @@ def pip_install(
     args: Sequence[str],
     online: bool = False,
     wheels_dir: Optional[Union[str, Path]] = None,
+    env: Optional[Dict[str, str]] = None,
 ) -> None:
     """
     Use pip to install a package or package(s) as specified in @args.
@@ -687,6 +688,7 @@ def pip_install(
     full_args += list(args)
     subprocess.run(
         full_args,
+        env=env,
         check=True,
     )
 
@@ -733,9 +735,15 @@ def _do_ensure(
     :param wheels_dir: If specified, search this path for packages.
     """
     absent = []
+    local_packages = []
     present = []
     canary = None
     for name, info in group.items():
+        if name.startswith("$SRCROOT/"):
+            srcroot = Path(__file__).parents[2]
+            pkgpath = name.replace("$SRCROOT/", f"file://{srcroot}/")
+            local_packages.append(pkgpath)
+            continue
         constraint = _make_version_constraint(info, False)
         matcher = Matcher(name + constraint)
         print(f"mkvenv: checking for {matcher}", file=sys.stderr)
@@ -770,15 +778,33 @@ def _do_ensure(
             print(f"mkvenv: installing {', '.join(absent)}", file=sys.stderr)
             try:
                 pip_install(args=absent, online=online, wheels_dir=wheels_dir)
-                return None
+                absent = []
             except subprocess.CalledProcessError:
                 pass
 
-        return diagnose(
-            absent[0],
-            online,
-            wheels_dir,
-            canary,
+        if absent:
+            return diagnose(
+                absent[0],
+                online,
+                wheels_dir,
+                canary,
+            )
+
+    # Handle local packages separately and last so we can use different
+    # installation arguments (-e), and so that any dependencies that may
+    # be covered above will be handled according to the depfile
+    # specifications.
+    if local_packages:
+        print(f"mkvenv: installing {', '.join(local_packages)}",
+              file=sys.stderr)
+        env = dict(os.environ)
+        env['PIP_CONFIG_SETTINGS'] = "editable_mode=compat"
+        pip_install(
+            args=["--no-build-isolation",
+                  "-e"] + local_packages,
+            online=online,
+            wheels_dir=wheels_dir,
+            env=env,
         )
 
     return None
-- 
2.51.1