[PATCH] docs: document what configure does with virtual environments

[PATCH] docs: document what configure does with virtual environments

[Paolo Bonzini]

Given the recent confusion around how QEMU detects the system
Meson installation, and/or decides to install its own, it is
time to fill in the "Python virtual environments and the QEMU
build system" section of the documentation.

As a curiosity, a first and partial draft of the text was generated
by an LLM[1].  It required quite a bit of editing and probably did not
save much time, but some expressions do remain in the finished text.

[1] https://chat.openai.com/share/42c1500d-71c1-480b-bab9-7ccc2c155365

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 docs/devel/build-system.rst | 54 ++++++++++++++++++++++++++++++++++---
 pythondeps.toml             |  3 ++-
 2 files changed, 53 insertions(+), 4 deletions(-)

diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
index 21f78da7d1d..4def2e55e73 100644
--- a/docs/devel/build-system.rst
+++ b/docs/devel/build-system.rst
@@ -122,10 +122,49 @@ functioning.  These are performed using a few more helper functions:
    indicated by $TMPC.
 
 
-Python virtual environments and the QEMU build system
------------------------------------------------------
+Python virtual environments and the build process
+-------------------------------------------------
+
+An important part of configure is to create a Python virtual environment
+(venv) during the configuration phase, using the Python interpreter that
+``configure`` identified, or that was requested via the ``--python``
+command line option and the ``$PYTHON`` variable from the environment.
+The venv resides in the ``pyvenv`` directory in the build tree,
+and provides consistency in how the build process runs Python code.
+In particular it avoids a potential mismatch, where Meson and Sphinx
+binaries on the PATH might operate in a different Python environment
+than the one chosen by the user during the build process.
+
+At this stage, ``configure`` also queries the chosen Python interpreter
+about QEMU's build dependencies.  ``configure`` does  *not*
+pick the ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH;
+likewise, there are no options such as ``--meson`` or ``sphinx-build``.
+If QEMU does not find a dependency, check that it was installed in the
+right ``site-packages`` directory or with the right ``pip`` program.
+
+If the package is available as a system package for the chosen
+interpreter, ``configure`` prepares a small script that invokes it
+from the venv itself.  If not, ``configure`` can also optionally
+install dependencies in the virtual environment with ``pip``.
+Downloading is triggered only when a ``configure`` option (currently,
+only ``--enable-docs``) is explicitly enabled but the dependencies are
+not present, and can also be disabled with ``--disable-download``.[#pip]_
+
+.. [#pip_] Avocado can also be installed with ``pip`` in the virtual
+           environment when running ``make check-avocado``.  In this
+           case, it is not currently possible to block the downloading.
+
+The required versions of the packages are stored in a configuration file
+``pythondeps.toml``.  The format is custom to QEMU, but it is documented
+at the top of the file itself and it should be easy to understand.  The
+requirements should make it possible to use the version that is packaged
+that is provided by supported distros.
+
+When dependencies are downloaded, instead, ``configure`` uses a "known
+good" version that is also listed in ``pythondeps.toml``.  In this
+scenario, ``pythondeps.toml`` behaves like the "lock file" used by
+``cargo``, ``poetry`` or other dependency management systems.
 
-TBD
 
 Stage 2: Meson
 ==============
@@ -376,6 +415,15 @@ This is needed to obey the --python= option passed to the configure
 script, which may point to something other than the first python3
 binary on the path.
 
+By the time Meson runs, Python dependencies are available in the virtual
+environment and should be invoked though the scripts that ``configure``
+places under ``pyvenv``.  One way to do so is as follows, using Meson's
+``find_program`` function::
+
+  sphinx_build = find_program(
+       fs.parent(python.full_path()) / 'sphinx-build',
+       required: get_option('docs'))
+
 
 Stage 3: Make
 =============
diff --git a/pythondeps.toml b/pythondeps.toml
index 0a35ebcf9f0..4beadfd96f5 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -10,7 +10,8 @@
 # - accepted: accepted versions when using a system package
 # - installed: fixed version to install in the virtual environment
 #              if a system package is not found; if not specified,
-#              the minimum and maximum
+#              defaults to the same as "accepted" or, if also missing,
+#              to the newest version available on PyPI.
 # - canary: if specified, use this program name to present more
 #           precise error diagnostics to the user.  For example,
 #           'sphinx-build' can be used as a bellwether for the
-- 
2.41.0

Re: [PATCH] docs: document what configure does with virtual environments

[John Snow]

Whoops, didn't mean to reply off-list.

On Thu, Nov 9, 2023 at 4:34 PM John Snow <jsnow@redhat.com> wrote:
>
> On Thu, Nov 9, 2023 at 10:59 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > Given the recent confusion around how QEMU detects the system
> > Meson installation, and/or decides to install its own, it is
> > time to fill in the "Python virtual environments and the QEMU
> > build system" section of the documentation.
> >
> > As a curiosity, a first and partial draft of the text was generated
> > by an LLM[1].  It required quite a bit of editing and probably did not
> > save much time, but some expressions do remain in the finished text.
>
> boo :p
>
> >
> > [1] https://chat.openai.com/share/42c1500d-71c1-480b-bab9-7ccc2c155365
> >
> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> > ---
> >  docs/devel/build-system.rst | 54 ++++++++++++++++++++++++++++++++++---
> >  pythondeps.toml             |  3 ++-
> >  2 files changed, 53 insertions(+), 4 deletions(-)
> >
> > diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
> > index 21f78da7d1d..4def2e55e73 100644
> > --- a/docs/devel/build-system.rst
> > +++ b/docs/devel/build-system.rst
> > @@ -122,10 +122,49 @@ functioning.  These are performed using a few more helper functions:
> >     indicated by $TMPC.
> >
> >
> > -Python virtual environments and the QEMU build system
> > ------------------------------------------------------
> > +Python virtual environments and the build process
> > +-------------------------------------------------
> > +
> > +An important part of configure is to create a Python virtual environment
> > +(venv) during the configuration phase, using the Python interpreter that
> > +``configure`` identified, or that was requested via the ``--python``
> > +command line option and the ``$PYTHON`` variable from the environment.
>
> Which takes precedence, in what order? "and" makes it sound as if both
> --python and $PYTHON are considered, but I don't think that's actually
> true.
>
> > +The venv resides in the ``pyvenv`` directory in the build tree,
> > +and provides consistency in how the build process runs Python code.
> > +In particular it avoids a potential mismatch, where Meson and Sphinx
>
> I think you can drop the comma. This is so pedantic that if you left
> it in to spite me, I'd not blame you. :)
>
> > +binaries on the PATH might operate in a different Python environment
> > +than the one chosen by the user during the build process.
> > +
> > +At this stage, ``configure`` also queries the chosen Python interpreter
> > +about QEMU's build dependencies.  ``configure`` does  *not*
> > +pick the ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH;
> > +likewise, there are no options such as ``--meson`` or ``sphinx-build``.
>
> should we say ``--sphinx-build``?
>
> I also might say "does not ^necessarily pick the ..." because they
> could be the same, it just isn't the criteria it uses to choose them.
>
> > +If QEMU does not find a dependency, check that it was installed in the
> > +right ``site-packages`` directory or with the right ``pip`` program.
>
> I don't actually know what this means. >_>
>
> > +
> > +If the package is available as a system package for the chosen
>
> technically, I think if the package is available at all for the chosen
> interpreter. We supported nested venvs, so it's more than just system
> packages.
>
> > +interpreter, ``configure`` prepares a small script that invokes it
> > +from the venv itself.  If not, ``configure`` can also optionally
> > +install dependencies in the virtual environment with ``pip``.
>
> What may not be clear here is that this can refer to installing from
> bundled sources in the source tree. IIRC, it's used in preference to
> internet sources, isn't it? (Unless that changed.)
>
> > +Downloading is triggered only when a ``configure`` option (currently,
> > +only ``--enable-docs``) is explicitly enabled but the dependencies are
> > +not present, and can also be disabled with ``--disable-download``.[#pip]_
> > +
> > +.. [#pip_] Avocado can also be installed with ``pip`` in the virtual
> > +           environment when running ``make check-avocado``.  In this
> > +           case, it is not currently possible to block the downloading.
> > +
>
> Similarly to my above comment: if downloads are disabled, we do have
> some vendored packages we'll attempt to use.
>
> > +The required versions of the packages are stored in a configuration file
> > +``pythondeps.toml``.  The format is custom to QEMU, but it is documented
> > +at the top of the file itself and it should be easy to understand.  The
> > +requirements should make it possible to use the version that is packaged
> > +that is provided by supported distros.
> > +
> > +When dependencies are downloaded, instead, ``configure`` uses a "known
> > +good" version that is also listed in ``pythondeps.toml``.  In this
> > +scenario, ``pythondeps.toml`` behaves like the "lock file" used by
> > +``cargo``, ``poetry`` or other dependency management systems.
> >
> > -TBD
> >
> >  Stage 2: Meson
> >  ==============
> > @@ -376,6 +415,15 @@ This is needed to obey the --python= option passed to the configure
> >  script, which may point to something other than the first python3
> >  binary on the path.
> >
> > +By the time Meson runs, Python dependencies are available in the virtual
> > +environment and should be invoked though the scripts that ``configure``
>
> through
>
> > +places under ``pyvenv``.  One way to do so is as follows, using Meson's
> > +``find_program`` function::
> > +
> > +  sphinx_build = find_program(
> > +       fs.parent(python.full_path()) / 'sphinx-build',
> > +       required: get_option('docs'))
> > +
> >
> >  Stage 3: Make
> >  =============
> > diff --git a/pythondeps.toml b/pythondeps.toml
> > index 0a35ebcf9f0..4beadfd96f5 100644
> > --- a/pythondeps.toml
> > +++ b/pythondeps.toml
> > @@ -10,7 +10,8 @@
> >  # - accepted: accepted versions when using a system package
> >  # - installed: fixed version to install in the virtual environment
> >  #              if a system package is not found; if not specified,
> > -#              the minimum and maximum
> > +#              defaults to the same as "accepted" or, if also missing,
> > +#              to the newest version available on PyPI.
> >  # - canary: if specified, use this program name to present more
> >  #           precise error diagnostics to the user.  For example,
> >  #           'sphinx-build' can be used as a bellwether for the
> > --
> > 2.41.0
> >

Re: [PATCH] docs: document what configure does with virtual environments

[Paolo Bonzini]

On Thu, Nov 9, 2023 at 10:35 PM John Snow <jsnow@redhat.com> wrote:
> > > +The venv resides in the ``pyvenv`` directory in the build tree,
> > > +and provides consistency in how the build process runs Python code.
> > > +In particular it avoids a potential mismatch, where Meson and Sphinx
> >
> > I think you can drop the comma. This is so pedantic that if you left
> > it in to spite me, I'd not blame you. :)

I'll keep it then. :)

> > should we say ``--sphinx-build``?

Yes, typo.

> > I also might say "does not ^necessarily pick the ..." because they
> > could be the same, it just isn't the criteria it uses to choose them.

I'll replace "pick" with "look for".

> > > +If QEMU does not find a dependency, check that it was installed in the
> > > +right ``site-packages`` directory or with the right ``pip`` program.
> >
> > I don't actually know what this means. >_>

It's meant to explain what happened with homebrew. Rephrased in v2:

This avoids a potential mismatch, where Meson and Sphinx binaries on the
PATH might operate in a different Python environment than the one chosen
by the user during the build process.  On the other hand, it introduces
a potential source of confusion where the user installs a dependency but
``configure`` is not able to find it.  When this happens, the dependency
was installed in the ``site-packages`` directory of another interpreter,
or with the wrong ``pip`` program.

Paolo