[PATCH v3 0/6] Python: Drop support for Python 3.6

[PATCH v3 0/6] Python: Drop support for Python 3.6

[John Snow]

CI: https://gitlab.com/jsnow/qemu/-/pipelines/783612696
    [Updated for v3, still all green.]
GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37

Hi, discussion about this series is ongoing. This series (v3) is not
meant to address all of that discussion, but rather is an updated
baseline for what we are capable of right now, today, without much
additional engineering. It's meant to serve as a reference for further
discussion.

To my knowledge, the inconveniences caused by this patchset as currently
written are:

(1) Users of CentOS 8 and OpenSUSE 15.4 would need to install an
    additional python package that will exist side-by-side with their
    base platform's Python 3.6 package.

    "zypper install python39" or "dnf install python38" is enough;
    configure will do the rest of the work.

    It's my understanding that this is largely a non-issue.

(2) Due to our Sphinx plugin that imports QAPI code from the tree,
    distro-provided versions of Sphinx that are installed and tied to
    Python 3.6 will no longer be suitable. Users may forego building
    docs or install a suitable sphinx using "pip".

    It's my understanding that this one is "kind of a bummer".

I feel that the inconvenience caused by (1) is minimized as is possible;
the inconvenience caused by (2) is slightly worse and I concede the
workaround has some complexities that I would otherwise seek to avoid.

As far as I am aware, the way forward is to work with Paolo to implement
a proper venv solution for the build tree that will help mitigate the
fallout from (2) by automating the use of a pip-provided Sphinx in the
cases where the distro-provided version is insufficient.

OK, seeya later!
--js

John Snow (6):
  configure: Look for auxiliary Python installations
  configure: Add courtesy hint to Python version failure message
  DO-NOT-MERGE: testing: Add Python >= 3.7 to Centos, OpenSuSE
  DO-NOT-MERGE: testing: add pip-installed sphinx-build to CentOS 8
  meson: prefer 'sphinx-build' to 'sphinx-build-3'
  Python: Drop support for Python 3.6

 docs/conf.py                                  |  4 +-
 docs/meson.build                              |  2 +-
 configure                                     | 41 ++++++++++++++-----
 python/Makefile                               | 10 ++---
 python/setup.cfg                              |  7 ++--
 python/tests/minreqs.txt                      |  2 +-
 scripts/qapi/mypy.ini                         |  2 +-
 tests/docker/dockerfiles/centos8.docker       |  5 +++
 tests/docker/dockerfiles/opensuse-leap.docker |  1 +
 9 files changed, 50 insertions(+), 24 deletions(-)

-- 
2.39.0

[PATCH v3 1/6] configure: Look for auxiliary Python installations

[John Snow]

At the moment, we look for just "python3" and "python", which is good
enough almost all of the time. But ... if you are on a platform that
uses an older Python by default and only offers a newer Python as an
option, you'll have to specify --python=/usr/bin/foo every time.

As a courtesy, we can make a cursory attempt to locate a suitable Python
binary ourselves, looking for the remaining well-known binaries. This
also has the added benefit of making configure "just work" more often
on various BSD distributions that do not have the concept of a
"platform default python".

This configure loop will prefer, in order:

1. Whatever is specified in $PYTHON
2. python3
3. python (Which is usually 2.x, but might be 3.x on some platforms.)
4. python3.11 down through python3.6

Notes:

- Python virtual environments provide binaries for "python3", "python",
  and whichever version you used to create the venv,
  e.g. "python3.8". If configure is invoked from inside of a venv, this
  configure loop will not "break out" of that venv unless that venv is
  created using an explicitly non-suitable version of Python that we
  cannot use.

- In the event that no suitable python is found, the first python found
  is the version used to generate the human-readable error message.

- The error message isn't printed right away to allow later
  configuration code to pick up an explicitly configured python.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/configure b/configure
index cf6db3d5518..6abf5a72078 100755
--- a/configure
+++ b/configure
@@ -592,20 +592,40 @@ esac
 
 : ${make=${MAKE-make}}
 
-# We prefer python 3.x. A bare 'python' is traditionally
-# python 2.x, but some distros have it as python 3.x, so
-# we check that too
+
+check_py_version() {
+    # We require python >= 3.6.
+    # NB: a True python conditional creates a non-zero return code (Failure)
+    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
+}
+
 python=
+first_python=
 explicit_python=no
-for binary in "${PYTHON-python3}" python
+# Check for $PYTHON, python3, python, then explicitly-versioned interpreters.
+for binary in "${PYTHON-python3}" ${PYTHON:+python3} python \
+                                  python3.11 python3.10 python3.9 \
+                                  python3.8 python3.7 python3.6
 do
     if has "$binary"
     then
         python=$(command -v "$binary")
-        break
+        if test -z "$first_python"; then
+           first_python=$python
+        fi
+        if check_py_version "$python"; then
+            # This one is good.
+            first_python=
+            break
+        fi
     fi
 done
 
+# If first_python is set, we didn't find a suitable binary.
+# Use this one for possible future error messages.
+if test -n "$first_python"; then
+    python="$first_python"
+fi
 
 # Check for ancillary tools used in testing
 genisoimage=
@@ -1037,9 +1057,7 @@ then
     error_exit "GNU make ($make) not found"
 fi
 
-# Note that if the Python conditional here evaluates True we will exit
-# with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
+if ! check_py_version "$python"; then
   error_exit "Cannot use '$python', Python >= 3.6 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
-- 
2.39.0

[PATCH v3 2/6] configure: Add courtesy hint to Python version failure message

[John Snow]

If we begin requiring Python 3.7+, a few platforms are going to need to
install an additional Python interpreter package.

As a courtesy to the user, suggest the optional package they might need
to install. This will hopefully minimize any downtime caused by the
change in Python dependency.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index 6abf5a72078..0d0cca53f09 100755
--- a/configure
+++ b/configure
@@ -1059,7 +1059,10 @@ fi
 
 if ! check_py_version "$python"; then
   error_exit "Cannot use '$python', Python >= 3.6 is required." \
-      "Use --python=/path/to/python to specify a supported Python."
+             "Use --python=/path/to/python to specify a supported Python." \
+             "Maybe try:" \
+             "  openSUSE Leap 15.3+: zypper install python39" \
+             "  CentOS 8: dnf install python38"
 fi
 
 # Suppress writing compiled files
-- 
2.39.0

[PATCH v3 3/6] DO-NOT-MERGE: testing: Add Python >= 3.7 to Centos, OpenSuSE

[John Snow]

This is just a proof-of-concept patch, as these files are lcitool
generated. The real fix will involve updating the lcitool configuration
and updating these files that way.

Paolo has been working on this part:
https://lists.gnu.org/archive/html/qemu-devel/2023-01/msg03547.html

This patch, meanwhile, is just to prove that bumping our dependency does
not introduce any regressions in our test suite or developer
processes. It also is meant to demonstrate the relatively small changes
needed to begin utilizing 3.7 as a minimum.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/docker/dockerfiles/centos8.docker       | 1 +
 tests/docker/dockerfiles/opensuse-leap.docker | 1 +
 2 files changed, 2 insertions(+)

diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index fbc953c6dcc..a3bfddf382d 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -95,6 +95,7 @@ RUN dnf distro-sync -y && \
         pkgconfig \
         pulseaudio-libs-devel \
         python3 \
+        python38 \
         python3-PyYAML \
         python3-numpy \
         python3-pillow \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 4b2c02d6abf..9e688c1d441 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -89,6 +89,7 @@ RUN zypper update -y && \
            pam-devel \
            pcre-devel-static \
            pkgconfig \
+           python39 \
            python3-Pillow \
            python3-PyYAML \
            python3-Sphinx \
-- 
2.39.0

[PATCH v3 4/6] DO-NOT-MERGE: testing: add pip-installed sphinx-build to CentOS 8

[John Snow]

This is just for sake of demonstration again; by modifying the
dockerfile we can continue to support building docs on CentOS 8 with
minimal pain.

Note: This uses pip to install packages as root; this is hygienically
not a *great* idea for arbitrary systems. For CI, I am abusing the fact
that this is a custom-built environment that does not get altered after
being built. Individual developers should almost certainly just use a
virtual environment to avoid interfering with their system.

That process can look like this:

1. Install the new python interpreter:
   > sudo dnf install python38

2. Go to a folder where you'd be happy to store some python crud:
   > cd ~/.cache

3. Create a virtual environment using the new python:
   > python3.8 -m venv qemu_venv

4. Install a modern version of sphinx into this venv:
   > ./qemu_venv/bin/pip install sphinx

5. Try running sphinx-build:
   > ./qemu_venv/bin/sphinx-build --help

From here, you can specify ~/.cache/qemu_venv/bin/sphinx-build as your
sphinx binary to configure in order to get doc building. This approach
doesn't interfere with anything; it requires you specifically to opt
into using it, which is likely the safest option if you don't have a lot
of Python knowhow.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/docker/dockerfiles/centos8.docker | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index a3bfddf382d..a53ccada55a 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -129,6 +129,10 @@ RUN dnf distro-sync -y && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
+RUN /usr/bin/python3.8 -m ensurepip && \
+    /usr/bin/python3.8 -m pip --no-cache-dir install sphinx sphinx-rtd-theme
+
+
 ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 ENV LANG "en_US.UTF-8"
 ENV MAKE "/usr/bin/make"
-- 
2.39.0

[PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[John Snow]

Once upon a time, "sphinx-build" on certain RPM platforms invoked
specifically a Python 2.x version, while "sphinx-build-3" was a distro
shim for the Python 3.x version.

These days, none of our supported platforms utilize a 2.x version, so it
should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
which will prefer pip/venv installed versions of sphinx if they're
available.

This adds an extremely convenient ability to test document building
ability in QEMU across multiple versions of Sphinx for the purposes of
compatibility testing.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/meson.build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/meson.build b/docs/meson.build
index 9136fed3b73..906034f9a87 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -1,5 +1,5 @@
 if get_option('sphinx_build') == ''
-  sphinx_build = find_program(['sphinx-build-3', 'sphinx-build'],
+  sphinx_build = find_program(['sphinx-build', 'sphinx-build-3'],
                               required: get_option('docs'))
 else
   sphinx_build = find_program(get_option('sphinx_build'),
-- 
2.39.0

[PATCH v3 6/6] Python: Drop support for Python 3.6

[John Snow]

Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
been dropping support for this version and it is becoming more
cumbersome to support. Avocado-framework and qemu.qmp each have their
own reasons for wanting to drop Python 3.6, but won't until QEMU does.

Versions of Python available in our supported build platforms as of today,
with optional versions available in parentheses:

openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2)
CentOS Stream 8:    3.6.8  (3.8.13, 3.9.16)
CentOS Stream 9:    3.9.13
Fedora 36:          3.10
Fedora 37:          3.11
Debian 11:          3.9.2
Alpine 3.14, 3.15:  3.9.16
Alpine 3.16, 3.17:  3.10.10
Ubuntu 20.04 LTS:   3.8.10
Ubuntu 22.04 LTS:   3.10.4
NetBSD 9.3:         3.9.13*
FreeBSD 12.4:       3.9.16
FreeBSD 13.1:       3.9.16
OpenBSD 7.2:        3.9.16

Note: Our VM tests install 3.7 specifically for freebsd and netbsd; the
default for "python" or "python3" in FreeBSD is 3.9.16. NetBSD does not
appear to have a default meta-package, but offers several options, the
lowest of which is 3.7.15. "python39" appears to be a pre-requisite to
one of the other packages we request in tests/vm/netbsd.

Since it is safe under our supported platform policy, bump our minimum
supported version of Python to 3.7.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py             |  4 ++--
 configure                |  8 ++++----
 python/Makefile          | 10 +++++-----
 python/setup.cfg         |  7 +++----
 python/tests/minreqs.txt |  2 +-
 scripts/qapi/mypy.ini    |  2 +-
 6 files changed, 16 insertions(+), 17 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index 73a287a4f27..d40448f35d9 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -37,9 +37,9 @@
 # In newer versions of Sphinx this will display nicely; in older versions
 # Sphinx will also produce a Python backtrace but at least the information
 # gets printed...
-if sys.version_info < (3,6):
+if sys.version_info < (3,7):
     raise ConfigError(
-        "QEMU requires a Sphinx that uses Python 3.6 or better\n")
+        "QEMU requires a Sphinx that uses Python 3.7 or better\n")
 
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
diff --git a/configure b/configure
index 0d0cca53f09..ea0d0e49b2d 100755
--- a/configure
+++ b/configure
@@ -594,9 +594,9 @@ esac
 
 
 check_py_version() {
-    # We require python >= 3.6.
+    # We require python >= 3.7.
     # NB: a True python conditional creates a non-zero return code (Failure)
-    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
+    "$1" -c 'import sys; sys.exit(sys.version_info < (3,7))'
 }
 
 python=
@@ -605,7 +605,7 @@ explicit_python=no
 # Check for $PYTHON, python3, python, then explicitly-versioned interpreters.
 for binary in "${PYTHON-python3}" ${PYTHON:+python3} python \
                                   python3.11 python3.10 python3.9 \
-                                  python3.8 python3.7 python3.6
+                                  python3.8 python3.7
 do
     if has "$binary"
     then
@@ -1058,7 +1058,7 @@ then
 fi
 
 if ! check_py_version "$python"; then
-  error_exit "Cannot use '$python', Python >= 3.6 is required." \
+  error_exit "Cannot use '$python', Python >= 3.7 is required." \
              "Use --python=/path/to/python to specify a supported Python." \
              "Maybe try:" \
              "  openSUSE Leap 15.3+: zypper install python39" \
diff --git a/python/Makefile b/python/Makefile
index c5bd6ff83ac..f660d999143 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -9,14 +9,14 @@ help:
 	@echo "make check-minreqs:"
 	@echo "    Run tests in the minreqs virtual environment."
 	@echo "    These tests use the oldest dependencies."
-	@echo "    Requires: Python 3.6"
-	@echo "    Hint (Fedora): 'sudo dnf install python3.6'"
+	@echo "    Requires: Python 3.7"
+	@echo "    Hint (Fedora): 'sudo dnf install python3.7'"
 	@echo ""
 	@echo "make check-tox:"
 	@echo "    Run tests against multiple python versions."
 	@echo "    These tests use the newest dependencies."
-	@echo "    Requires: Python 3.6 - 3.10, and tox."
-	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.10'"
+	@echo "    Requires: Python 3.7 - 3.11, and tox."
+	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.11'"
 	@echo "    The variable QEMU_TOX_EXTRA_ARGS can be use to pass extra"
 	@echo "    arguments to tox".
 	@echo ""
@@ -58,7 +58,7 @@ pipenv check-pipenv:
 min-venv: $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate
 $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate: setup.cfg tests/minreqs.txt
 	@echo "VENV $(QEMU_MINVENV_DIR)"
-	@python3.6 -m venv $(QEMU_MINVENV_DIR)
+	@python3.7 -m venv $(QEMU_MINVENV_DIR)
 	@(								\
 		echo "ACTIVATE $(QEMU_MINVENV_DIR)";			\
 		. $(QEMU_MINVENV_DIR)/bin/activate;			\
diff --git a/python/setup.cfg b/python/setup.cfg
index 9e923d97628..1e8392a045c 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -14,7 +14,6 @@ classifiers =
     Natural Language :: English
     Operating System :: OS Independent
     Programming Language :: Python :: 3 :: Only
-    Programming Language :: Python :: 3.6
     Programming Language :: Python :: 3.7
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
@@ -23,7 +22,7 @@ classifiers =
     Typing :: Typed
 
 [options]
-python_requires = >= 3.6
+python_requires = >= 3.7
 packages =
     qemu.qmp
     qemu.machine
@@ -76,7 +75,7 @@ exclude = __pycache__,
 
 [mypy]
 strict = True
-python_version = 3.6
+python_version = 3.7
 warn_unused_configs = True
 namespace_packages = True
 warn_unused_ignores = False
@@ -158,7 +157,7 @@ multi_line_output=3
 # of python available on your system to run this test.
 
 [tox:tox]
-envlist = py36, py37, py38, py39, py310, py311
+envlist = py37, py38, py39, py310, py311
 skip_missing_interpreters = true
 
 [testenv]
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
index dfb8abb155f..55cc6b41d85 100644
--- a/python/tests/minreqs.txt
+++ b/python/tests/minreqs.txt
@@ -1,5 +1,5 @@
 # This file lists the ***oldest possible dependencies*** needed to run
-# "make check" successfully under ***Python 3.6***. It is used primarily
+# "make check" successfully under ***Python 3.7***. It is used primarily
 # by GitLab CI to ensure that our stated minimum versions in setup.cfg
 # are truthful and regularly validated.
 #
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 66253564297..3463307ddc7 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -1,7 +1,7 @@
 [mypy]
 strict = True
 disallow_untyped_calls = False
-python_version = 3.6
+python_version = 3.7
 
 [mypy-qapi.schema]
 disallow_untyped_defs = False
-- 
2.39.0

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Once upon a time, "sphinx-build" on certain RPM platforms invoked
> specifically a Python 2.x version, while "sphinx-build-3" was a distro
> shim for the Python 3.x version.
>
> These days, none of our supported platforms utilize a 2.x version, so it
> should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
> which will prefer pip/venv installed versions of sphinx if they're
> available.
>
> This adds an extremely convenient ability to test document building
> ability in QEMU across multiple versions of Sphinx for the purposes of
> compatibility testing.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/meson.build | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/docs/meson.build b/docs/meson.build
> index 9136fed3b73..906034f9a87 100644
> --- a/docs/meson.build
> +++ b/docs/meson.build
> @@ -1,5 +1,5 @@
>  if get_option('sphinx_build') == ''
> -  sphinx_build = find_program(['sphinx-build-3', 'sphinx-build'],
> +  sphinx_build = find_program(['sphinx-build', 'sphinx-build-3'],
>                                required: get_option('docs'))
>  else
>    sphinx_build = find_program(get_option('sphinx_build'),

Do we still need to check for sphinx-build-3?  Or asked differently, is
there any supported build host that provides only sphinx-build-3?

Re: [PATCH v3 0/6] Python: Drop support for Python 3.6

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> CI: https://gitlab.com/jsnow/qemu/-/pipelines/783612696
>     [Updated for v3, still all green.]
> GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37
>
> Hi, discussion about this series is ongoing. This series (v3) is not
> meant to address all of that discussion, but rather is an updated
> baseline for what we are capable of right now, today, without much
> additional engineering. It's meant to serve as a reference for further
> discussion.

Misses the RFC tag then :)

> To my knowledge, the inconveniences caused by this patchset as currently
> written are:
>
> (1) Users of CentOS 8 and OpenSUSE 15.4 would need to install an
>     additional python package that will exist side-by-side with their
>     base platform's Python 3.6 package.
>
>     "zypper install python39" or "dnf install python38" is enough;
>     configure will do the rest of the work.
>
>     It's my understanding that this is largely a non-issue.
>
> (2) Due to our Sphinx plugin that imports QAPI code from the tree,

I can read this as "Due to our Sphinx plugin (which by the way imports
some QAPI code)" or as "Due to out Sphinx plugin importing QAPI code".
The former is more accurate.  We need a newer Sphinx because we use a
plugin, the plugin is written in Python, so our new Python requirement
applies.  Fine print: the code the plugin imports from QAPI is going to
break first.

>     distro-provided versions of Sphinx that are installed and tied to
>     Python 3.6 will no longer be suitable. Users may forego building
>     docs or install a suitable sphinx using "pip".
>
>     It's my understanding that this one is "kind of a bummer".
>
> I feel that the inconvenience caused by (1) is minimized as is possible;
> the inconvenience caused by (2) is slightly worse and I concede the
> workaround has some complexities that I would otherwise seek to avoid.
>
> As far as I am aware, the way forward is to work with Paolo to implement
> a proper venv solution for the build tree that will help mitigate the
> fallout from (2) by automating the use of a pip-provided Sphinx in the
> cases where the distro-provided version is insufficient.

So, your current plan is to rebase this series less its DO-NOT-MERGE
parts, on top of Paolo's.  Correct?

> OK, seeya later!

Re: [PATCH v3 6/6] Python: Drop support for Python 3.6

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
> been dropping support for this version and it is becoming more
> cumbersome to support. Avocado-framework and qemu.qmp each have their
> own reasons for wanting to drop Python 3.6, but won't until QEMU does.

I'd prefer more thorough rationale.  I can dig it out of v2's review if
you like.  Might be best to wait for the rebase on Paolo's work, so we
know the context.

> Versions of Python available in our supported build platforms as of today,
> with optional versions available in parentheses:
>
> openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2)
> CentOS Stream 8:    3.6.8  (3.8.13, 3.9.16)
> CentOS Stream 9:    3.9.13
> Fedora 36:          3.10
> Fedora 37:          3.11
> Debian 11:          3.9.2
> Alpine 3.14, 3.15:  3.9.16
> Alpine 3.16, 3.17:  3.10.10
> Ubuntu 20.04 LTS:   3.8.10
> Ubuntu 22.04 LTS:   3.10.4
> NetBSD 9.3:         3.9.13*
> FreeBSD 12.4:       3.9.16
> FreeBSD 13.1:       3.9.16
> OpenBSD 7.2:        3.9.16
>
> Note: Our VM tests install 3.7 specifically for freebsd and netbsd; the
> default for "python" or "python3" in FreeBSD is 3.9.16. NetBSD does not
> appear to have a default meta-package, but offers several options, the
> lowest of which is 3.7.15. "python39" appears to be a pre-requisite to
> one of the other packages we request in tests/vm/netbsd.
>
> Since it is safe under our supported platform policy, bump our minimum
> supported version of Python to 3.7.
>
> Signed-off-by: John Snow <jsnow@redhat.com>

Re: [PATCH v3 2/6] configure: Add courtesy hint to Python version failure message

[Philippe Mathieu-Daudé]

On 21/2/23 02:24, John Snow wrote:
> If we begin requiring Python 3.7+, a few platforms are going to need to
> install an additional Python interpreter package.
> 
> As a courtesy to the user, suggest the optional package they might need
> to install. This will hopefully minimize any downtime caused by the
> change in Python dependency.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   configure | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH v3 2/6] configure: Add courtesy hint to Python version failure message

[Paolo Bonzini]

On 2/21/23 02:24, John Snow wrote:
> If we begin requiring Python 3.7+, a few platforms are going to need to
> install an additional Python interpreter package.
> 
> As a courtesy to the user, suggest the optional package they might need
> to install. This will hopefully minimize any downtime caused by the
> change in Python dependency.
> 
> Signed-off-by: John Snow<jsnow@redhat.com>
> ---
>   configure | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/configure b/configure
> index 6abf5a72078..0d0cca53f09 100755
> --- a/configure
> +++ b/configure
> @@ -1059,7 +1059,10 @@ fi
>   
>   if ! check_py_version "$python"; then
>     error_exit "Cannot use '$python', Python >= 3.6 is required." \
> -      "Use --python=/path/to/python to specify a supported Python."
> +             "Use --python=/path/to/python to specify a supported Python." \
> +             "Maybe try:" \
> +             "  openSUSE Leap 15.3+: zypper install python39" \
> +             "  CentOS 8: dnf install python38"
>   fi
>   
>   # Suppress writing compiled files
> -- 2.39.0

Let's delay this past the actual patch to introduce 3.7.

Paolo

Re: [PATCH v3 1/6] configure: Look for auxiliary Python installations

[Paolo Bonzini]

On 2/21/23 02:24, John Snow wrote:
> At the moment, we look for just "python3" and "python", which is good
> enough almost all of the time. But ... if you are on a platform that
> uses an older Python by default and only offers a newer Python as an
> option, you'll have to specify --python=/usr/bin/foo every time.
> 
> As a courtesy, we can make a cursory attempt to locate a suitable Python
> binary ourselves, looking for the remaining well-known binaries. This
> also has the added benefit of making configure "just work" more often
> on various BSD distributions that do not have the concept of a
> "platform default python".
> 
> This configure loop will prefer, in order:
> 
> 1. Whatever is specified in $PYTHON
> 2. python3
> 3. python (Which is usually 2.x, but might be 3.x on some platforms.)
> 4. python3.11 down through python3.6
> 
> Notes:
> 
> - Python virtual environments provide binaries for "python3", "python",
>    and whichever version you used to create the venv,
>    e.g. "python3.8". If configure is invoked from inside of a venv, this
>    configure loop will not "break out" of that venv unless that venv is
>    created using an explicitly non-suitable version of Python that we
>    cannot use.
> 
> - In the event that no suitable python is found, the first python found
>    is the version used to generate the human-readable error message.
> 
> - The error message isn't printed right away to allow later
>    configuration code to pick up an explicitly configured python.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   configure | 34 ++++++++++++++++++++++++++--------
>   1 file changed, 26 insertions(+), 8 deletions(-)
> 
> diff --git a/configure b/configure
> index cf6db3d5518..6abf5a72078 100755
> --- a/configure
> +++ b/configure
> @@ -592,20 +592,40 @@ esac
>   
>   : ${make=${MAKE-make}}
>   
> -# We prefer python 3.x. A bare 'python' is traditionally
> -# python 2.x, but some distros have it as python 3.x, so
> -# we check that too
> +
> +check_py_version() {
> +    # We require python >= 3.6.
> +    # NB: a True python conditional creates a non-zero return code (Failure)
> +    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
> +}
> +
>   python=
> +first_python=
>   explicit_python=no
> -for binary in "${PYTHON-python3}" python
> +# Check for $PYTHON, python3, python, then explicitly-versioned interpreters.
> +for binary in "${PYTHON-python3}" ${PYTHON:+python3} python \
> +                                  python3.11 python3.10 python3.9 \
> +                                  python3.8 python3.7 python3.6

I think if PYTHON is set we shouldn't look at anything else.

Paolo

>   do
>       if has "$binary"
>       then
>           python=$(command -v "$binary")
> -        break
> +        if test -z "$first_python"; then
> +           first_python=$python
> +        fi
> +        if check_py_version "$python"; then
> +            # This one is good.
> +            first_python=
> +            break
> +        fi
>       fi
>   done
>   
> +# If first_python is set, we didn't find a suitable binary.
> +# Use this one for possible future error messages.
> +if test -n "$first_python"; then
> +    python="$first_python"
> +fi
>   
>   # Check for ancillary tools used in testing
>   genisoimage=
> @@ -1037,9 +1057,7 @@ then
>       error_exit "GNU make ($make) not found"
>   fi
>   
> -# Note that if the Python conditional here evaluates True we will exit
> -# with status 1 which is a shell 'false' value.
> -if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
> +if ! check_py_version "$python"; then
>     error_exit "Cannot use '$python', Python >= 3.6 is required." \
>         "Use --python=/path/to/python to specify a supported Python."
>   fi

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[Paolo Bonzini]

On 2/21/23 02:24, John Snow wrote:
> Once upon a time, "sphinx-build" on certain RPM platforms invoked
> specifically a Python 2.x version, while "sphinx-build-3" was a distro
> shim for the Python 3.x version.
> 
> These days, none of our supported platforms utilize a 2.x version, so it
> should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
> which will prefer pip/venv installed versions of sphinx if they're
> available.
> 
> This adds an extremely convenient ability to test document building
> ability in QEMU across multiple versions of Sphinx for the purposes of
> compatibility testing.

Can we just use "$PYTHON -m sphinx.cmd.build" instead, to ensure that we don't
escape the virtual environment?  Or even better, we could have a simple script
like this:

#! /usr/bin/env python3

from pkg_resources import load_entry_point

if __name__ == '__main__':
     if sys.argv[1] == '--check':
         try:
             load_entry_point(sys.argv[2], 'console_scripts', sys.argv[3])
             sys.exit(0)
         except ImportError:
             sys.exit(1)
     else:
         entry_point = load_entry_point(sys.argv[1], 'console_scripts', sys.argv[2])
         # The second argument to python-run.py becomes sys.argv[0]
         del sys.argv[0:1]
         sys.exit(entry_point())

then docs/meson.build can do this:

python_run = find_program('scripts/python-run.py')
build_docs = false
if get_feature('docs') \
   .require(run_command(python_run, '--check', 'sphinx', 'sphinx-build',
                        check: false).returncode() == 0,
            error: 'Could not find sphinx installation') \
   .allowed()
   # The sphinx module is installed
   SPHINX_ARGS = ['env', 'CONFDIR=' + qemu_confdir,
                  python_run, 'sphinx', 'sphinx-build', '-q']
   ...
   build_docs = (sphinx_build_test_out.returncode() == 0)
   ...
endif

This again ensures that sphinx-build will not escape the virtual environment
if there is one.  configure can also use the script to run meson, though that
can come later.

Paolo

Re: [PATCH v3 0/6] Python: Drop support for Python 3.6

[Paolo Bonzini]

On 2/21/23 08:00, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> CI: https://gitlab.com/jsnow/qemu/-/pipelines/783612696
>>      [Updated for v3, still all green.]
>> GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37
>>
>> Hi, discussion about this series is ongoing. This series (v3) is not
>> meant to address all of that discussion, but rather is an updated
>> baseline for what we are capable of right now, today, without much
>> additional engineering. It's meant to serve as a reference for further
>> discussion.
> 
> Misses the RFC tag then :)
> 
>> To my knowledge, the inconveniences caused by this patchset as currently
>> written are:
>>
>> (1) Users of CentOS 8 and OpenSUSE 15.4 would need to install an
>>      additional python package that will exist side-by-side with their
>>      base platform's Python 3.6 package.
>>
>>      "zypper install python39" or "dnf install python38" is enough;
>>      configure will do the rest of the work.
>>
>>      It's my understanding that this is largely a non-issue.
>>
>> (2) Due to our Sphinx plugin that imports QAPI code from the tree,
> 
> I can read this as "Due to our Sphinx plugin (which by the way imports
> some QAPI code)" or as "Due to out Sphinx plugin importing QAPI code".
> The former is more accurate.  We need a newer Sphinx because we use a
> plugin, the plugin is written in Python, so our new Python requirement
> applies.  Fine print: the code the plugin imports from QAPI is going to
> break first.
> 
>>      distro-provided versions of Sphinx that are installed and tied to
>>      Python 3.6 will no longer be suitable. Users may forego building
>>      docs or install a suitable sphinx using "pip".
>>
>>      It's my understanding that this one is "kind of a bummer".
>>
>> I feel that the inconvenience caused by (1) is minimized as is possible;
>> the inconvenience caused by (2) is slightly worse and I concede the
>> workaround has some complexities that I would otherwise seek to avoid.
>>
>> As far as I am aware, the way forward is to work with Paolo to implement
>> a proper venv solution for the build tree that will help mitigate the
>> fallout from (2) by automating the use of a pip-provided Sphinx in the
>> cases where the distro-provided version is insufficient.
> 
> So, your current plan is to rebase this series less its DO-NOT-MERGE
> parts, on top of Paolo's.  Correct?

Yes, I will post a non-RFC v4 once all the feedback is gathered.

Paolo

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[Paolo Bonzini]

On 2/21/23 12:31, Paolo Bonzini wrote:
> On 2/21/23 02:24, John Snow wrote:
>> Once upon a time, "sphinx-build" on certain RPM platforms invoked
>> specifically a Python 2.x version, while "sphinx-build-3" was a distro
>> shim for the Python 3.x version.
>>
>> These days, none of our supported platforms utilize a 2.x version, so it
>> should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
>> which will prefer pip/venv installed versions of sphinx if they're
>> available.
>>
>> This adds an extremely convenient ability to test document building
>> ability in QEMU across multiple versions of Sphinx for the purposes of
>> compatibility testing.
> 
> Can we just use "$PYTHON -m sphinx.cmd.build" instead, to ensure that we 
> don't
> escape the virtual environment?  Or even better, we could have a simple 
> script
> like this:
> 
> #! /usr/bin/env python3
> 
> from pkg_resources import load_entry_point
> 
> if __name__ == '__main__':
>      if sys.argv[1] == '--check':
>          try:
>              load_entry_point(sys.argv[2], 'console_scripts', sys.argv[3])
>              sys.exit(0)
>          except ImportError:
>              sys.exit(1)
>      else:
>          entry_point = load_entry_point(sys.argv[1], 'console_scripts', 
> sys.argv[2])
>          # The second argument to python-run.py becomes sys.argv[0]
>          del sys.argv[0:1]
>          sys.exit(entry_point())
> 
> then docs/meson.build can do this:
> 
> python_run = find_program('scripts/python-run.py')
> build_docs = false
> if get_feature('docs') \
>    .require(run_command(python_run, '--check', 'sphinx', 'sphinx-build',
>                         check: false).returncode() == 0,
>             error: 'Could not find sphinx installation') \
>    .allowed()
>    # The sphinx module is installed
>    SPHINX_ARGS = ['env', 'CONFDIR=' + qemu_confdir,
>                   python_run, 'sphinx', 'sphinx-build', '-q']
>    ...
>    build_docs = (sphinx_build_test_out.returncode() == 0)
>    ...
> endif
> 
> This again ensures that sphinx-build will not escape the virtual 
> environment
> if there is one.  configure can also use the script to run meson, though 
> that
> can come later.

Ok, it's a bit harder but it works.  Patch coming later today.

Paolo

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]

On Tue, Feb 21, 2023, 1:50 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Once upon a time, "sphinx-build" on certain RPM platforms invoked
> > specifically a Python 2.x version, while "sphinx-build-3" was a distro
> > shim for the Python 3.x version.
> >
> > These days, none of our supported platforms utilize a 2.x version, so it
> > should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
> > which will prefer pip/venv installed versions of sphinx if they're
> > available.
> >
> > This adds an extremely convenient ability to test document building
> > ability in QEMU across multiple versions of Sphinx for the purposes of
> > compatibility testing.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  docs/meson.build | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/docs/meson.build b/docs/meson.build
> > index 9136fed3b73..906034f9a87 100644
> > --- a/docs/meson.build
> > +++ b/docs/meson.build
> > @@ -1,5 +1,5 @@
> >  if get_option('sphinx_build') == ''
> > -  sphinx_build = find_program(['sphinx-build-3', 'sphinx-build'],
> > +  sphinx_build = find_program(['sphinx-build', 'sphinx-build-3'],
> >                                required: get_option('docs'))
> >  else
> >    sphinx_build = find_program(get_option('sphinx_build'),
>
> Do we still need to check for sphinx-build-3?  Or asked differently, is
> there any supported build host that provides only sphinx-build-3?
>

Yes, modern Fedora still uses "sphinx-build-3" as the name in /usr/bin for
the rpm-packaged version of sphinx.

It's just that the only platforms where "sphinx-build" is the 2.x version
are platforms on which we want to drop 3.6 support anyway, so it's OK to
invert the search priority in the context of this series.

(All pip/pypi versions use "sphinx-build" as the binary name. In effect,
this patch means we prefer pip/pypi versions if they're in your $PATH.)


>

[-- Attachment #2: Type: text/html, Size: 3091 bytes --]

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2942 bytes --]

On Tue, Feb 21, 2023, 6:31 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 2/21/23 02:24, John Snow wrote:
> > Once upon a time, "sphinx-build" on certain RPM platforms invoked
> > specifically a Python 2.x version, while "sphinx-build-3" was a distro
> > shim for the Python 3.x version.
> >
> > These days, none of our supported platforms utilize a 2.x version, so it
> > should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
> > which will prefer pip/venv installed versions of sphinx if they're
> > available.
> >
> > This adds an extremely convenient ability to test document building
> > ability in QEMU across multiple versions of Sphinx for the purposes of
> > compatibility testing.
>
> Can we just use "$PYTHON -m sphinx.cmd.build" instead, to ensure that we
> don't
> escape the virtual environment?  Or even better, we could have a simple
> script
> like this:
>
> #! /usr/bin/env python3
>
> from pkg_resources import load_entry_point
>
> if __name__ == '__main__':
>      if sys.argv[1] == '--check':
>          try:
>              load_entry_point(sys.argv[2], 'console_scripts', sys.argv[3])
>              sys.exit(0)
>          except ImportError:
>              sys.exit(1)
>      else:
>          entry_point = load_entry_point(sys.argv[1], 'console_scripts',
> sys.argv[2])
>          # The second argument to python-run.py becomes sys.argv[0]
>          del sys.argv[0:1]
>          sys.exit(entry_point())
>
> then docs/meson.build can do this:
>
> python_run = find_program('scripts/python-run.py')
> build_docs = false
> if get_feature('docs') \
>    .require(run_command(python_run, '--check', 'sphinx', 'sphinx-build',
>                         check: false).returncode() == 0,
>             error: 'Could not find sphinx installation') \
>    .allowed()
>    # The sphinx module is installed
>    SPHINX_ARGS = ['env', 'CONFDIR=' + qemu_confdir,
>                   python_run, 'sphinx', 'sphinx-build', '-q']
>    ...
>    build_docs = (sphinx_build_test_out.returncode() == 0)
>    ...
> endif
>
> This again ensures that sphinx-build will not escape the virtual
> environment
> if there is one.  configure can also use the script to run meson, though
> that
> can come later.
>
> Paolo
>

Yeah, I proposed we use "python3 -m sphinx.cmd.build" once, but Peter did
not like the idea of Sphinx becoming a python dependency instead of being
treated as a black box.

Obviously circumstances are shifting somewhat and we may be more open to
treating Sphinx as a python dependency given that we need to enforce
compatibility with custom plugins written in qemu.git.

If I was trying to please absolutely nobody but me, I'd certainly use the
`$python -m sphinx` approach; especially because it means that for
qapi-gen, the code is run under the same environment in both cases (native
qapi-gen exec and sphinx doc building).

I'm for it, but lost appetite for making the argument some time back.

>

[-- Attachment #2: Type: text/html, Size: 4066 bytes --]

Re: [PATCH v3 1/6] configure: Look for auxiliary Python installations

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 4069 bytes --]

On Tue, Feb 21, 2023, 6:03 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 2/21/23 02:24, John Snow wrote:
> > At the moment, we look for just "python3" and "python", which is good
> > enough almost all of the time. But ... if you are on a platform that
> > uses an older Python by default and only offers a newer Python as an
> > option, you'll have to specify --python=/usr/bin/foo every time.
> >
> > As a courtesy, we can make a cursory attempt to locate a suitable Python
> > binary ourselves, looking for the remaining well-known binaries. This
> > also has the added benefit of making configure "just work" more often
> > on various BSD distributions that do not have the concept of a
> > "platform default python".
> >
> > This configure loop will prefer, in order:
> >
> > 1. Whatever is specified in $PYTHON
> > 2. python3
> > 3. python (Which is usually 2.x, but might be 3.x on some platforms.)
> > 4. python3.11 down through python3.6
> >
> > Notes:
> >
> > - Python virtual environments provide binaries for "python3", "python",
> >    and whichever version you used to create the venv,
> >    e.g. "python3.8". If configure is invoked from inside of a venv, this
> >    configure loop will not "break out" of that venv unless that venv is
> >    created using an explicitly non-suitable version of Python that we
> >    cannot use.
> >
> > - In the event that no suitable python is found, the first python found
> >    is the version used to generate the human-readable error message.
> >
> > - The error message isn't printed right away to allow later
> >    configuration code to pick up an explicitly configured python.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >   configure | 34 ++++++++++++++++++++++++++--------
> >   1 file changed, 26 insertions(+), 8 deletions(-)
> >
> > diff --git a/configure b/configure
> > index cf6db3d5518..6abf5a72078 100755
> > --- a/configure
> > +++ b/configure
> > @@ -592,20 +592,40 @@ esac
> >
> >   : ${make=${MAKE-make}}
> >
> > -# We prefer python 3.x. A bare 'python' is traditionally
> > -# python 2.x, but some distros have it as python 3.x, so
> > -# we check that too
> > +
> > +check_py_version() {
> > +    # We require python >= 3.6.
> > +    # NB: a True python conditional creates a non-zero return code
> (Failure)
> > +    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
> > +}
> > +
> >   python=
> > +first_python=
> >   explicit_python=no
> > -for binary in "${PYTHON-python3}" python
> > +# Check for $PYTHON, python3, python, then explicitly-versioned
> interpreters.
> > +for binary in "${PYTHON-python3}" ${PYTHON:+python3} python \
> > +                                  python3.11 python3.10 python3.9 \
> > +                                  python3.8 python3.7 python3.6
>
> I think if PYTHON is set we shouldn't look at anything else.
>
> Paolo
>

PYTHON is one we made up, right?


> >   do
> >       if has "$binary"
> >       then
> >           python=$(command -v "$binary")
> > -        break
> > +        if test -z "$first_python"; then
> > +           first_python=$python
> > +        fi
> > +        if check_py_version "$python"; then
> > +            # This one is good.
> > +            first_python=
> > +            break
> > +        fi
> >       fi
> >   done
> >
> > +# If first_python is set, we didn't find a suitable binary.
> > +# Use this one for possible future error messages.
> > +if test -n "$first_python"; then
> > +    python="$first_python"
> > +fi
> >
> >   # Check for ancillary tools used in testing
> >   genisoimage=
> > @@ -1037,9 +1057,7 @@ then
> >       error_exit "GNU make ($make) not found"
> >   fi
> >
> > -# Note that if the Python conditional here evaluates True we will exit
> > -# with status 1 which is a shell 'false' value.
> > -if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
> > +if ! check_py_version "$python"; then
> >     error_exit "Cannot use '$python', Python >= 3.6 is required." \
> >         "Use --python=/path/to/python to specify a supported Python."
> >   fi
>
>

[-- Attachment #2: Type: text/html, Size: 5739 bytes --]

Re: [PATCH v3 1/6] configure: Look for auxiliary Python installations

[Daniel P. Berrangé]

On Tue, Feb 21, 2023 at 12:37:43PM -0500, John Snow wrote:
> On Tue, Feb 21, 2023, 6:03 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> > On 2/21/23 02:24, John Snow wrote:
> > > At the moment, we look for just "python3" and "python", which is good
> > > enough almost all of the time. But ... if you are on a platform that
> > > uses an older Python by default and only offers a newer Python as an
> > > option, you'll have to specify --python=/usr/bin/foo every time.
> > >
> > > As a courtesy, we can make a cursory attempt to locate a suitable Python
> > > binary ourselves, looking for the remaining well-known binaries. This
> > > also has the added benefit of making configure "just work" more often
> > > on various BSD distributions that do not have the concept of a
> > > "platform default python".
> > >
> > > This configure loop will prefer, in order:
> > >
> > > 1. Whatever is specified in $PYTHON
> > > 2. python3
> > > 3. python (Which is usually 2.x, but might be 3.x on some platforms.)
> > > 4. python3.11 down through python3.6
> > >
> > > Notes:
> > >
> > > - Python virtual environments provide binaries for "python3", "python",
> > >    and whichever version you used to create the venv,
> > >    e.g. "python3.8". If configure is invoked from inside of a venv, this
> > >    configure loop will not "break out" of that venv unless that venv is
> > >    created using an explicitly non-suitable version of Python that we
> > >    cannot use.
> > >
> > > - In the event that no suitable python is found, the first python found
> > >    is the version used to generate the human-readable error message.
> > >
> > > - The error message isn't printed right away to allow later
> > >    configuration code to pick up an explicitly configured python.
> > >
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >   configure | 34 ++++++++++++++++++++++++++--------
> > >   1 file changed, 26 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/configure b/configure
> > > index cf6db3d5518..6abf5a72078 100755
> > > --- a/configure
> > > +++ b/configure
> > > @@ -592,20 +592,40 @@ esac
> > >
> > >   : ${make=${MAKE-make}}
> > >
> > > -# We prefer python 3.x. A bare 'python' is traditionally
> > > -# python 2.x, but some distros have it as python 3.x, so
> > > -# we check that too
> > > +
> > > +check_py_version() {
> > > +    # We require python >= 3.6.
> > > +    # NB: a True python conditional creates a non-zero return code
> > (Failure)
> > > +    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
> > > +}
> > > +
> > >   python=
> > > +first_python=
> > >   explicit_python=no
> > > -for binary in "${PYTHON-python3}" python
> > > +# Check for $PYTHON, python3, python, then explicitly-versioned
> > interpreters.
> > > +for binary in "${PYTHON-python3}" ${PYTHON:+python3} python \
> > > +                                  python3.11 python3.10 python3.9 \
> > > +                                  python3.8 python3.7 python3.6
> >
> > I think if PYTHON is set we shouldn't look at anything else.
> >
> > Paolo
> >
> 
> PYTHON is one we made up, right?

$PYTHON is explicitly set in all our dockerfiles. We should
ensure we honour it and not fallback to anything else when
it is set. ie it would be a user error to set it to point
to a python that is missing/broken, so the user should
expect an error, not fallback to another version.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Feb 21, 2023, 1:50 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Once upon a time, "sphinx-build" on certain RPM platforms invoked
>> > specifically a Python 2.x version, while "sphinx-build-3" was a distro
>> > shim for the Python 3.x version.
>> >
>> > These days, none of our supported platforms utilize a 2.x version, so it
>> > should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
>> > which will prefer pip/venv installed versions of sphinx if they're
>> > available.
>> >
>> > This adds an extremely convenient ability to test document building
>> > ability in QEMU across multiple versions of Sphinx for the purposes of
>> > compatibility testing.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  docs/meson.build | 2 +-
>> >  1 file changed, 1 insertion(+), 1 deletion(-)
>> >
>> > diff --git a/docs/meson.build b/docs/meson.build
>> > index 9136fed3b73..906034f9a87 100644
>> > --- a/docs/meson.build
>> > +++ b/docs/meson.build
>> > @@ -1,5 +1,5 @@
>> >  if get_option('sphinx_build') == ''
>> > -  sphinx_build = find_program(['sphinx-build-3', 'sphinx-build'],
>> > +  sphinx_build = find_program(['sphinx-build', 'sphinx-build-3'],
>> >                                required: get_option('docs'))
>> >  else
>> >    sphinx_build = find_program(get_option('sphinx_build'),
>>
>> Do we still need to check for sphinx-build-3?  Or asked differently, is
>> there any supported build host that provides only sphinx-build-3?
>>
>
> Yes, modern Fedora still uses "sphinx-build-3" as the name in /usr/bin for
> the rpm-packaged version of sphinx.

For what it's worth, python3-sphinx-5.0.2-2.fc37.noarch provides

    /usr/bin/sphinx-build
    /usr/bin/sphinx-build-3
    /usr/bin/sphinx-build-3.11

where the latter two are symbolic links to the first.  No need to check
for sphinx-build-3 here.

> It's just that the only platforms where "sphinx-build" is the 2.x version
> are platforms on which we want to drop 3.6 support anyway, so it's OK to
> invert the search priority in the context of this series.
>
> (All pip/pypi versions use "sphinx-build" as the binary name. In effect,
> this patch means we prefer pip/pypi versions if they're in your $PATH.)

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[John Snow]

On Wed, Feb 22, 2023 at 2:15 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > On Tue, Feb 21, 2023, 1:50 AM Markus Armbruster <armbru@redhat.com> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > Once upon a time, "sphinx-build" on certain RPM platforms invoked
> >> > specifically a Python 2.x version, while "sphinx-build-3" was a distro
> >> > shim for the Python 3.x version.
> >> >
> >> > These days, none of our supported platforms utilize a 2.x version, so it
> >> > should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
> >> > which will prefer pip/venv installed versions of sphinx if they're
> >> > available.
> >> >
> >> > This adds an extremely convenient ability to test document building
> >> > ability in QEMU across multiple versions of Sphinx for the purposes of
> >> > compatibility testing.
> >> >
> >> > Signed-off-by: John Snow <jsnow@redhat.com>
> >> > ---
> >> >  docs/meson.build | 2 +-
> >> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >> >
> >> > diff --git a/docs/meson.build b/docs/meson.build
> >> > index 9136fed3b73..906034f9a87 100644
> >> > --- a/docs/meson.build
> >> > +++ b/docs/meson.build
> >> > @@ -1,5 +1,5 @@
> >> >  if get_option('sphinx_build') == ''
> >> > -  sphinx_build = find_program(['sphinx-build-3', 'sphinx-build'],
> >> > +  sphinx_build = find_program(['sphinx-build', 'sphinx-build-3'],
> >> >                                required: get_option('docs'))
> >> >  else
> >> >    sphinx_build = find_program(get_option('sphinx_build'),
> >>
> >> Do we still need to check for sphinx-build-3?  Or asked differently, is
> >> there any supported build host that provides only sphinx-build-3?
> >>
> >
> > Yes, modern Fedora still uses "sphinx-build-3" as the name in /usr/bin for
> > the rpm-packaged version of sphinx.
>
> For what it's worth, python3-sphinx-5.0.2-2.fc37.noarch provides
>
>     /usr/bin/sphinx-build
>     /usr/bin/sphinx-build-3
>     /usr/bin/sphinx-build-3.11
>
> where the latter two are symbolic links to the first.  No need to check
> for sphinx-build-3 here.

Oh, I see. I guess it should be fine, but only if we explicitly drop
support for the 3.6 version that comes with CentOS. I'm not entirely
sure if "sphinx-build-3" is used anywhere else, I *think* it's just an
rpm-ism.

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, Feb 22, 2023 at 2:15 AM Markus Armbruster <armbru@redhat.com> wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Tue, Feb 21, 2023, 1:50 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > Once upon a time, "sphinx-build" on certain RPM platforms invoked
>> >> > specifically a Python 2.x version, while "sphinx-build-3" was a distro
>> >> > shim for the Python 3.x version.
>> >> >
>> >> > These days, none of our supported platforms utilize a 2.x version, so it
>> >> > should be safe to search for 'sphinx-build' prior to 'sphinx-build-3',
>> >> > which will prefer pip/venv installed versions of sphinx if they're
>> >> > available.
>> >> >
>> >> > This adds an extremely convenient ability to test document building
>> >> > ability in QEMU across multiple versions of Sphinx for the purposes of
>> >> > compatibility testing.
>> >> >
>> >> > Signed-off-by: John Snow <jsnow@redhat.com>
>> >> > ---
>> >> >  docs/meson.build | 2 +-
>> >> >  1 file changed, 1 insertion(+), 1 deletion(-)
>> >> >
>> >> > diff --git a/docs/meson.build b/docs/meson.build
>> >> > index 9136fed3b73..906034f9a87 100644
>> >> > --- a/docs/meson.build
>> >> > +++ b/docs/meson.build
>> >> > @@ -1,5 +1,5 @@
>> >> >  if get_option('sphinx_build') == ''
>> >> > -  sphinx_build = find_program(['sphinx-build-3', 'sphinx-build'],
>> >> > +  sphinx_build = find_program(['sphinx-build', 'sphinx-build-3'],
>> >> >                                required: get_option('docs'))
>> >> >  else
>> >> >    sphinx_build = find_program(get_option('sphinx_build'),
>> >>
>> >> Do we still need to check for sphinx-build-3?  Or asked differently, is
>> >> there any supported build host that provides only sphinx-build-3?
>> >>
>> >
>> > Yes, modern Fedora still uses "sphinx-build-3" as the name in /usr/bin for
>> > the rpm-packaged version of sphinx.
>>
>> For what it's worth, python3-sphinx-5.0.2-2.fc37.noarch provides
>>
>>     /usr/bin/sphinx-build
>>     /usr/bin/sphinx-build-3
>>     /usr/bin/sphinx-build-3.11
>>
>> where the latter two are symbolic links to the first.  No need to check
>> for sphinx-build-3 here.
>
> Oh, I see. I guess it should be fine, but only if we explicitly drop
> support for the 3.6 version that comes with CentOS. I'm not entirely
> sure if "sphinx-build-3" is used anywhere else, I *think* it's just an
> rpm-ism.

I can see just two reasons for trying sphinx-build-3:

1. sphinx-build does not exist.

2. sphinx-build exists, but uses Python 2, which doesn't work with our
   Sphinx extension.

The commit message seems to claim it's not 2.

So, what is it?

Re: [PATCH v3 5/6] meson: prefer 'sphinx-build' to 'sphinx-build-3'

[Paolo Bonzini]

On Thu, Feb 23, 2023 at 5:40 AM John Snow <jsnow@redhat.com> wrote:> >
For what it's worth, python3-sphinx-5.0.2-2.fc37.noarch provides
> >
> >     /usr/bin/sphinx-build
> >     /usr/bin/sphinx-build-3
> >     /usr/bin/sphinx-build-3.11
> >
> > where the latter two are symbolic links to the first.  No need to check
> > for sphinx-build-3 here.
>
> Oh, I see. I guess it should be fine, but only if we explicitly drop
> support for the 3.6 version that comes with CentOS. I'm not entirely
> sure if "sphinx-build-3" is used anywhere else, I *think* it's just an
> rpm-ism.

Are you sure CentOS has a problem there?  I checked
python3-sphinx-1.7.6-3.el8 and it has sphinx-build as a symlink too:

$ rpm2cpio ../python3-sphinx-1.7.6-3.el8.noarch.rpm |cpio -idv
$ ls -l usr/bin/
total 16
-rwxr-xr-x. 1 pbonzini users 403 Feb 23 09:50 sphinx-apidoc
lrwxrwxrwx. 1 pbonzini users  13 Feb 23 09:50 sphinx-apidoc-3 -> sphinx-apidoc
lrwxrwxrwx. 1 pbonzini users  13 Feb 23 09:50 sphinx-apidoc-3.6 -> sphinx-apidoc
-rwxr-xr-x. 1 pbonzini users 405 Feb 23 09:50 sphinx-autogen
lrwxrwxrwx. 1 pbonzini users  14 Feb 23 09:50 sphinx-autogen-3 -> sphinx-autogen
lrwxrwxrwx. 1 pbonzini users  14 Feb 23 09:50 sphinx-autogen-3.6 ->
sphinx-autogen
-rwxr-xr-x. 1 pbonzini users 401 Feb 23 09:50 sphinx-build
lrwxrwxrwx. 1 pbonzini users  12 Feb 23 09:50 sphinx-build-3 -> sphinx-build
lrwxrwxrwx. 1 pbonzini users  12 Feb 23 09:50 sphinx-build-3.6 -> sphinx-build
-rwxr-xr-x. 1 pbonzini users 411 Feb 23 09:50 sphinx-quickstart
lrwxrwxrwx. 1 pbonzini users  17 Feb 23 09:50 sphinx-quickstart-3 ->
sphinx-quickstart
lrwxrwxrwx. 1 pbonzini users  17 Feb 23 09:50 sphinx-quickstart-3.6 ->
sphinx-quickstart

And it's 3.6-based:

$ ls -l usr/lib
total 0
drwxr-xr-x. 3 pbonzini users 27 Feb 23 09:50 python3.6
$ head -4 usr/bin/sphinx-build
#!/usr/libexec/platform-python
# EASY-INSTALL-ENTRY-SCRIPT: 'Sphinx==1.7.6','console_scripts','sphinx-build'
__requires__ = 'Sphinx==1.7.6'
import re

This alternative, simpler patch should do it:

diff --git a/docs/meson.build b/docs/meson.build
index 9136fed3b730..1ab5a9882018 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -1,10 +1,5 @@
-if get_option('sphinx_build') == ''
-  sphinx_build = find_program(['sphinx-build-3', 'sphinx-build'],
-                              required: get_option('docs'))
-else
-  sphinx_build = find_program(get_option('sphinx_build'),
-                              required: get_option('docs'))
-endif
+sphinx_build = find_program(get_option('sphinx_build'),
+                            required: get_option('docs'))

 # Check if tools are available to build documentation.
 build_docs = false
diff --git a/meson_options.txt b/meson_options.txt
index 7e5801db90f9..92440429b1d0 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -12,7 +12,7 @@ option('pkgversion', type : 'string', value : '',
        description: 'use specified string as sub-version of the package')
 option('smbd', type : 'string', value : '',
        description: 'Path to smbd for slirp networking')
-option('sphinx_build', type : 'string', value : '',
+option('sphinx_build', type : 'string', value : 'sphinx-build',
        description: 'Use specified sphinx-build for building document')
 option('iasl', type : 'string', value : '',
        description: 'Path to ACPI disassembler')

Paolo

Re: [PATCH v3 1/6] configure: Look for auxiliary Python installations

[Eric Blake]

On Tue, Feb 21, 2023 at 12:03:44PM +0100, Paolo Bonzini wrote:
> > 
> > This configure loop will prefer, in order:
> > 
> > 1. Whatever is specified in $PYTHON
> > 2. python3
> > 3. python (Which is usually 2.x, but might be 3.x on some platforms.)
> > 4. python3.11 down through python3.6
> > 
> > +
> >   python=
> > +first_python=
> >   explicit_python=no
> > -for binary in "${PYTHON-python3}" python
> > +# Check for $PYTHON, python3, python, then explicitly-versioned interpreters.
> > +for binary in "${PYTHON-python3}" ${PYTHON:+python3} python \
> > +                                  python3.11 python3.10 python3.9 \
> > +                                  python3.8 python3.7 python3.6
> 
> I think if PYTHON is set we shouldn't look at anything else.

Now that you mention that, it does sound more like what autoconf does
(if the user specified something, honor it; otherwise, find the best
match in a list of candidates).

> 
> Paolo
> 
> >   do
> >       if has "$binary"
> >       then
> >           python=$(command -v "$binary")
> > -        break
> > +        if test -z "$first_python"; then
> > +           first_python=$python
> > +        fi
> > +        if check_py_version "$python"; then
> > +            # This one is good.
> > +            first_python=
> > +            break
> > +        fi

One easy way to do that is add this here:

             if test -n "$PYTHON"; then break; fi

> >       fi
> >   done
> > +# If first_python is set, we didn't find a suitable binary.
> > +# Use this one for possible future error messages.
> > +if test -n "$first_python"; then
> > +    python="$first_python"
> > +fi

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org