Build platform guarantees, docs, tests, and snakes in the garden

Build platform guarantees, docs, tests, and snakes in the garden

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3014 bytes --]

Hi, I've long been a little confused about the specifics of our build
platform guarantee and how it applies to documentation and testing.

My *current* understanding is that our build platform guarantee applies to
both unit tests and building documentation, but that this requirement may
not be as absolute as I imagine it.

The way I have endeavored to manage the Python tooling in our tree so far
is to preserve, without fail, our ability to perform fully offline builds
on all supported platforms (provided that the right distro repo packages
are available). The Python virtual environment created at configure time
bends over backwards to use system packages *whenever possible*, and the
list of exceptions - notably Meson itself - uses vendored packages only in
very specific cases where it is possible to vendor such packages. Fetching
packages from PyPI is generally offered only as a convenience for developer
workstations to, in general, save developers from having to know anything
about Python. (I think I've done a good job there, to be honest!)

(Notably: Meson is pure python and has no dependencies, so it is possible
to vendor it for offline builds. Tools like Sphinx, however, have many
dependencies and are not so easily vendored. Thus, we have created a
tenuous arrangement where we are allowed to use versions of Meson that
otherwise would break our build platform guarantee.)

Lately, we've had some issues with the wide range of Sphinx versions we
support presenting various cross-platform difficulties. In particular,
Akihiko Odaki has sent patches to bump our Sphinx version to at least
6.2.1, because platforms with Python 3.13.1 can no longer run Sphinx 3.x at
all, so having that be our "default install version" causes issues on newer
platforms.

However, if we take as iron-clad our commitment to the build platform
promise -- *and* guarantee offline/tarball builds as well -- then Debian 12
(as an example) only offers Sphinx 5.3.0 and not newer unless we allow
internet access to fetch Sphinx 6.2.1. This is not a problem for developer
workstations at all, but I am unclear on what problems this may cause for
tarball releases and downstream offline/isolated/reproducible builds, if
any.

In this case, we can (probably) "fix" the issue by continuing to allow
older Sphinx while preferring a newer Sphinx version when it is missing,
but then we lose the ability to make code cleanups and drop a lot of
back-compat crud. If memory serves, there were other issues recently where
older versions of Sphinx behaved differently from newer versions, causing
intermittent failures that were hard to track down.

What I'd like to know is: what precisely are our options in this scenario?
Do we consider it acceptable for some platforms to be unable to build docs
offline? How highly do we value the ability to locally build docs for any
given release?

Before I throw my weight behind any given option, I just want to know what
we consider our non-negotiable obligations to be.

Thanks,
--js

[-- Attachment #2: Type: text/html, Size: 3324 bytes --]

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Akihiko Odaki]

On 2025/06/06 4:35, John Snow wrote:
> Hi, I've long been a little confused about the specifics of our build 
> platform guarantee and how it applies to documentation and testing.
> 
> My *current* understanding is that our build platform guarantee applies 
> to both unit tests and building documentation, but that this requirement 
> may not be as absolute as I imagine it.
> 
> The way I have endeavored to manage the Python tooling in our tree so 
> far is to preserve, without fail, our ability to perform fully offline 
> builds on all supported platforms (provided that the right distro repo 
> packages are available). The Python virtual environment created at 
> configure time bends over backwards to use system packages *whenever 
> possible*, and the list of exceptions - notably Meson itself - uses 
> vendored packages only in very specific cases where it is possible to 
> vendor such packages. Fetching packages from PyPI is generally offered 
> only as a convenience for developer workstations to, in general, save 
> developers from having to know anything about Python. (I think I've done 
> a good job there, to be honest!)

It is a nice work indeed. I spent substantial time to work on QEMU but 
the Sphinx problem mentioned later was the only problem I encountered.

> 
> (Notably: Meson is pure python and has no dependencies, so it is 
> possible to vendor it for offline builds. Tools like Sphinx, however, 
> have many dependencies and are not so easily vendored. Thus, we have 
> created a tenuous arrangement where we are allowed to use versions of 
> Meson that otherwise would break our build platform guarantee.)
> 
> Lately, we've had some issues with the wide range of Sphinx versions we 
> support presenting various cross-platform difficulties. In particular, 
> Akihiko Odaki has sent patches to bump our Sphinx version to at least 
> 6.2.1, because platforms with Python 3.13.1 can no longer run Sphinx 3.x 
> at all, so having that be our "default install version" causes issues on 
> newer platforms.
> 
> However, if we take as iron-clad our commitment to the build platform 
> promise -- *and* guarantee offline/tarball builds as well -- then Debian 
> 12 (as an example) only offers Sphinx 5.3.0 and not newer unless we 
> allow internet access to fetch Sphinx 6.2.1. This is not a problem for 
> developer workstations at all, but I am unclear on what problems this 
> may cause for tarball releases and downstream offline/isolated/ 
> reproducible builds, if any.
> 
> In this case, we can (probably) "fix" the issue by continuing to allow 
> older Sphinx while preferring a newer Sphinx version when it is missing, 
> but then we lose the ability to make code cleanups and drop a lot of 
> back-compat crud. If memory serves, there were other issues recently 
> where older versions of Sphinx behaved differently from newer versions, 
> causing intermittent failures that were hard to track down.
> 
> What I'd like to know is: what precisely are our options in this 
> scenario? Do we consider it acceptable for some platforms to be unable 
> to build docs offline? How highly do we value the ability to locally 
> build docs for any given release?
> 
> Before I throw my weight behind any given option, I just want to know 
> what we consider our non-negotiable obligations to be.
> 
> Thanks,
> --js
> 

Thank you for detailed explanation. I see a few possible options in this 
situation.

Pretending as a lawyer interpreting docs/about/build-platforms.rst as a 
law, I think it is allowed to require the Internet access. It says:

 > For the purposes of identifying supported software versions available
 > on Linux, the project will look at CentOS, Debian, Fedora, openSUSE,
 > RHEL, SLES and Ubuntu LTS. Other distros will be assumed to ship
 > similar software versions.
 >
 > For FreeBSD and OpenBSD, decisions will be made based on the contents
 > of the respective ports repository, while NetBSD will use the pkgsrc
 > repository.
 >
 > For macOS, `Homebrew`_ will be used, although `MacPorts`_ is expected
 > to carry similar versions.
 >
 > Some build dependencies may follow less conservative rules:

 > Python build dependencies
 >   Some of QEMU's build dependencies are written in Python.  Usually
 >   these are only packaged by distributions for the default Python
 >   runtime.
 >   If QEMU bumps its minimum Python version and a non-default runtime
 >   is required, it may be necessary to fetch python modules from the
 >   Python Package Index (PyPI) via ``pip``, in order to build QEMU.

So it is allowed to require a non-default runtime that is available on 
PyPI but not on Debian.

But it is also allowed to bundle a python module or to keep its version 
requirement low to help users just as we do for Meson, of course.

Another possible option is to provide a mechanism to download the 
required Python packages; users need the Internet to to prepare the 
build platform (e.g., Debian) and QEMU source code anyway, so it makes 
sense to require downloading the Python packages at the same time for 
the future offline usage.

"pip download" is probably for such a use case, and we can make sure 
that nobody get hurt by the Sphinx change or other Python dependency 
changes by wiring up it into python/scripts/mkvenv.py.

In summary, there are the following viable options:
(1) Don't bump the Sphinx version.
(2) Require the Internet access when running ./configure --enable-docs.
(3) Bundle Sphinx.
(4) Let users "pip download".

Skipping into the conclusion, I'm for (2). The other options have their 
own disadvantages:

(1) may cause intermittent failures with old Sphinx versions, which is 
too bad.
(3) is just difficult, according on your description.
(4) is nice, but requires more time and I don't think it's worthwhile to 
take the risk of old Sphinx versions until it is implemented.

In contrast, it seems (2) has the minimum risk and acceptable level of 
annoyance. The documentation is for humans, and any human using QEMU 
should have one computer with the Internet access at least. Perhaps some 
users will be forced to change the build flag to stop building the 
documentation on an offline computer and to prepare a build environment 
on an online computer, but it's acceptable I guess.

Regards,
Akihiko Odaki

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Hi, I've long been a little confused about the specifics of our build
> platform guarantee and how it applies to documentation and testing.

"Guarantee" feels too strong.  See below.

> My *current* understanding is that our build platform guarantee applies to
> both unit tests and building documentation, but that this requirement may
> not be as absolute as I imagine it.
>
> The way I have endeavored to manage the Python tooling in our tree so far
> is to preserve, without fail, our ability to perform fully offline builds
> on all supported platforms (provided that the right distro repo packages
> are available).

Relevant part of docs/about/build-platforms.rst:

    Some build dependencies may follow less conservative rules:

    Python runtime
      Distributions with long-term support often provide multiple versions
      of the Python runtime.  While QEMU will initially aim to support the
      distribution's default runtime, it may later increase its minimum version
      to any newer python that is available as an option from the vendor.
      In this case, it will be necessary to use the ``--python`` command line
      option of the ``configure`` script to point QEMU to a supported
      version of the Python runtime.

      As of QEMU |version|, the minimum supported version of Python is 3.9.

    Python build dependencies
      Some of QEMU's build dependencies are written in Python.  Usually these
      are only packaged by distributions for the default Python runtime.
      If QEMU bumps its minimum Python version and a non-default runtime is
      required, it may be necessary to fetch python modules from the Python
      Package Index (PyPI) via ``pip``, in order to build QEMU.

We "initially aim to support the distribution's default runtime".  Once
we don't, fetching from PyPI "may be necessary".  This seems to imply
that such fetching will be necessary when we use the default runtime.

I read "aim" as "make an effort".  This isn't exactly a "guarantee".
It's much, much weaker than "to preserve, without fail, our ability to
perform fully offline builds on all supported platforms".

Keeping fully offline builds working is certainly desirable, but not
regardless of cost.

>                 The Python virtual environment created at configure time
> bends over backwards to use system packages *whenever possible*, and the
> list of exceptions - notably Meson itself - uses vendored packages only in
> very specific cases where it is possible to vendor such packages. Fetching
> packages from PyPI is generally offered only as a convenience for developer
> workstations to, in general, save developers from having to know anything
> about Python. (I think I've done a good job there, to be honest!)

You have: few people have noticed your work.

> (Notably: Meson is pure python and has no dependencies, so it is possible
> to vendor it for offline builds. Tools like Sphinx, however, have many
> dependencies and are not so easily vendored. Thus, we have created a
> tenuous arrangement where we are allowed to use versions of Meson that
> otherwise would break our build platform guarantee.)
>
> Lately, we've had some issues with the wide range of Sphinx versions we
> support presenting various cross-platform difficulties. In particular,
> Akihiko Odaki has sent patches to bump our Sphinx version to at least
> 6.2.1, because platforms with Python 3.13.1 can no longer run Sphinx 3.x at
> all, so having that be our "default install version" causes issues on newer
> platforms.
>
> However, if we take as iron-clad our commitment to the build platform
> promise -- *and* guarantee offline/tarball builds as well -- then Debian 12

I do not think such a commitment exists; see my reading of
build-platforms.rst above.

Even if it did, treating it as iron-clad would be foolish.  We need to
consider cost vs. reward, always.

> (as an example) only offers Sphinx 5.3.0 and not newer unless we allow
> internet access to fetch Sphinx 6.2.1. This is not a problem for developer
> workstations at all, but I am unclear on what problems this may cause for
> tarball releases and downstream offline/isolated/reproducible builds, if
> any.
>
> In this case, we can (probably) "fix" the issue by continuing to allow
> older Sphinx while preferring a newer Sphinx version when it is missing,
> but then we lose the ability to make code cleanups and drop a lot of
> back-compat crud. If memory serves, there were other issues recently where
> older versions of Sphinx behaved differently from newer versions, causing
> intermittent failures that were hard to track down.
>
> What I'd like to know is: what precisely are our options in this scenario?
> Do we consider it acceptable for some platforms to be unable to build docs
> offline? How highly do we value the ability to locally build docs for any
> given release?

I believe the value of fully offline builds goes down as the build
platform ages.

Initially, the distribution will (hopefully!) want to package the then
current version of QEMU.  We want to make that easy.  Enabling fully
offline builds are part of that.

Perhaps the distribution later wants to package / build newer versions
of QEMU fully offline on the now aged release.  I'd expect this to
happen less and less as the distribution ages.  Making it easy is still
desirable, but worth less and less trouble to us.

> Before I throw my weight behind any given option, I just want to know what
> we consider our non-negotiable obligations to be.

In my opinion: none.  We should consider the tradeoffs, and make
pragmatic decisions.

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Paolo Bonzini]

On 6/5/25 21:35, John Snow wrote:
> However, if we take as iron-clad our commitment to the build platform 
> promise -- *and* guarantee offline/tarball builds as well -- then Debian 
> 12 (as an example) only offers Sphinx 5.3.0 and not newer unless we 
> allow internet access to fetch Sphinx 6.2.1. This is not a problem for 
> developer workstations at all, but I am unclear on what problems this 
> may cause for tarball releases and downstream offline/isolated/ 
> reproducible builds, if any.
> 
> In this case, we can (probably) "fix" the issue by continuing to allow 
> older Sphinx while preferring a newer Sphinx version when it is missing, 
> but then we lose the ability to make code cleanups and drop a lot of 
> back-compat crud. If memory serves, there were other issues recently 
> where older versions of Sphinx behaved differently from newer versions, 
> causing intermittent failures that were hard to track down.

The *ideal* solution would be to:

- accept: 4.3.2 or newer, which is what Ubuntu 22.04 has

- install: 6.2.1, which is what supports Python 3.13

This lets all supported distros build documentation if they use the 
default Python runtime.  It would still require a couple hacks in 
compat.py: SOURCE_LOCATION_FIX and nested_parse_with_titles().

I am not sure however whether to count the latter, for two reasons. 
First, it has this:

     # necessary so that the child nodes get the right source/line set
     content_node.document = directive.state.document

so it is not a pure compatibility hack.  Second, and opposite, currently 
none of the uses of nested_parse_with_titles() go through compat.py's 
version, therefore it probably can be removed altogether.

That leaves only SOURCE_LOCATION_FIX.

As an aside, if the compat.py hacks survive, I would add comments to 
document which distros need the hacks.

> What I'd like to know is: what precisely are our options in this 
> scenario? Do we consider it acceptable for some platforms to be unable 
> to build docs offline?

Certainly for platforms not using the default Python runtime, which 
right now is only SLES.  For others...

> How highly do we value the ability to locally 
> build docs for any given release?

... I think I value this a bit higher than Markus, but not really 
because of offline builds.  Rather, keeping the "accepted" key lower 
(i.e. supporting the packaged sphinx on a wide range of distros) makes 
it easier to bump the "installed" key when needed, as in this failure to 
run 5.3.0 under Python 3.13.

This time there was a version that works on both the oldest and newest 
Python that we support, but there may not always be one because sphinx 
is all too happy at dropping support for EOL'd versions of Python.

Paolo

> Before I throw my weight behind any given option, I just want to know 
> what we consider our non-negotiable obligations to be.
> 
> Thanks,
> --js
> 

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 6/5/25 21:35, John Snow wrote:
>> However, if we take as iron-clad our commitment to the build platform promise -- *and* guarantee offline/tarball builds as well -- then Debian 12 (as an example) only offers Sphinx 5.3.0 and not newer unless we allow internet access to fetch Sphinx 6.2.1. This is not a problem for developer workstations at all, but I am unclear on what problems this may cause for tarball releases and downstream offline/isolated/ reproducible builds, if any.
>> In this case, we can (probably) "fix" the issue by continuing to allow older Sphinx while preferring a newer Sphinx version when it is missing, but then we lose the ability to make code cleanups and drop a lot of back-compat crud. If memory serves, there were other issues recently where older versions of Sphinx behaved differently from newer versions, causing intermittent failures that were hard to track down.
>
> The *ideal* solution would be to:
>
> - accept: 4.3.2 or newer, which is what Ubuntu 22.04 has
>
> - install: 6.2.1, which is what supports Python 3.13

I guess this relates to pythondeps.toml line

    sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }

I further guess "accepted" means "reject anything older", and
"installed" means "preferred version".

> This lets all supported distros build documentation if they use the default Python runtime.  It would still require a couple hacks in compat.py: SOURCE_LOCATION_FIX and nested_parse_with_titles().
>
> I am not sure however whether to count the latter, for two reasons. First, it has this:
>
>     # necessary so that the child nodes get the right source/line set
>     content_node.document = directive.state.document
>
> so it is not a pure compatibility hack.  Second, and opposite, currently none of the uses of nested_parse_with_titles() go through compat.py's version, therefore it probably can be removed altogether.
>
> That leaves only SOURCE_LOCATION_FIX.
>
> As an aside, if the compat.py hacks survive, I would add comments to document which distros need the hacks.
>
>> What I'd like to know is: what precisely are our options in this scenario? Do we consider it acceptable for some platforms to be unable to build docs offline?
>
> Certainly for platforms not using the default Python runtime, which right now is only SLES.  For others...
>
>> How highly do we value the ability to locally build docs for any given release?

Purely offline, or not?

> ... I think I value this a bit higher than Markus, but not really because of offline builds.  Rather, keeping the "accepted" key lower (i.e. supporting the packaged sphinx on a wide range of distros) makes it easier to bump the "installed" key when needed, as in this failure to run 5.3.0 under Python 3.13.

Showing my ignorance again...  I don't understand how keeping "accepted"
lower helps.

> This time there was a version that works on both the oldest and newest Python that we support, but there may not always be one because sphinx is all too happy at dropping support for EOL'd versions of Python.

Pretty strong hint we shouldn't try to support EOL'd versions of Python
either.

> Paolo
>
>> Before I throw my weight behind any given option, I just want to know what we consider our non-negotiable obligations to be.
>> Thanks,
>> --js

Re: Build platform guarantees, docs, tests, and snakes in the garden

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 6893 bytes --]

On Tue, Jun 24, 2025 at 2:45 AM Markus Armbruster <armbru@redhat.com> wrote:

> Paolo Bonzini <pbonzini@redhat.com> writes:
>
> > On 6/5/25 21:35, John Snow wrote:
> >> However, if we take as iron-clad our commitment to the build platform
> promise -- *and* guarantee offline/tarball builds as well -- then Debian 12
> (as an example) only offers Sphinx 5.3.0 and not newer unless we allow
> internet access to fetch Sphinx 6.2.1. This is not a problem for developer
> workstations at all, but I am unclear on what problems this may cause for
> tarball releases and downstream offline/isolated/ reproducible builds, if
> any.
> >> In this case, we can (probably) "fix" the issue by continuing to allow
> older Sphinx while preferring a newer Sphinx version when it is missing,
> but then we lose the ability to make code cleanups and drop a lot of
> back-compat crud. If memory serves, there were other issues recently where
> older versions of Sphinx behaved differently from newer versions, causing
> intermittent failures that were hard to track down.
> >
> > The *ideal* solution would be to:
> >
> > - accept: 4.3.2 or newer, which is what Ubuntu 22.04 has
> >
> > - install: 6.2.1, which is what supports Python 3.13
>
> I guess this relates to pythondeps.toml line
>
>     sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary =
> "sphinx-build" }
>
> I further guess "accepted" means "reject anything older", and
> "installed" means "preferred version".
>

Yes. Accepted is what we'll tolerate if it's already installed, "installed"
is what we will prefer to install.


>
> > This lets all supported distros build documentation if they use the
> default Python runtime.  It would still require a couple hacks in
> compat.py: SOURCE_LOCATION_FIX and nested_parse_with_titles().
> >
> > I am not sure however whether to count the latter, for two reasons.
> First, it has this:
> >
> >     # necessary so that the child nodes get the right source/line set
> >     content_node.document = directive.state.document
> >
> > so it is not a pure compatibility hack.  Second, and opposite, currently
> none of the uses of nested_parse_with_titles() go through compat.py's
> version, therefore it probably can be removed altogether.
> >
> > That leaves only SOURCE_LOCATION_FIX.
> >
> > As an aside, if the compat.py hacks survive, I would add comments to
> document which distros need the hacks.
> >
> >> What I'd like to know is: what precisely are our options in this
> scenario? Do we consider it acceptable for some platforms to be unable to
> build docs offline?
> >
> > Certainly for platforms not using the default Python runtime, which
> right now is only SLES.  For others...
> >
> >> How highly do we value the ability to locally build docs for any given
> release?
>
> Purely offline, or not?


Purely offline is my concern as it is the requirement I least understand.

For developer builds from the git tree, it doesn't matter: you have PyPI,
and we have mkvenv.py to set up an environment we are confident will work
for the purposes of building QEMU and running tests.

For package builds in foreign lands, I grow less certain ...

I know Fedora and RHEL et al build in a strictly isolated environment where
we *cannot* pull PyPI packages. For Fedora this is generally less of a
problem as the distro repository usually has bleeding edge packages for us
to use. mkvenv.py still runs in RPM builds, it just happens to not need
anything it doesn't already have, so it succeeds.

For RHEL it's slightly less of a concern as the version of QEMU packaged
tends to also be old, so the dependencies needed by that version should be
matched well by what's available in that distro repo.

Where I am less certain is for various "streams" or "modules" or whatever
for e.g. RHEL that may package newer versions of QEMU but may not have
other bleeding edge requirements. I am also entirely unfamiliar with the
release cadence of Debian, Ubuntu, OpenSUSE, etc. I am concerned
(hypothetically) about cases where Python3.9 is available as an optional
package, but newer versions of e.g. Sphinx are not.

Is that a problem? Do we care?

In theory, it could affect not just documentation building but unit tests
as well, depending on where and how we "break" our promise.

I suspect I won't really be able to figure out if it's an issue or not
until I just "yeehaw!" and do it and see who chirps, but I don't actually
like operating in such a cavalier manner as a maintainer ...


>
> > ... I think I value this a bit higher than Markus, but not really
> because of offline builds.  Rather, keeping the "accepted" key lower (i.e.
> supporting the packaged sphinx on a wide range of distros) makes it easier
> to bump the "installed" key when needed, as in this failure to run 5.3.0
> under Python 3.13.
>
> Showing my ignorance again...  I don't understand how keeping "accepted"
> lower helps.
>
> > This time there was a version that works on both the oldest and newest
> Python that we support, but there may not always be one because sphinx is
> all too happy at dropping support for EOL'd versions of Python.
>
> Pretty strong hint we shouldn't try to support EOL'd versions of Python
> either.
>

This is the problem I keep running into, generally. Our build platform
promise generally includes platforms that use EOL versions of Python by
default - which is not a problem in and of itself, however when combined
with our support for bleeding edge platforms, it becomes difficult to
support both an EOL'd version of Python *and* the latest releases, as they
are quite aggressive in dropping support for deprecated features and
libraries on the newer end.

Add to that the difficulty of keeping the linters and type checkers happy
across such a wide matrix, and it becomes burdensome to keep things
humming. In general my impression is that linters, doc systems, type
systems etc are designed to be run under strictly pinned versions, not
"whatever the user happens to have available", which is at stark odds with
how we manage our build, doc and test system environment today.

I do not know what the "solution" to that contrast is, but I do get the "we
are doing something wrong" feeling a lot when managing our stack. I am
proud it works as well as it does and across such a diverse host of
platforms, but I get the sense we're one of the few or only projects out
there doing it like this.

I would be as content as anyone else if Python didn't have such an
aggressive deprecation window and didn't *frequently* break the packaging
environment.


>
> > Paolo
> >
> >> Before I throw my weight behind any given option, I just want to know
> what we consider our non-negotiable obligations to be.
> >> Thanks,
> >> --js
>
>

[-- Attachment #2: Type: text/html, Size: 8410 bytes --]

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Jun 24, 2025 at 2:45 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> Paolo Bonzini <pbonzini@redhat.com> writes:
>>
>> > On 6/5/25 21:35, John Snow wrote:
>> >> However, if we take as iron-clad our commitment to the build platform
>> promise -- *and* guarantee offline/tarball builds as well -- then Debian 12
>> (as an example) only offers Sphinx 5.3.0 and not newer unless we allow
>> internet access to fetch Sphinx 6.2.1. This is not a problem for developer
>> workstations at all, but I am unclear on what problems this may cause for
>> tarball releases and downstream offline/isolated/ reproducible builds, if
>> any.
>> >> In this case, we can (probably) "fix" the issue by continuing to allow
>> older Sphinx while preferring a newer Sphinx version when it is missing,
>> but then we lose the ability to make code cleanups and drop a lot of
>> back-compat crud. If memory serves, there were other issues recently where
>> older versions of Sphinx behaved differently from newer versions, causing
>> intermittent failures that were hard to track down.
>> >
>> > The *ideal* solution would be to:
>> >
>> > - accept: 4.3.2 or newer, which is what Ubuntu 22.04 has
>> >
>> > - install: 6.2.1, which is what supports Python 3.13
>>
>> I guess this relates to pythondeps.toml line
>>
>>     sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary =
>> "sphinx-build" }
>>
>> I further guess "accepted" means "reject anything older", and
>> "installed" means "preferred version".
>
> Yes. Accepted is what we'll tolerate if it's already installed, "installed"
> is what we will prefer to install.

Thanks!

>> > This lets all supported distros build documentation if they use the
>> > default Python runtime.  It would still require a couple hacks in
>> > compat.py: SOURCE_LOCATION_FIX and nested_parse_with_titles().
>> >
>> > I am not sure however whether to count the latter, for two reasons.
>> > First, it has this:
>> >
>> >     # necessary so that the child nodes get the right source/line set
>> >     content_node.document = directive.state.document
>> >
>> > so it is not a pure compatibility hack.  Second, and opposite, currently
>> > none of the uses of nested_parse_with_titles() go through compat.py's
>> > version, therefore it probably can be removed altogether.
>> >
>> > That leaves only SOURCE_LOCATION_FIX.
>> >
>> > As an aside, if the compat.py hacks survive, I would add comments to
>> > document which distros need the hacks.
>> >
>> >> What I'd like to know is: what precisely are our options in this
>> >> scenario? Do we consider it acceptable for some platforms to be unable to
>> >> build docs offline?
>> >
>> > Certainly for platforms not using the default Python runtime, which
>> > right now is only SLES.  For others...
>> >
>> >> How highly do we value the ability to locally build docs for any given
>> >> release?
>>
>> Purely offline, or not?
>
>
> Purely offline is my concern as it is the requirement I least understand.
>
> For developer builds from the git tree, it doesn't matter: you have PyPI,
> and we have mkvenv.py to set up an environment we are confident will work
> for the purposes of building QEMU and running tests.

Yes, things should work regardless of what we choose for "accepted" and
"installed": if what the host got is "accepted", we use it, else we put
"installed" in the environment.

> For package builds in foreign lands, I grow less certain ...
>
> I know Fedora and RHEL et al build in a strictly isolated environment where
> we *cannot* pull PyPI packages.

This makes sense.

>                                 For Fedora this is generally less of a
> problem as the distro repository usually has bleeding edge packages for us
> to use. mkvenv.py still runs in RPM builds, it just happens to not need
> anything it doesn't already have, so it succeeds.
>
> For RHEL it's slightly less of a concern as the version of QEMU packaged
> tends to also be old, so the dependencies needed by that version should be
> matched well by what's available in that distro repo.
>
> Where I am less certain is for various "streams" or "modules" or whatever
> for e.g. RHEL that may package newer versions of QEMU but may not have
> other bleeding edge requirements. I am also entirely unfamiliar with the
> release cadence of Debian, Ubuntu, OpenSUSE, etc. I am concerned
> (hypothetically) about cases where Python3.9 is available as an optional
> package, but newer versions of e.g. Sphinx are not.
>
> Is that a problem? Do we care?

If you want to package $software, you get to package its dependencies.
Simple as that.

Widening the version range of our dependencies can make this a bit
easier.

For a distribution's .0, current stable versions get packaged.  We
better make this as easy as we can.  So the version range needs to
include "current stable".  But why wouldn't it include that ever?

Later on, a wider version range can enable "uneven" upgrades: upgrading
to a new QEMU without also upgrading certain dependencies.

How valuable is this for us?  The answer should inform our choice of
version range.  Extending the range further back can be costly, and it
better be worth it.

I'm rather skeptical here.  Feels like a lot of angst about what
distributions might possibly want to do, with precious little evidence
for what they actually do.  Show me distribution package upgrades
(uneven or not) we enabled by keeping our version range wide, and I'll
reconsider.

> In theory, it could affect not just documentation building but unit tests
> as well, depending on where and how we "break" our promise.
>
> I suspect I won't really be able to figure out if it's an issue or not
> until I just "yeehaw!" and do it and see who chirps, but I don't actually
> like operating in such a cavalier manner as a maintainer ...
>
>
>>
>> > ... I think I value this a bit higher than Markus, but not really
>> > because of offline builds.  Rather, keeping the "accepted" key lower (i.e.
>> > supporting the packaged sphinx on a wide range of distros) makes it easier
>> > to bump the "installed" key when needed, as in this failure to run 5.3.0
>> > under Python 3.13.
>>
>> Showing my ignorance again...  I don't understand how keeping "accepted"
>> lower helps.
>>
>> > This time there was a version that works on both the oldest and newest
>> > Python that we support, but there may not always be one because sphinx is
>> > all too happy at dropping support for EOL'd versions of Python.
>>
>> Pretty strong hint we shouldn't try to support EOL'd versions of Python
>> either.
>>
>
> This is the problem I keep running into, generally. Our build platform
> promise generally includes platforms that use EOL versions of Python by
> default - which is not a problem in and of itself, however when combined
> with our support for bleeding edge platforms, it becomes difficult to
> support both an EOL'd version of Python *and* the latest releases, as they
> are quite aggressive in dropping support for deprecated features and
> libraries on the newer end.
>
> Add to that the difficulty of keeping the linters and type checkers happy
> across such a wide matrix, and it becomes burdensome to keep things
> humming. In general my impression is that linters, doc systems, type
> systems etc are designed to be run under strictly pinned versions, not
> "whatever the user happens to have available", which is at stark odds with
> how we manage our build, doc and test system environment today.

For development aids like linters and type checkers, graceful
degradation can make sense.  They must work for developers (who we can
expect to use a reasonably current development environment).  I can't
see why they wouldn't work in a similarly current packaging environment
(i.e. offline), and that's nice.  But I'd loathe to invest much in
keeping development aids work in a non-current environment, especially
not offline.

> I do not know what the "solution" to that contrast is, but I do get the "we
> are doing something wrong" feeling a lot when managing our stack. I am
> proud it works as well as it does and across such a diverse host of
> platforms, but I get the sense we're one of the few or only projects out
> there doing it like this.

Bob is driving on the highway listening to the radio.  Suddenly the
music stops for an announcement: there's a wrong-way driver on the
highway!  "One?", Bob exclaims, "*hundreds* of them!"

When nobody else does what we try to do, we might be pioneers, or we
might be fools.

> I would be as content as anyone else if Python didn't have such an
> aggressive deprecation window and didn't *frequently* break the packaging
> environment.

We got to play the hand we're dealt.

>> > Paolo
>> >
>> >> Before I throw my weight behind any given option, I just want to know
>> >> what we consider our non-negotiable obligations to be.
>> >>
>> >> Thanks,
>> >> --js

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2304 bytes --]

Il mar 24 giu 2025, 02:45 Markus Armbruster <armbru@redhat.com> ha scritto:

> > ... I think I value this a bit higher than Markus, but not really
> because of offline builds.  Rather, keeping the "accepted" key lower (i.e.
> supporting the packaged sphinx on a wide range of distros) makes it easier
> to bump the "installed" key when needed, as in this failure to run 5.3.0
> under Python 3.13.
>
> Showing my ignorance again...  I don't understand how keeping "accepted"
> lower helps.
>

Because it makes it easier to use distro Python. If distro Python is
<accepted, configure's will try to use the "installed" version. If that
version in turn is too new for distro Python, you're screwed. So you want
to be as conservative as needed for accepted, but not more.

Regarding fool or pioneer: for sure we're extraordinarily kind towards
distros. To some extent we have to do that because of 1) the possible
competition of other VMMs that completely ignore distros (e.g. because they
just use cargo)—packaging is an area where C still has an edge and we want
to keep that edge 2) we're an infrastructure component that can't just tell
users to grab a flatpak.

The distro policy (mostly conceived by Dan) has served us well, with only
small adjustments needed to have newish version of Meson/Rust(*), and
non-prehistoric versions of Python. I don't see a need to change it, since
at this point we have the tools needed to manage the complexity.

Paolo

(*) Most of the Rust issues would solve themselves by telling users of
Ubuntu 22.04 and Debian bookworm to install the upstream tool chain with
rustup instead of relying on distro rustc packages. Unlike Linux, which
uses unstable features, QEMU sticks to what's been stabilized and that
means newer releases sometimes.

> This time there was a version that works on both the oldest and newest
> Python that we support, but there may not always be one because sphinx is
> all too happy at dropping support for EOL'd versions of Python.
>
> Pretty strong hint we shouldn't try to support EOL'd versions of Python
> either.
>
> > Paolo
> >
> >> Before I throw my weight behind any given option, I just want to know
> what we consider our non-negotiable obligations to be.
> >> Thanks,
> >> --js
>
>

[-- Attachment #2: Type: text/html, Size: 3185 bytes --]

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2521 bytes --]

Il mer 2 lug 2025, 15:24 Paolo Bonzini <pbonzini@redhat.com> ha scritto:

>
>
> Il mar 24 giu 2025, 02:45 Markus Armbruster <armbru@redhat.com> ha
> scritto:
>
>> > ... I think I value this a bit higher than Markus, but not really
>> because of offline builds.  Rather, keeping the "accepted" key lower (i.e.
>> supporting the packaged sphinx on a wide range of distros) makes it easier
>> to bump the "installed" key when needed, as in this failure to run 5.3.0
>> under Python 3.13.
>>
>> Showing my ignorance again...  I don't understand how keeping "accepted"
>> lower helps.
>>
>
> Because it makes it easier to use distro Python. If distro Python is
> <accepted,
>

Sorry: if distro *sphinx* is <accepted.

Paolo

configure's will try to use the "installed" version. If that version in
> turn is too new for distro Python, you're screwed. So you want to be as
> conservative as needed for accepted, but not more.
>
> Regarding fool or pioneer: for sure we're extraordinarily kind towards
> distros. To some extent we have to do that because of 1) the possible
> competition of other VMMs that completely ignore distros (e.g. because they
> just use cargo)—packaging is an area where C still has an edge and we want
> to keep that edge 2) we're an infrastructure component that can't just tell
> users to grab a flatpak.
>
> The distro policy (mostly conceived by Dan) has served us well, with only
> small adjustments needed to have newish version of Meson/Rust(*), and
> non-prehistoric versions of Python. I don't see a need to change it, since
> at this point we have the tools needed to manage the complexity.
>
> Paolo
>
> (*) Most of the Rust issues would solve themselves by telling users of
> Ubuntu 22.04 and Debian bookworm to install the upstream tool chain with
> rustup instead of relying on distro rustc packages. Unlike Linux, which
> uses unstable features, QEMU sticks to what's been stabilized and that
> means newer releases sometimes.
>
> > This time there was a version that works on both the oldest and newest
>> Python that we support, but there may not always be one because sphinx is
>> all too happy at dropping support for EOL'd versions of Python.
>>
>> Pretty strong hint we shouldn't try to support EOL'd versions of Python
>> either.
>>
>> > Paolo
>> >
>> >> Before I throw my weight behind any given option, I just want to know
>> what we consider our non-negotiable obligations to be.
>> >> Thanks,
>> >> --js
>>
>>

[-- Attachment #2: Type: text/html, Size: 3989 bytes --]

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Il mer 2 lug 2025, 15:24 Paolo Bonzini <pbonzini@redhat.com> ha scritto:
>
>>
>>
>> Il mar 24 giu 2025, 02:45 Markus Armbruster <armbru@redhat.com> ha
>> scritto:
>>
>>> > ... I think I value this a bit higher than Markus, but not really
>>> > because of offline builds.  Rather, keeping the "accepted" key lower (i.e.
>>> > supporting the packaged sphinx on a wide range of distros) makes it easier
>>> > to bump the "installed" key when needed, as in this failure to run 5.3.0
>>> > under Python 3.13.
>>>
>>> Showing my ignorance again...  I don't understand how keeping "accepted"
>>> lower helps.
>>>
>>
>> Because it makes it easier to use distro Python. If distro Python is
>> <accepted,
>>
>
> Sorry: if distro *sphinx* is <accepted.
>
> Paolo
>
>> configure's will try to use the "installed" version. If that version in
>> turn is too new for distro Python, you're screwed. So you want to be as
>> conservative as needed for accepted, but not more.

So, we get into trouble when accept the distro versions for some, but
not all depenencies, and end up with a mix of "accepted" and "installed"
versions that doesn't play together.

Ways to avoid this scenario:

* Ensure the "installed" version play with all the "accepted" versions
  of everything else.  I.e. as we move "installed" up, "accepted" needs
  to move up as well.

* We keep "accepted" for everything low enough so that we don't fall
  back to "installed" on any distro we care about.  Feels brittle.

Any others?

>> Regarding fool or pioneer: for sure we're extraordinarily kind towards
>> distros. To some extent we have to do that because of 1) the possible
>> competition of other VMMs that completely ignore distros (e.g. because they
>> just use cargo)—packaging is an area where C still has an edge and we want
>> to keep that edge 2) we're an infrastructure component that can't just tell
>> users to grab a flatpak.
>>
>> The distro policy (mostly conceived by Dan) has served us well, with only
>> small adjustments needed to have newish version of Meson/Rust(*), and
>> non-prehistoric versions of Python. I don't see a need to change it, since
>> at this point we have the tools needed to manage the complexity.

One of the "tools" being John Snow, I'm afraid.  Making the full Sphinx
version range work was impressive (and expensive!) work.  I take a
rather dim view on this kind of expensive heroics.  If John gets run
over by a bus, our hand may well be forced: I can't maintain the result
of his heroism, that's for sure.

>> Paolo
>>
>> (*) Most of the Rust issues would solve themselves by telling users of
>> Ubuntu 22.04 and Debian bookworm to install the upstream tool chain with
>> rustup instead of relying on distro rustc packages. Unlike Linux, which
>> uses unstable features, QEMU sticks to what's been stabilized and that
>> means newer releases sometimes.
>>
>> > This time there was a version that works on both the oldest and newest
>>> Python that we support, but there may not always be one because sphinx is
>>> all too happy at dropping support for EOL'd versions of Python.
>>>
>>> Pretty strong hint we shouldn't try to support EOL'd versions of Python
>>> either.
>>>
>>> > Paolo
>>> >
>>> >> Before I throw my weight behind any given option, I just want to know
>>> >> what we consider our non-negotiable obligations to be.
>>> >>
>>> >> Thanks,
>>> >> --js

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Daniel P. Berrangé]

On Wed, Jul 02, 2025 at 03:24:09PM -0400, Paolo Bonzini wrote:
> Il mar 24 giu 2025, 02:45 Markus Armbruster <armbru@redhat.com> ha scritto:
> 
> > > ... I think I value this a bit higher than Markus, but not really
> > because of offline builds.  Rather, keeping the "accepted" key lower (i.e.
> > supporting the packaged sphinx on a wide range of distros) makes it easier
> > to bump the "installed" key when needed, as in this failure to run 5.3.0
> > under Python 3.13.
> >
> > Showing my ignorance again...  I don't understand how keeping "accepted"
> > lower helps.
> >
> 
> Because it makes it easier to use distro Python. If distro Python is
> <accepted, configure's will try to use the "installed" version. If that
> version in turn is too new for distro Python, you're screwed. So you want
> to be as conservative as needed for accepted, but not more.
> 
> Regarding fool or pioneer: for sure we're extraordinarily kind towards
> distros. To some extent we have to do that because of 1) the possible
> competition of other VMMs that completely ignore distros (e.g. because they
> just use cargo)—packaging is an area where C still has an edge and we want
> to keep that edge 2) we're an infrastructure component that can't just tell
> users to grab a flatpak.
> 
> The distro policy (mostly conceived by Dan) has served us well, with only
> small adjustments needed to have newish version of Meson/Rust(*), and
> non-prehistoric versions of Python. I don't see a need to change it, since
> at this point we have the tools needed to manage the complexity.

Note that much of the commentary about distros versions has been in
relation to the distro packagers, but that was not my only target
in writing the distro policy. It was equally aimed at contributors
using such distros, as well as 3rd party vendors building solutions
on top of designated distro versions

You can say contributors should just pick newer containers for their
build env, or manually download newer deps, or have QEMU build fancy
scripts to auto-download newer deps. All of those options have a cost
to them, as compared to using what is already present in the distro.

In terms of 3rd party vendors, they can have similar roles to a distro
vendor, but are more likely to package up newer QEMU versions to run
on pre-existing distros.

A further goal of the support policy was to provide a mechanism to
eliminate exactly these kind of mail threads. Before we had the policy,
every single time someone wanted to bump the min version of any dep
we would have debates over whether it was OK or not, there was always
someone who wanted the old version of the distro forever.  Defining
the policy has allowed us to unconditionally bump the min versions of
our times on a usually reasonable timeframe, without needing to engage
in debate. We can just point people to our support policy when they
complained that they really wanted old versions X, Y, & Z.

Every time we make an exception to the policy, we undermine the benefits
we obtain from it, taking us back the old world where our min versions
were an inconsistent & arbitrary set, with little clear understanding
of when we would change, either by maintainers or users.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Build platform guarantees, docs, tests, and snakes in the garden

[John Snow]

On Mon, Jul 7, 2025 at 5:11 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Wed, Jul 02, 2025 at 03:24:09PM -0400, Paolo Bonzini wrote:
> > Il mar 24 giu 2025, 02:45 Markus Armbruster <armbru@redhat.com> ha scritto:
> >
> > > > ... I think I value this a bit higher than Markus, but not really
> > > because of offline builds.  Rather, keeping the "accepted" key lower (i.e.
> > > supporting the packaged sphinx on a wide range of distros) makes it easier
> > > to bump the "installed" key when needed, as in this failure to run 5.3.0
> > > under Python 3.13.
> > >
> > > Showing my ignorance again...  I don't understand how keeping "accepted"
> > > lower helps.
> > >
> >
> > Because it makes it easier to use distro Python. If distro Python is
> > <accepted, configure's will try to use the "installed" version. If that
> > version in turn is too new for distro Python, you're screwed. So you want
> > to be as conservative as needed for accepted, but not more.
> >
> > Regarding fool or pioneer: for sure we're extraordinarily kind towards
> > distros. To some extent we have to do that because of 1) the possible
> > competition of other VMMs that completely ignore distros (e.g. because they
> > just use cargo)—packaging is an area where C still has an edge and we want
> > to keep that edge 2) we're an infrastructure component that can't just tell
> > users to grab a flatpak.
> >
> > The distro policy (mostly conceived by Dan) has served us well, with only
> > small adjustments needed to have newish version of Meson/Rust(*), and
> > non-prehistoric versions of Python. I don't see a need to change it, since
> > at this point we have the tools needed to manage the complexity.
>
> Note that much of the commentary about distros versions has been in
> relation to the distro packagers, but that was not my only target
> in writing the distro policy. It was equally aimed at contributors
> using such distros, as well as 3rd party vendors building solutions
> on top of designated distro versions

It's not that I ignore such cases, it's just that distro packagers
have the more difficult to accommodate use case, so I tend to focus in
on that when discussing pros/cons of various policies. If I
accommodate that use case, I accommodate everyone else necessarily, I
believe. It's just a heuristic and not a truth, but so far a useful
one.

>
> You can say contributors should just pick newer containers for their
> build env, or manually download newer deps, or have QEMU build fancy
> scripts to auto-download newer deps. All of those options have a cost
> to them, as compared to using what is already present in the distro.

You are right. However, the mkvenv configuration tool we pioneered has
been largely un-noticed by contributors and appears to "just work" for
the last several years. I believe that cost has been *largely*
amortized by yours truly, and I have spared almost every other
contributor from paying it.

The fancy scripts we have do in fact bend over backward to use
distro-standard packages whenever at all possible, falling back to
vendored packages and only reaching out to the internet when it
*absolutely has to*. I think this has served us perfectly well and
quite nearly invisibly (to everyone else except me, anyway.)

So, more or less: your concern is something I share, but I think I
have it satisfactorily addressed - hence my seeming overfocus on
distro packagers.

>
> In terms of 3rd party vendors, they can have similar roles to a distro
> vendor, but are more likely to package up newer QEMU versions to run
> on pre-existing distros.

Yes. I think they are also usually more willing and able to bend the
rules of the base platform, though. I don't want to break platform
support for things like documentation and tests *needlessly*, but the
frequent changes to Python packaging subsystems do rather often
present a difficulty when it comes to supporting both "EOL" versions
of Python while simultaneously supporting the bleeding edge.

I seek to develop and codify a suitable compromise for these
situations as they continue to arise and, in all likelihood, will not
stop cropping up once per year or so.

As Markus puts it: If a distro (or a third party vendor of a distro)
wants to backport bleeding edge packages to a stable platform, they
must also be willing at times to backport build dependencies. I think
that's a reasonable view, though not license to go wild ignoring older
platforms. In my case, it's only Python packages from over five years
ago that present a difficulty - which is not exactly bleeding edge
stuff.

(Of course, my personal wish is that the Python ecosystem wasn't quite
so aggressive about deprecating things, but I just simply do not
control that, nor the weather...)

>
> A further goal of the support policy was to provide a mechanism to
> eliminate exactly these kind of mail threads. Before we had the policy,
> every single time someone wanted to bump the min version of any dep
> we would have debates over whether it was OK or not, there was always
> someone who wanted the old version of the distro forever.  Defining
> the policy has allowed us to unconditionally bump the min versions of
> our times on a usually reasonable timeframe, without needing to engage
> in debate. We can just point people to our support policy when they
> complained that they really wanted old versions X, Y, & Z.

I agree, that is indeed the nice thing about the policy.

... And it's why I would like to codify our behavior around the
exemptions we make for e.g. requiring newer Python on platforms like
OpenSUSE Leap 15.6, and quite possibly soon, CentOS Stream 9. It will
eliminate the need to have this discussion every time.

>
> Every time we make an exception to the policy, we undermine the benefits
> we obtain from it, taking us back the old world where our min versions
> were an inconsistent & arbitrary set, with little clear understanding
> of when we would change, either by maintainers or users.

I agree again!

I would like to codify something like this for our support policy:

"On otherwise supported build platforms, QEMU *may* require a Python
interpreter that is considered actively maintained, which is usually a
version released within the last five years. When platforms that ship,
by default, an EOL Python interpreter also offer an optional package
for a newer, actively maintained Python interpreter, QEMU may require
this repository package for configuring and building QEMU."

(Simpler language: I am trying to say that Platforms like OpenSUSE
that have an ancient Python by default but also ship newer optional
versions may require one of those newer, optional versions to build
QEMU. On platforms that do NOT offer a newer Python version, I am
suggesting that I will be shit-out-of-luck. I think this is a pretty
mild compromise, all told.)

"On these platforms, unit tests and documentation may possibly require
non-distribution packaged versions of Python dependencies such as
Sphinx in order to run using the more modern Python interpreter."

(Simpler language: Platforms like OpenSUSE do not package Sphinx for
those newer, optional Python interpreters and the user will either
need to allow access to PyPI for configure to load what it needs just
in time, or make preparations to load those packages in advance. This
is the most unfortunate truth of our workaround for OpenSUSE, but I am
not aware of any way to mitigate it. The core build will still work,
but we may indeed lose access to documentation and unit tests from
time to time using only the platform repositories in these situations.
I think this is a graceful degradation, but it is one I fight to avoid
at all costs. It seems to be a necessary evil at times.

OpenSUSE Leap has a Python that is too old to reasonably support, but
they offer an optional modern version that can be loaded instead. In
that event, it is possible the user may need PyPI packages to run
tests and build documentation. To my knowledge, OpenSUSE Leap 15.6 is
currently our only supported platform where this exception need occur,
but it is very possible that a similar exception may arise for CentOS
Stream 9/RHEL9 in the not-so-distant future when Python 3.9 is EOL and
other packages start aggressively dropping support for it.)

I'd like to re-iterate that my motivation here is not to stop
supporting EOL python versions "just because they are EOL", but rather
instead because the tooling and libraries surrounding Python
aggressively drop support for those versions once they become EOL.
That is, 3.9 is not difficult to support until e.g. Python 3.14 comes
out and setuptools, pip, and sphinx begin targeting 3.14 at the
expense of 3.9 and there is an increased burden of hacks and
workarounds required to target 3.9-3.14 inclusive.

As it stands, I have no plans or motivation to discontinue support for
3.9, but based on prior EOL events (3.6, 3.7, and 3.8) there is
inevitably some incompatibility for which dropping support is usually
the easiest path forward. I am just thinking ahead, currently.

Basically, I would like to promise I will "do my best" to support the
"base platforms as-is", but recognize that an exception for Python is
occasionally required, and, to prevent future discussions like this
(essentially yearly, now) would like to codify the precise nature of
that exemption and what degradation we consider an acceptable loss in
those cases. Namely, core build support is easier to maintain using
only distro packages, but unit testing and documentation are
occasionally a bit more difficult.

I would love to adhere to the policy as written as immutable law, but
it has created genuine friction and non-trivial cost on my end in the
past - I only want to convey that "I think it's a good policy!" and "I
have tried to follow it as gospel!" but raise the issue that it's
quite hard to do in some cases and we do need exemptions for things
like OpenSUSE that are communicated clearly.



>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: Build platform guarantees, docs, tests, and snakes in the garden

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 4411 bytes --]

Il mer 9 lug 2025, 20:39 John Snow <jsnow@redhat.com> ha scritto:

> You are right. However, the mkvenv configuration tool we pioneered has
> been largely un-noticed by contributors and appears to "just work" for
> the last several years. I believe that cost has been *largely*
> amortized by yours truly, and I have spared almost every other
> contributor from paying it.
>

Especially for sphinx.

mkvenv/pythondeps.toml has been very stable, perhaps beyond expectations,
so at least it was a one time cost.

So, more or less: your concern is something I share, but I think I
> have it satisfactorily addressed - hence my seeming overfocus on
> distro packagers.
>

I agree, and I should thank you for hearing me out on the somewhat crazy
idea that's mkvenv.

> In terms of 3rd party vendors, they can have similar roles to a distro
> > vendor, but are more likely to package up newer QEMU versions to run
> > on pre-existing distros.
>
> Yes. I think they are also usually more willing and able to bend the
> rules of the base platform, though.


Willing, I am not sure. Able yes, which is what matters even if they're not
really willing. :)

And they probably care much less about docs to be honest.

I seek to develop and codify a suitable compromise for these
> situations as they continue to arise and, in all likelihood, will not
> stop cropping up once per year or so. [...]  In my case, it's only Python
> packages from over five years

ago that present a difficulty - which is not exactly bleeding edge
> stuff.
>
> I would like to codify something like this for our support policy:
>
> "On otherwise supported build platforms, QEMU *may* require a Python
> interpreter that is considered actively maintained, which is usually a
> version released within the last five years. When platforms that ship,
> by default, an EOL Python interpreter also offer an optional package
> for a newer, actively maintained Python interpreter, QEMU may require
> this repository package for configuring and building QEMU."
>

That's basically an extension and clarification of what already exists at
https://www.qemu.org/docs/master/about/build-platforms.html ("Python
runtime").

Piggybacking on Python EOL as a necessary but not sufficient condition is
at the same time conservative and useful.

(Simpler language: I am trying to say that Platforms like OpenSUSE
> that have an ancient Python by default but also ship newer optional
> versions may require one of those newer, optional versions to build
> QEMU


OpenSUSE is not a problem since their base Python is not even supported by
QEMU. CentOS Stream is the more sticky one, unless you have a good reason
to require 3.10 (personally I don't, other than nicer type annotations I
have no love for e.g. match expressions).

On platforms that do NOT offer a newer Python version, I am
> suggesting that I will be shit-out-of-luck. I think this is a pretty
> mild compromise, all told.)
>

Yeah, their problem. Both Red Hat and SUSE have figured it out.

"On these platforms, unit tests and documentation may possibly require
> non-distribution packaged versions of Python dependencies such as
> Sphinx in order to run using the more modern Python interpreter."
>

That's the cop out for Red Hat, where you accept 3.9 for building and
require 3.10 for docs and tests? Not sure about it, especially tests. I
would either restrict the limitation to docs, or just declare 3.10 the
minimum once 3.9 is EOL (say January 2026).

I'd like to re-iterate that my motivation here is not to stop
> supporting EOL python versions "just because they are EOL", but rather
> instead because the tooling and libraries surrounding Python
> aggressively drop support for those versions once they become EOL.
> That is, 3.9 is not difficult to support until e.g. Python 3.14 comes
> out and setuptools, pip, and sphinx begin targeting 3.14 at the
> expense of 3.9 and there is an increased burden of hacks and
> workarounds required to target 3.9-3.14 inclusive.
>

Yes, this is the real problem. And it's better to have a plan ahead of time.

(As an aside, Meson is the good guy here. It supports old versions of
Python just fine The only reason to push Meson versions past 1.2 was to
support Rust-adjacent features and bugfixes that especially after 1.5 we're
mostly written by yours truly, specifically to evolve Rust support in the
direction that QEMU is interested in).

Paolo

[-- Attachment #2: Type: text/html, Size: 7452 bytes --]

Re: Build platform guarantees, docs, tests, and snakes in the garden

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 7225 bytes --]

On Wed, Jul 9, 2025, 3:40 PM Paolo Bonzini <pbonzini@redhat.com> wrote:

>
>
> Il mer 9 lug 2025, 20:39 John Snow <jsnow@redhat.com> ha scritto:
>
>> You are right. However, the mkvenv configuration tool we pioneered has
>> been largely un-noticed by contributors and appears to "just work" for
>> the last several years. I believe that cost has been *largely*
>> amortized by yours truly, and I have spared almost every other
>> contributor from paying it.
>>
>
> Especially for sphinx.
>
> mkvenv/pythondeps.toml has been very stable, perhaps beyond expectations,
> so at least it was a one time cost.
>
> So, more or less: your concern is something I share, but I think I
>> have it satisfactorily addressed - hence my seeming overfocus on
>> distro packagers.
>>
>
> I agree, and I should thank you for hearing me out on the somewhat crazy
> idea that's mkvenv.
>

I knew it was possible, I was not so sure it'd be as stable as it has been.
What a delight!


> > In terms of 3rd party vendors, they can have similar roles to a distro
>> > vendor, but are more likely to package up newer QEMU versions to run
>> > on pre-existing distros.
>>
>> Yes. I think they are also usually more willing and able to bend the
>> rules of the base platform, though.
>
>
> Willing, I am not sure. Able yes, which is what matters even if they're
> not really willing. :)
>
> And they probably care much less about docs to be honest.
>
> I seek to develop and codify a suitable compromise for these
>> situations as they continue to arise and, in all likelihood, will not
>> stop cropping up once per year or so. [...]  In my case, it's only Python
>> packages from over five years
>
> ago that present a difficulty - which is not exactly bleeding edge
>> stuff.
>>
>> I would like to codify something like this for our support policy:
>>
>> "On otherwise supported build platforms, QEMU *may* require a Python
>> interpreter that is considered actively maintained, which is usually a
>> version released within the last five years. When platforms that ship,
>> by default, an EOL Python interpreter also offer an optional package
>> for a newer, actively maintained Python interpreter, QEMU may require
>> this repository package for configuring and building QEMU."
>>
>
> That's basically an extension and clarification of what already exists at
> https://www.qemu.org/docs/master/about/build-platforms.html ("Python
> runtime").
>
> Piggybacking on Python EOL as a necessary but not sufficient condition is
> at the same time conservative and useful.
>
> (Simpler language: I am trying to say that Platforms like OpenSUSE
>> that have an ancient Python by default but also ship newer optional
>> versions may require one of those newer, optional versions to build
>> QEMU
>
>
> OpenSUSE is not a problem since their base Python is not even supported by
> QEMU. CentOS Stream is the more sticky one, unless you have a good reason
> to require 3.10 (personally I don't, other than nicer type annotations I
> have no love for e.g. match expressions).
>

Just an example for a distro we've already bent the rules for. Their base
Python is still technically 3.6 and they don't technically package Sphinx
for the newer interpreter, so it's a useful example case even if it is
arguably beyond our support window now.

>
> On platforms that do NOT offer a newer Python version, I am
>> suggesting that I will be shit-out-of-luck. I think this is a pretty
>> mild compromise, all told.)
>>
>
> Yeah, their problem. Both Red Hat and SUSE have figured it out.
>

I meant to suggest it's *my* problem, but I think distributions are aware
that not providing an (optional) non-EOL Python is not a viable option
these days.

Basically affirming: you will be able to build QEMU on a supported platform
with just your distro's repositories, and that's a promise.


> "On these platforms, unit tests and documentation may possibly require
>> non-distribution packaged versions of Python dependencies such as
>> Sphinx in order to run using the more modern Python interpreter."
>>
>
> That's the cop out for Red Hat, where you accept 3.9 for building and
> require 3.10 for docs and tests? Not sure about it, especially tests. I
> would either restrict the limitation to docs, or just declare 3.10 the
> minimum once 3.9 is EOL (say January 2026).
>

I mean to say that I don't consider loading a newer Python provided by your
distribution as a violation of our platform promise: in many ways it is
"just another dep".

I wouldn't raise the minimum Python version unless affected distributions
had an alternate interpreter in their repository. OpenSUSE and CentOS both
offer this, so the option is viable.

However, this solution does not account for other Python dependencies like
Sphinx, which are very likely *not* repackaged by the distro for the newer
interpreter.

A normal end user can run configure and get the new deps automatically and
invisibly, no problem. Offline/isolated builds are slightly trickier.

So I mean to clarify: QEMU will always be able to be built with your distro
packages (sometimes with newer, alternative Python versions) but in this
specific case, some "optional" features where Python is concerned may
require side-loading Python dependencies from outside of your system
repository.

I.e. there may be degradation, but only for unit tests and docs, and only
when your base platform Python is EOL (five years old. It can be fixed by
using PyPI, allowing internet during configure, disabling tests and docs,
or doing the labor of adding a new package to your builddep environment.

So I pledge the core build will only need the newer Python from your system
repo, but tests and docs I only pledge support for if your Python is not
EOL (approx 5 years old or less.)

That's my proposal.

(Unless... haha. The idea is so stupid I don't even want to write it on
list. Ping me on matrix/IRC...)


> I'd like to re-iterate that my motivation here is not to stop
>> supporting EOL python versions "just because they are EOL", but rather
>> instead because the tooling and libraries surrounding Python
>> aggressively drop support for those versions once they become EOL.
>> That is, 3.9 is not difficult to support until e.g. Python 3.14 comes
>> out and setuptools, pip, and sphinx begin targeting 3.14 at the
>> expense of 3.9 and there is an increased burden of hacks and
>> workarounds required to target 3.9-3.14 inclusive.
>>
>
> Yes, this is the real problem. And it's better to have a plan ahead of
> time.
>
> (As an aside, Meson is the good guy here. It supports old versions of
> Python just fine The only reason to push Meson versions past 1.2 was to
> support Rust-adjacent features and bugfixes that especially after 1.5 we're
> mostly written by yours truly, specifically to evolve Rust support in the
> direction that QEMU is interested in).
>

I love Meson. Pure Python, zero deps. Trivial to vendor. It's a delight and
I'm glad they have such reasonable policies. It's a rare treat that never
causes problems for me <3


> Paolo
>

[-- Attachment #2: Type: text/html, Size: 11982 bytes --]