Minimum sphinx version?

Minimum sphinx version?

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2863 bytes --]

Hi, following up from my KVM forum talk (
https://www.youtube.com/watch?v=GGbz3WwvqLw ) I wanted to ask about what
version of Sphinx we could use as a minimum version to help ease the burden
of modernizing our QMP docs without a long backwards compatibility tail
(and hundreds of lines of compatibility goop to make it happen.)

First, some necessary background on the current state of our Python
environment and build system:

Paolo and I worked on a Python virtual environment tied to the QEMU build
system (python/scripts/mkvenv.py) which performs various setup actions
necessary to instantiate python *stuff* to ensure the build works and is
always using the correct, configured Python interpreter.

Currently, configure will invoke mkvenv in such a fashion that it prefers
any local packages you have installed: notably, meson, sphinx, and
sphinx-rtd-theme if you already have them installed from your distro's
repository will be used *if they use the version of python you specified to
configure* (or they match the version that configure guessed if you didn't
specify one.)

For meson, if you don't meet the minimum version, we will install a version
for you from our vendored whl files in the repository/tarball.

For sphinx, if you don't meet the minimum version and you have specified
--enable-docs, we'll install a version from the internet, from PyPI. If you
did not specify --enable-docs, we disable docs.

We did it this way because:
- It's *really fast* if you already have the dependencies you need
- meson is easy to vendor as a .whl because it's pure python, has *zero*
dependencies, and it's absolutely required for the build.
- docs are not *required* to build QEMU
- We went out of our way to ensure that QEMU could be built in an offline,
isolated environment on all of our supported platforms.

That being said:

The current reality is that Sphinx 3.4.3 is our minimum because RHEL 9
offers that as the distro package and I have not dared bump our version
beyond that for fear of disrupting our ability to build docs on RHEL 9
without internet.

What I'd like to ask is: How adamant are we that we can build docs on older
platforms? Do we consider it part of our platform promise? Can we bump
Sphinx to a slightly newer version at the expense of offline doc builds for
RHEL 9?

To reiterate: interactive developer builds from git or source tarballs
would continue to work (and be able to build docs just fine) so long as
--enable-docs is passed, but will require an internet connection to set up
the python environment with a new-enough sphinx. RHEL 9 rpm builds of
bleeding edge QEMU may be cut off from building docs in isolated RPMBUILD
environments unless that repository backports a newer sphinx that can be
identified as a dependency in the specfile.

How much of a problem do we consider that to be?

CCing the usual suspects.

--js

[-- Attachment #2: Type: text/html, Size: 3393 bytes --]

Re: Minimum sphinx version?

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Hi, following up from my KVM forum talk (
> https://www.youtube.com/watch?v=GGbz3WwvqLw ) I wanted to ask about what
> version of Sphinx we could use as a minimum version to help ease the burden
> of modernizing our QMP docs without a long backwards compatibility tail
> (and hundreds of lines of compatibility goop to make it happen.)

Let me add some perspective...

Current docs/sphinx/qapidoc.py is 370 SLOC.  In my experience, it
resists improvement pretty hard, by Sphinx throwing impenetrable stack
traces at the slightest provocation.

Your replacement, which generated *much* superior rendered documentation
is how much, not counting compatibility goop?

The compatibility goop adds how much?

As far as I know, the replacement proper is nice, maintainable code,
while the goop is, well, goop.  Anything breaks there, we run for the
hills and wait for the John Snow cavalry to save the day.

> First, some necessary background on the current state of our Python
> environment and build system:
>
> Paolo and I worked on a Python virtual environment tied to the QEMU build
> system (python/scripts/mkvenv.py) which performs various setup actions
> necessary to instantiate python *stuff* to ensure the build works and is
> always using the correct, configured Python interpreter.
>
> Currently, configure will invoke mkvenv in such a fashion that it prefers
> any local packages you have installed: notably, meson, sphinx, and
> sphinx-rtd-theme if you already have them installed from your distro's
> repository will be used *if they use the version of python you specified to
> configure* (or they match the version that configure guessed if you didn't
> specify one.)
>
> For meson, if you don't meet the minimum version, we will install a version
> for you from our vendored whl files in the repository/tarball.
>
> For sphinx, if you don't meet the minimum version and you have specified
> --enable-docs, we'll install a version from the internet, from PyPI. If you
> did not specify --enable-docs, we disable docs.

Double-checking I understand what you wrote:

* We support Python 3.8 and later.

  3.8 was released in 2019, upstream full support ended in 2021, and
  security fixes are about to end.

  docs/about/build-platforms.rst claims 3.7.  I'll fix it.

* We support Meson 1.1.0 and later.

  If the system doesn't provide, we fall back to bundled Meson 1.2.3.

* We support Sphinx 3.4.3 or later.

  If the system doesn't provide, we pull one from PyPI with
  --enable-docs, else disable docs.  Which version do we pull?

  There's a check for >=1.7.0 in docs/meson.build.  Could that go?

* We support sphinx-rtd-theme.

  Does it come with Sphinx?

> We did it this way because:
> - It's *really fast* if you already have the dependencies you need
> - meson is easy to vendor as a .whl because it's pure python, has *zero*
> dependencies, and it's absolutely required for the build.
> - docs are not *required* to build QEMU
> - We went out of our way to ensure that QEMU could be built in an offline,
> isolated environment on all of our supported platforms.

Loaded question, but I know you can answer this better than I could: how
common is it to support a wide range of versions for Python tools?

> That being said:
>
> The current reality is that Sphinx 3.4.3 is our minimum because RHEL 9
> offers that as the distro package and I have not dared bump our version
> beyond that for fear of disrupting our ability to build docs on RHEL 9
> without internet.
>
> What I'd like to ask is: How adamant are we that we can build docs on older
> platforms? Do we consider it part of our platform promise? Can we bump
> Sphinx to a slightly newer version at the expense of offline doc builds for
> RHEL 9?
>
> To reiterate: interactive developer builds from git or source tarballs
> would continue to work (and be able to build docs just fine) so long as
> --enable-docs is passed, but will require an internet connection to set up
> the python environment with a new-enough sphinx. RHEL 9 rpm builds of
> bleeding edge QEMU may be cut off from building docs in isolated RPMBUILD
> environments unless that repository backports a newer sphinx that can be
> identified as a dependency in the specfile.
>
> How much of a problem do we consider that to be?
>
> CCing the usual suspects.
>
> --js

Re: Minimum sphinx version?

[Paolo Bonzini]

On 9/30/24 19:16, John Snow wrote:
> The current reality is that Sphinx 3.4.3 is our minimum because RHEL 9 
> offers that as the distro package and I have not dared bump our version 
> beyond that for fear of disrupting our ability to build docs on RHEL 9 
> without internet.
> 
> What I'd like to ask is: How adamant are we that we can build docs on 
> older platforms? Do we consider it part of our platform promise? Can we 
> bump Sphinx to a slightly newer version at the expense of offline doc 
> builds for RHEL 9?

Could we stop building QMP docs, if Sphinx is too old?  While at the 
same time keeping the same 3.x version and the ability to build man 
pages, which is the really important part.

It should be as easy as adding a -D option to SPHINX_ARGS based on 
sphinx_version in docs/meson.build; bonus points for adding a docs-qmp 
feature option, like

enable_docs_qmp = get_option('docs-qmp') \
   .disable_auto_if(sphinx_version.version_compare('<4')) \
   .allowed()
if enable_docs_qmp
   SPHINX_ARGS += ['-Ddocs_qmp=1']
endif

and for adding enable_docs_qmp to the summary.  Unfortunately I don't 
think -D is accessible from rST, so you'd have to check it from the 
extension and put a "QMP documentation not included in this manual" 
placeholder.  Not great, I admit.

The current check on sphinx being 1.7 or newer is obsolute, but I won't 
remove it for now so that you already have the code to compute 
sphinx_version.

Paolo

> To reiterate: interactive developer builds from git or source tarballs 
> would continue to work (and be able to build docs just fine) so long as 
> --enable-docs is passed, but will require an internet connection to set 
> up the python environment with a new-enough sphinx. RHEL 9 rpm builds of 
> bleeding edge QEMU may be cut off from building docs in isolated 
> RPMBUILD environments unless that repository backports a newer sphinx 
> that can be identified as a dependency in the specfile.
> 
> How much of a problem do we consider that to be?
> 
> CCing the usual suspects.
> 
> --js
> 

Re: Minimum sphinx version?

[Daniel P. Berrangé]

On Mon, Sep 30, 2024 at 01:16:42PM -0400, John Snow wrote:
> Hi, following up from my KVM forum talk (
> https://www.youtube.com/watch?v=GGbz3WwvqLw ) I wanted to ask about what
> version of Sphinx we could use as a minimum version to help ease the burden
> of modernizing our QMP docs without a long backwards compatibility tail
> (and hundreds of lines of compatibility goop to make it happen.)
> 
> First, some necessary background on the current state of our Python
> environment and build system:
> 
> Paolo and I worked on a Python virtual environment tied to the QEMU build
> system (python/scripts/mkvenv.py) which performs various setup actions
> necessary to instantiate python *stuff* to ensure the build works and is
> always using the correct, configured Python interpreter.
> 
> Currently, configure will invoke mkvenv in such a fashion that it prefers
> any local packages you have installed: notably, meson, sphinx, and
> sphinx-rtd-theme if you already have them installed from your distro's
> repository will be used *if they use the version of python you specified to
> configure* (or they match the version that configure guessed if you didn't
> specify one.)
> 
> For meson, if you don't meet the minimum version, we will install a version
> for you from our vendored whl files in the repository/tarball.
> 
> For sphinx, if you don't meet the minimum version and you have specified
> --enable-docs, we'll install a version from the internet, from PyPI. If you
> did not specify --enable-docs, we disable docs.
> 
> We did it this way because:
> - It's *really fast* if you already have the dependencies you need
> - meson is easy to vendor as a .whl because it's pure python, has *zero*
> dependencies, and it's absolutely required for the build.
> - docs are not *required* to build QEMU
> - We went out of our way to ensure that QEMU could be built in an offline,
> isolated environment on all of our supported platforms.
> 
> That being said:
> 
> The current reality is that Sphinx 3.4.3 is our minimum because RHEL 9
> offers that as the distro package and I have not dared bump our version
> beyond that for fear of disrupting our ability to build docs on RHEL 9
> without internet.
> 
> What I'd like to ask is: How adamant are we that we can build docs on older
> platforms? Do we consider it part of our platform promise? Can we bump
> Sphinx to a slightly newer version at the expense of offline doc builds for
> RHEL 9?

As you say, with meson, if the distro does not provide a new enough
version, then we pull a suitable version into the venv, overriding
what the distro provides.

IMHO that has set a precedent for /any/ python component we need during
build.

To put it another way....

The distro build platform matrix rules *only* apply to python runtime,
not to python modules. We will bump the min version of python modules
as suits QEMU, without being held back by distro versions.

With this POV, there is nothing special about sphinx.

If we want to bump min sphinx to get a newer version, then we should
just do it.

The only important bit is that our tool for populating the venv must
"do the right thing" detecting that the distro version is too old,
and downloading newer sphinx automatically, so users don't suddenly
loose docs.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Minimum sphinx version?

[Daniel P. Berrangé]

On Tue, Oct 01, 2024 at 10:28:50AM +0200, Paolo Bonzini wrote:
> On 9/30/24 19:16, John Snow wrote:
> > The current reality is that Sphinx 3.4.3 is our minimum because RHEL 9
> > offers that as the distro package and I have not dared bump our version
> > beyond that for fear of disrupting our ability to build docs on RHEL 9
> > without internet.
> > 
> > What I'd like to ask is: How adamant are we that we can build docs on
> > older platforms? Do we consider it part of our platform promise? Can we
> > bump Sphinx to a slightly newer version at the expense of offline doc
> > builds for RHEL 9?
> 
> Could we stop building QMP docs, if Sphinx is too old?  While at the same
> time keeping the same 3.x version and the ability to build man pages, which
> is the really important part.

I'm curious how much of SPhinx we actually leverage when generatnug
man pages ? Our content is primarily RST format. Would it be feasible
to just use rst2man for man pages, and just use sphinx for the HTML
docs ?

As I said in my other email though, IMHO, if the distro sphinx is too
old we should just auto-install the newer version that we need, as we
have set a precedent with meson. That way there's no need to disable
docs building

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|