[PATCH 0/2] docs/python: bump minimum Sphinx version

[PATCH 0/2] docs/python: bump minimum Sphinx version

[John Snow]

With recent deprecations, we can advance our minimum sphinx version
safely. This is heavily motivated by new qapidoc work which is much
easier to maintain cross-version compatibility for - see difficulties in
our dbus documentation which only works on sphinx >= 4.

We can only guarantee >= 3.4.3 now, but that's still vastly easier than
maintaining compatibility all the way back to 1.x.

GitLab: https://gitlab.com/jsnow/qemu/-/pipelines/1357902509

(failures appear to be unrelated to the series.)

John Snow (2):
  Python: bump minimum sphinx version to 3.4.3
  docs: remove Sphinx 1.x compatibility code

 docs/conf.py             |  7 +++----
 docs/sphinx/hxtool.py    | 21 ++++-----------------
 docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
 docs/sphinx/kernellog.py | 28 ----------------------------
 docs/sphinx/qapidoc.py   | 29 +++--------------------------
 pythondeps.toml          |  2 +-
 6 files changed, 23 insertions(+), 102 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py

-- 
2.45.0

[PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

[John Snow]

With RHEL 8 support retired (It's been two years today since RHEL 9
came out), our very oldest build platform version of Sphinx is now
3.4.3; and keeping backwards compatibility for versions as old as v1.6
when using domain extensions is a lot of work we don't need to do.

This patch is motivated by my work creating a new QAPI domain, which
unlike the dbus documentation, cannot be allowed to regress by creating
a "dummy" doc. Easier is to improve our minimum version as far as we can
push it forwards, reducing my burden in creating cross-compatibility
hacks and patches.

A sampling of sphinx versions from various distributions, courtesy
https://repology.org/project/python:sphinx/versions

Alpine 3.16: v4.3.0 (QEMU support ended 2024-05-23)
Alpine 3.17: v5.3.0
Alpine 3.18: v6.1.3
Alpine 3.19: v6.2.1
Ubuntu 20.04 LTS: EOL
Ubuntu 22.04 LTS: v4.3.2
Ubuntu 22.10: EOL
Ubuntu 23.04: EOL
Ubuntu 23.10: v5.3.0
Ubuntu 24.04 LTS: v7.2.6
Debian 11: v3.4.3 (QEMU support ends 2024-07-xx)
Debian 12: v5.3.0
Fedora 38: EOL
Fedora 39: v6.2.1
Fedora 40: v7.2.6
CentOS Stream 8: v1.7.6 (QEMU support ended 2024-05-17)
CentOS Stream 9: v3.4.3
OpenSUSE Leap 15.4: EOL
OpenSUSE Leap 15.5: 2.3.1, 4.2.0 and 7.2.6

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py           |  7 +++----
 docs/sphinx/qapidoc.py | 29 +++--------------------------
 pythondeps.toml        |  2 +-
 3 files changed, 7 insertions(+), 31 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index aae0304ac6e..876f6768815 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -53,10 +53,9 @@
 
 # If your documentation needs a minimal Sphinx version, state it here.
 #
-# Sphinx 1.5 and earlier can't build our docs because they are too
-# picky about the syntax of the argument to the option:: directive
-# (see Sphinx bugs #646, #3366).
-needs_sphinx = '1.6'
+# 3.4.3 is the oldest version of Sphinx that ships on a platform we
+# pledge build support for.
+needs_sphinx = '3.4.3'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index f270b494f01..9a0cfcbce75 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -31,6 +31,7 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 from sphinx.errors import ExtensionError
+from sphinx.util.docutils import switch_source_input
 from sphinx.util.nodes import nested_parse_with_titles
 import sphinx
 from qapi.gen import QAPISchemaVisitor
@@ -38,15 +39,6 @@
 from qapi.schema import QAPISchema
 
 
-# Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
-# use switch_source_input. Check borrowed from kerneldoc.py.
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
-
-
 __version__ = '1.0'
 
 
@@ -517,23 +509,8 @@ def do_parse(self, rstlist, node):
         subheadings (titles) without confusing the rendering of
         anything else.
         """
-        # This is from kerneldoc.py -- it works around an API change in
-        # Sphinx between 1.6 and 1.7. Unlike kerneldoc.py, we use
-        # sphinx.util.nodes.nested_parse_with_titles() rather than the
-        # plain self.state.nested_parse(), and so we can drop the saving
-        # of title_styles and section_level that kerneldoc.py does,
-        # because nested_parse_with_titles() does that for us.
-        if Use_SSI:
-            with switch_source_input(self.state, rstlist):
-                nested_parse_with_titles(self.state, rstlist, node)
-        else:
-            save = self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(
-                rstlist, self.state.memo.reporter)
-            try:
-                nested_parse_with_titles(self.state, rstlist, node)
-            finally:
-                self.state.memo.reporter = save
+        with switch_source_input(self.state, rstlist):
+            nested_parse_with_titles(self.state, rstlist, node)
 
 
 def setup(app):
diff --git a/pythondeps.toml b/pythondeps.toml
index 9c16602d303..bc656376caa 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -23,7 +23,7 @@ meson = { accepted = ">=0.63.0", installed = "1.2.3", canary = "meson" }
 
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
-sphinx = { accepted = ">=1.6", installed = "5.3.0", canary = "sphinx-build" }
+sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
 [avocado]
-- 
2.45.0

[PATCH 2/2] docs: remove Sphinx 1.x compatibility code

[John Snow]

In general, the Use_SSI workaround is no longer needed, and neither is
the pre-1.6 logging shim for kerneldoc.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/hxtool.py    | 21 ++++-----------------
 docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
 docs/sphinx/kernellog.py | 28 ----------------------------
 3 files changed, 16 insertions(+), 71 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py

diff --git a/docs/sphinx/hxtool.py b/docs/sphinx/hxtool.py
index 3729084a36c..a84723be19e 100644
--- a/docs/sphinx/hxtool.py
+++ b/docs/sphinx/hxtool.py
@@ -24,16 +24,10 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 from sphinx.errors import ExtensionError
+from sphinx.util.docutils import switch_source_input
 from sphinx.util.nodes import nested_parse_with_titles
 import sphinx
 
-# Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
-# use switch_source_input. Check borrowed from kerneldoc.py.
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
 
 __version__ = '1.0'
 
@@ -185,16 +179,9 @@ def run(self):
     # of title_styles and section_level that kerneldoc.py does,
     # because nested_parse_with_titles() does that for us.
     def do_parse(self, result, node):
-        if Use_SSI:
-            with switch_source_input(self.state, result):
-                nested_parse_with_titles(self.state, result, node)
-        else:
-            save = self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(result, self.state.memo.reporter)
-            try:
-                nested_parse_with_titles(self.state, result, node)
-            finally:
-                self.state.memo.reporter = save
+        with switch_source_input(self.state, result):
+            nested_parse_with_titles(self.state, result, node)
+
 
 def setup(app):
     """ Register hxtool-doc directive with Sphinx"""
diff --git a/docs/sphinx/kerneldoc.py b/docs/sphinx/kerneldoc.py
index 72c403a7379..3aa972f2e89 100644
--- a/docs/sphinx/kerneldoc.py
+++ b/docs/sphinx/kerneldoc.py
@@ -38,20 +38,14 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 
-#
-# AutodocReporter is only good up to Sphinx 1.7
-#
 import sphinx
+from sphinx.util import logging
+from sphinx.util.docutils import switch_source_input
 
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
-
-import kernellog
 
 __version__  = '1.0'
+logger = logging.getLogger('kerneldoc')
+
 
 class KernelDocDirective(Directive):
     """Extract kernel-doc comments from the specified file"""
@@ -111,8 +105,7 @@ def run(self):
         cmd += [filename]
 
         try:
-            kernellog.verbose(env.app,
-                              'calling kernel-doc \'%s\'' % (" ".join(cmd)))
+            logger.verbose('calling kernel-doc \'%s\'' % (" ".join(cmd)))
 
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             out, err = p.communicate()
@@ -122,8 +115,10 @@ def run(self):
             if p.returncode != 0:
                 sys.stderr.write(err)
 
-                kernellog.warn(env.app,
-                               'kernel-doc \'%s\' failed with return code %d' % (" ".join(cmd), p.returncode))
+                logger.warning(
+                    'kernel-doc \'%s\' failed with return code %d' %
+                    (" ".join(cmd), p.returncode)
+                )
                 return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
             elif env.config.kerneldoc_verbosity > 0:
                 sys.stderr.write(err)
@@ -149,22 +144,13 @@ def run(self):
             return node.children
 
         except Exception as e:  # pylint: disable=W0703
-            kernellog.warn(env.app, 'kernel-doc \'%s\' processing failed with: %s' %
+            logger.warning('kernel-doc \'%s\' processing failed with: %s' %
                            (" ".join(cmd), str(e)))
             return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
 
     def do_parse(self, result, node):
-        if Use_SSI:
-            with switch_source_input(self.state, result):
-                self.state.nested_parse(result, 0, node, match_titles=1)
-        else:
-            save = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(result, self.state.memo.reporter)
-            self.state.memo.title_styles, self.state.memo.section_level = [], 0
-            try:
-                self.state.nested_parse(result, 0, node, match_titles=1)
-            finally:
-                self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = save
+        with switch_source_input(self.state, result):
+            self.state.nested_parse(result, 0, node, match_titles=1)
 
 
 def setup(app):
diff --git a/docs/sphinx/kernellog.py b/docs/sphinx/kernellog.py
deleted file mode 100644
index af924f51a7d..00000000000
--- a/docs/sphinx/kernellog.py
+++ /dev/null
@@ -1,28 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-#
-# Sphinx has deprecated its older logging interface, but the replacement
-# only goes back to 1.6.  So here's a wrapper layer to keep around for
-# as long as we support 1.4.
-#
-import sphinx
-
-if sphinx.__version__[:3] >= '1.6':
-    UseLogging = True
-    from sphinx.util import logging
-    logger = logging.getLogger('kerneldoc')
-else:
-    UseLogging = False
-
-def warn(app, message):
-    if UseLogging:
-        logger.warning(message)
-    else:
-        app.warn(message)
-
-def verbose(app, message):
-    if UseLogging:
-        logger.verbose(message)
-    else:
-        app.verbose(message)
-
-
-- 
2.45.0

Re: [PATCH 0/2] docs/python: bump minimum Sphinx version

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

On Tue, Jul 2, 2024 at 3:59 PM John Snow <jsnow@redhat.com> wrote:

> With recent deprecations, we can advance our minimum sphinx version
> safely. This is heavily motivated by new qapidoc work which is much
> easier to maintain cross-version compatibility for - see difficulties in
> our dbus documentation which only works on sphinx >= 4.
>
> We can only guarantee >= 3.4.3 now, but that's still vastly easier than
> maintaining compatibility all the way back to 1.x.
>
> GitLab: https://gitlab.com/jsnow/qemu/-/pipelines/1357902509
>
> (failures appear to be unrelated to the series.)
>
> John Snow (2):
>   Python: bump minimum sphinx version to 3.4.3
>   docs: remove Sphinx 1.x compatibility code
>
>  docs/conf.py             |  7 +++----
>  docs/sphinx/hxtool.py    | 21 ++++-----------------
>  docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
>  docs/sphinx/kernellog.py | 28 ----------------------------
>  docs/sphinx/qapidoc.py   | 29 +++--------------------------
>  pythondeps.toml          |  2 +-
>  6 files changed, 23 insertions(+), 102 deletions(-)
>  delete mode 100644 docs/sphinx/kernellog.py
>
> --
> 2.45.0
>

Bleurgh. I meant to shuffle some of the changes in patch 1 into patch 2,
I'll fix that on re-spin. If you want to review it anyway, just presume
I'll fix that next go-around.

--js

[-- Attachment #2: Type: text/html, Size: 1924 bytes --]

Re: [PATCH 0/2] docs/python: bump minimum Sphinx version

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Jul 2, 2024 at 3:59 PM John Snow <jsnow@redhat.com> wrote:
>
>> With recent deprecations, we can advance our minimum sphinx version
>> safely. This is heavily motivated by new qapidoc work which is much
>> easier to maintain cross-version compatibility for - see difficulties in
>> our dbus documentation which only works on sphinx >= 4.
>>
>> We can only guarantee >= 3.4.3 now, but that's still vastly easier than
>> maintaining compatibility all the way back to 1.x.
>>
>> GitLab: https://gitlab.com/jsnow/qemu/-/pipelines/1357902509
>>
>> (failures appear to be unrelated to the series.)
>>
>> John Snow (2):
>>   Python: bump minimum sphinx version to 3.4.3
>>   docs: remove Sphinx 1.x compatibility code
>>
>>  docs/conf.py             |  7 +++----
>>  docs/sphinx/hxtool.py    | 21 ++++-----------------
>>  docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
>>  docs/sphinx/kernellog.py | 28 ----------------------------
>>  docs/sphinx/qapidoc.py   | 29 +++--------------------------
>>  pythondeps.toml          |  2 +-
>>  6 files changed, 23 insertions(+), 102 deletions(-)
>>  delete mode 100644 docs/sphinx/kernellog.py
>>
>> --
>> 2.45.0
>>
>
> Bleurgh. I meant to shuffle some of the changes in patch 1 into patch 2,

I was wondering about the split :)

> I'll fix that on re-spin. If you want to review it anyway, just presume
> I'll fix that next go-around.

Acked-by: Markus Armbruster <armbru@redhat.com>

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

[Paolo Bonzini]

On 7/2/24 21:59, John Snow wrote:
> With RHEL 8 support retired (It's been two years today since RHEL 9
> came out), our very oldest build platform version of Sphinx is now
> 3.4.3; and keeping backwards compatibility for versions as old as v1.6
> when using domain extensions is a lot of work we don't need to do.

Technically that's unrelated: thanks to your venv work, :) builds on 
RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it 
runs under Python 3.6.  Therefore the version included in RHEL 8 does 
not matter for picking the minimum supported Sphinx version.

> Debian 11: v3.4.3 (QEMU support ends 2024-07-xx)

Nice. :)

> diff --git a/pythondeps.toml b/pythondeps.toml
> index 9c16602d303..bc656376caa 100644
> --- a/pythondeps.toml
> +++ b/pythondeps.toml
> @@ -23,7 +23,7 @@ meson = { accepted = ">=0.63.0", installed = "1.2.3", canary = "meson" }
>   
>   [docs]
>   # Please keep the installed versions in sync with docs/requirements.txt
> -sphinx = { accepted = ">=1.6", installed = "5.3.0", canary = "sphinx-build" }
> +sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
>   sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
>   
>   [avocado]

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Wed, Jul 3, 2024, 4:00 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 7/2/24 21:59, John Snow wrote:
> > With RHEL 8 support retired (It's been two years today since RHEL 9
> > came out), our very oldest build platform version of Sphinx is now
> > 3.4.3; and keeping backwards compatibility for versions as old as v1.6
> > when using domain extensions is a lot of work we don't need to do.
>
> Technically that's unrelated: thanks to your venv work, :) builds on
> RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it
> runs under Python 3.6.  Therefore the version included in RHEL 8 does
> not matter for picking the minimum supported Sphinx version.
>

...!

I think I can't mandate 4.x because of RHEL 9 builds though, and offline
requirements.


> > Debian 11: v3.4.3 (QEMU support ends 2024-07-xx)
>
> Nice. :)
>
> > diff --git a/pythondeps.toml b/pythondeps.toml
> > index 9c16602d303..bc656376caa 100644
> > --- a/pythondeps.toml
> > +++ b/pythondeps.toml
> > @@ -23,7 +23,7 @@ meson = { accepted = ">=0.63.0", installed = "1.2.3",
> canary = "meson" }
> >
> >   [docs]
> >   # Please keep the installed versions in sync with docs/requirements.txt
> > -sphinx = { accepted = ">=1.6", installed = "5.3.0", canary =
> "sphinx-build" }
> > +sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary =
> "sphinx-build" }
> >   sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
> >
> >   [avocado]
>
> Acked-by: Paolo Bonzini <pbonzini@redhat.com>
>
> Paolo
>
>

[-- Attachment #2: Type: text/html, Size: 2461 bytes --]

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

[Paolo Bonzini]

On Wed, Jul 3, 2024 at 2:06 PM John Snow <jsnow@redhat.com> wrote:
> On Wed, Jul 3, 2024, 4:00 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On 7/2/24 21:59, John Snow wrote:
>> > With RHEL 8 support retired (It's been two years today since RHEL 9
>> > came out), our very oldest build platform version of Sphinx is now
>> > 3.4.3; and keeping backwards compatibility for versions as old as v1.6
>> > when using domain extensions is a lot of work we don't need to do.
>>
>> Technically that's unrelated: thanks to your venv work, :) builds on
>> RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it
>> runs under Python 3.6.  Therefore the version included in RHEL 8 does
>> not matter for picking the minimum supported Sphinx version.
>
> I think I can't mandate 4.x because of RHEL 9 builds though, and offline requirements.

Offline requirements are not a problem; on RHEL 8 you just have to
install with pip in order to build docs offline. But yes, RHEL 9 is
still using platform Python and therefore 3.4.3 remains the limit even
after we stop supporting bullseye.

Paolo

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

On Wed, Jul 3, 2024 at 8:12 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On Wed, Jul 3, 2024 at 2:06 PM John Snow <jsnow@redhat.com> wrote:
> > On Wed, Jul 3, 2024, 4:00 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
> >> On 7/2/24 21:59, John Snow wrote:
> >> > With RHEL 8 support retired (It's been two years today since RHEL 9
> >> > came out), our very oldest build platform version of Sphinx is now
> >> > 3.4.3; and keeping backwards compatibility for versions as old as v1.6
> >> > when using domain extensions is a lot of work we don't need to do.
> >>
> >> Technically that's unrelated: thanks to your venv work, :) builds on
> >> RHEL 8 / CentOS Stream 8 do not pick the platform Sphinx, because it
> >> runs under Python 3.6.  Therefore the version included in RHEL 8 does
> >> not matter for picking the minimum supported Sphinx version.
> >
> > I think I can't mandate 4.x because of RHEL 9 builds though, and offline
> requirements.
>
> Offline requirements are not a problem; on RHEL 8 you just have to
> install with pip in order to build docs offline. But yes, RHEL 9 is
> still using platform Python and therefore 3.4.3 remains the limit even
> after we stop supporting bullseye.
>

To be clear I mean offline, isolated RPM builds under RHEL9 where I don't
think we can utilize PyPI at all; and vendoring Sphinx is I think not a
practical option due to the number of dependencies and non-pure Python deps.

It's not a problem for developer workflow, just downstream packaging.

Luckily OpenSUSE offers newer Sphinx, but RHEL doesn't yet. Maybe that can
be rectified eventually - possibly after 3.8 is EOL and there is increased
demand for newer Python packages to be made available in RHEL... but not
yet, today.


>
> Paolo
>
>
Thanks for the ack O:-)

[-- Attachment #2: Type: text/html, Size: 2632 bytes --]

Re: [PATCH 1/2] Python: bump minimum sphinx version to 3.4.3

[Paolo Bonzini]

On Wed, Jul 3, 2024 at 5:25 PM John Snow <jsnow@redhat.com> wrote:
> To be clear I mean offline, isolated RPM builds under RHEL9 where I don't think we can utilize PyPI at all; and vendoring Sphinx is I think not a practical option due to the number of dependencies and non-pure Python deps.
>
> It's not a problem for developer workflow, just downstream packaging.

Agreed, switching to PyPI packages is not something that should be
done lightly---only after the platform Python is EOL, and only if
there are benefits for QEMU to require 3.9 (like there were for 3.7).

In that case, downstream could still package QEMU but they'd have to
build python39-sphinx or something like that.

Paolo

> Luckily OpenSUSE offers newer Sphinx, but RHEL doesn't yet. Maybe that can be rectified eventually - possibly after 3.8 is EOL and there is increased demand for newer Python packages to be made available in RHEL... but not yet, today.
>
>>
>>
>> Paolo
>>
>
> Thanks for the ack O:-)