[PULL 0/6] Python patches

[PULL 0/6] Python patches

[John Snow]

The following changes since commit 4469bee2c529832d762af4a2f89468c926f02fe4:

  Merge tag 'nvme-next-pull-request' of https://gitlab.com/birkelund/qemu into staging (2024-07-11 14:32:51 -0700)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to dd23f9ec519db9c424223cff8767715de5532718:

  docs: remove Sphinx 1.x compatibility code (2024-07-12 16:46:21 -0400)

----------------------------------------------------------------
Python: 3.13 compat & sphinx minver bump

----------------------------------------------------------------

John Snow (6):
  python: linter changes for pylint 3.x
  python: Do not use pylint 3.2.4 with python 3.8
  iotests: Change imports for Python 3.13
  python: enable testing for 3.13
  Python: bump minimum sphinx version to 3.4.3
  docs: remove Sphinx 1.x compatibility code

 docs/conf.py                           |  7 ++---
 docs/sphinx/hxtool.py                  | 21 +++-----------
 docs/sphinx/kerneldoc.py               | 38 ++++++++------------------
 docs/sphinx/kernellog.py               | 28 -------------------
 docs/sphinx/qapidoc.py                 | 33 ++--------------------
 python/qemu/machine/machine.py         |  1 +
 python/qemu/utils/qemu_ga_client.py    |  2 +-
 python/setup.cfg                       |  4 ++-
 pythondeps.toml                        |  2 +-
 tests/docker/dockerfiles/python.docker |  1 +
 tests/qemu-iotests/testenv.py          |  7 ++++-
 tests/qemu-iotests/testrunner.py       |  9 ++++--
 12 files changed, 41 insertions(+), 112 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py

-- 
2.45.0

[PULL 1/6] python: linter changes for pylint 3.x

[John Snow]

New bleeding edge versions, new nits to iron out. This addresses the
'check-python-tox' optional GitLab test, while 'check-python-minreqs'
saw no regressions, since it's frozen on an older version of pylint.

Fixes:
qemu/machine/machine.py:345:52: E0606: Possibly using variable 'sock' before assignment (possibly-used-before-assignment)
qemu/utils/qemu_ga_client.py:168:4: R1711: Useless return at end of function or method (useless-return)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20240626232230.408004-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/machine/machine.py      | 1 +
 python/qemu/utils/qemu_ga_client.py | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py
index f648f6af451..ebb58d5b68c 100644
--- a/python/qemu/machine/machine.py
+++ b/python/qemu/machine/machine.py
@@ -335,6 +335,7 @@ def binary(self) -> str:
 
     def _pre_launch(self) -> None:
         if self._qmp_set:
+            sock = None
             if self._monitor_address is None:
                 self._sock_pair = socket.socketpair()
                 os.set_inheritable(self._sock_pair[0].fileno(), True)
diff --git a/python/qemu/utils/qemu_ga_client.py b/python/qemu/utils/qemu_ga_client.py
index 9a665e6e990..cf0fcf9a8bb 100644
--- a/python/qemu/utils/qemu_ga_client.py
+++ b/python/qemu/utils/qemu_ga_client.py
@@ -174,7 +174,7 @@ def suspend(self, mode: str) -> None:
             # On error exception will raise
         except asyncio.TimeoutError:
             # On success command will timed out
-            return
+            pass
 
     def shutdown(self, mode: str = 'powerdown') -> None:
         if mode not in ['powerdown', 'halt', 'reboot']:
-- 
2.45.0

[PULL 2/6] python: Do not use pylint 3.2.4 with python 3.8

[John Snow]

There is a bug in this version,
see: https://github.com/pylint-dev/pylint/issues/9751

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20240626232230.408004-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/setup.cfg | 1 +
 1 file changed, 1 insertion(+)

diff --git a/python/setup.cfg b/python/setup.cfg
index 48668609d3e..8ebd345d7ed 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -41,6 +41,7 @@ devel =
     isort >= 5.1.2
     mypy >= 1.4.0
     pylint >= 2.17.3
+    pylint != 3.2.4; python_version<"3.9"
     tox >= 3.18.0
     urwid >= 2.1.2
     urwid-readline >= 0.13
-- 
2.45.0

[PULL 3/6] iotests: Change imports for Python 3.13

[John Snow]

Python 3.13 isn't out yet, but it's in beta and Fedora is ramping up to
make it the default system interpreter for Fedora 41.

They moved our cheese for where ContextManager lives; add a conditional
to locate it while we support both pre-3.9 and 3.13+.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20240626232230.408004-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/testenv.py    | 7 ++++++-
 tests/qemu-iotests/testrunner.py | 9 ++++++---
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
index 588f30a4f14..96d69e56963 100644
--- a/tests/qemu-iotests/testenv.py
+++ b/tests/qemu-iotests/testenv.py
@@ -25,7 +25,12 @@
 import random
 import subprocess
 import glob
-from typing import List, Dict, Any, Optional, ContextManager
+from typing import List, Dict, Any, Optional
+
+if sys.version_info >= (3, 9):
+    from contextlib import AbstractContextManager as ContextManager
+else:
+    from typing import ContextManager
 
 DEF_GDB_OPTIONS = 'localhost:12345'
 
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index 7b322272e92..2e236c8fa39 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -27,11 +27,14 @@
 import shutil
 import sys
 from multiprocessing import Pool
-from typing import List, Optional, Any, Sequence, Dict, \
-        ContextManager
-
+from typing import List, Optional, Any, Sequence, Dict
 from testenv import TestEnv
 
+if sys.version_info >= (3, 9):
+    from contextlib import AbstractContextManager as ContextManager
+else:
+    from typing import ContextManager
+
 
 def silent_unlink(path: Path) -> None:
     try:
-- 
2.45.0

[PULL 4/6] python: enable testing for 3.13

[John Snow]

Python 3.13 is in beta and Fedora 41 is preparing to make it the default
system interpreter; enable testing for it.

(In the event problems develop prior to release, it should only impact
the check-python-tox job, which is not run by default and is allowed to
fail.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20240626232230.408004-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/setup.cfg                       | 3 ++-
 tests/docker/dockerfiles/python.docker | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/python/setup.cfg b/python/setup.cfg
index 8ebd345d7ed..3b4e2cc5501 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -19,6 +19,7 @@ classifiers =
     Programming Language :: Python :: 3.10
     Programming Language :: Python :: 3.11
     Programming Language :: Python :: 3.12
+    Programming Language :: Python :: 3.13
     Typing :: Typed
 
 [options]
@@ -184,7 +185,7 @@ multi_line_output=3
 # of python available on your system to run this test.
 
 [tox:tox]
-envlist = py38, py39, py310, py311, py312
+envlist = py38, py39, py310, py311, py312, py313
 skip_missing_interpreters = true
 
 [testenv]
diff --git a/tests/docker/dockerfiles/python.docker b/tests/docker/dockerfiles/python.docker
index a3c1321190c..8f0af9ef25f 100644
--- a/tests/docker/dockerfiles/python.docker
+++ b/tests/docker/dockerfiles/python.docker
@@ -14,6 +14,7 @@ ENV PACKAGES \
     python3.10 \
     python3.11 \
     python3.12 \
+    python3.13 \
     python3.8 \
     python3.9
 
-- 
2.45.0

[PULL 5/6] Python: bump minimum sphinx version to 3.4.3

[John Snow]

With RHEL 8 support retired (It's been two years since RHEL9 released),
our very oldest build platform version of Sphinx is now 3.4.3; and
keeping backwards compatibility for versions as old as v1.6 when using
domain extensions is a lot of work we don't need to do.

This patch is motivated by my work creating a new QAPI domain, which
unlike the dbus documentation, cannot be allowed to regress by creating
a "dummy" doc when operating under older sphinx versions. Easier is to
raise our minimum version as far as we can push it forwards, reducing my
burden in creating cross-compatibility hacks and patches.

A sampling of sphinx versions from various distributions, courtesy
https://repology.org/project/python:sphinx/versions

Alpine 3.16: v4.3.0 (QEMU support ended 2024-05-23)
Alpine 3.17: v5.3.0
Alpine 3.18: v6.1.3
Alpine 3.19: v6.2.1
Ubuntu 20.04 LTS: EOL
Ubuntu 22.04 LTS: v4.3.2
Ubuntu 22.10: EOL
Ubuntu 23.04: EOL
Ubuntu 23.10: v5.3.0
Ubuntu 24.04 LTS: v7.2.6
Debian 11: v3.4.3 (QEMU support ends 2024-07-xx)
Debian 12: v5.3.0
Fedora 38: EOL
Fedora 39: v6.2.1
Fedora 40: v7.2.6
CentOS Stream 8: v1.7.6 (QEMU support ended 2024-05-17)
CentOS Stream 9: v3.4.3
OpenSUSE Leap 15.4: EOL
OpenSUSE Leap 15.5: 2.3.1, 4.2.0 and 7.2.6

RHEL9 / CentOS Stream 9 becomes the new defining factor in staying at
Sphinx 3.4.3 due to downstream offline build requirements that force us
to use platform Sphinx instead of newer packages from PyPI.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-id: 20240703175235.239004-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py    | 7 +++----
 pythondeps.toml | 2 +-
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index aae0304ac6e..876f6768815 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -53,10 +53,9 @@
 
 # If your documentation needs a minimal Sphinx version, state it here.
 #
-# Sphinx 1.5 and earlier can't build our docs because they are too
-# picky about the syntax of the argument to the option:: directive
-# (see Sphinx bugs #646, #3366).
-needs_sphinx = '1.6'
+# 3.4.3 is the oldest version of Sphinx that ships on a platform we
+# pledge build support for.
+needs_sphinx = '3.4.3'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
diff --git a/pythondeps.toml b/pythondeps.toml
index 6aba0c9daaa..f6e590fdd86 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -23,7 +23,7 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
 
 [docs]
 # Please keep the installed versions in sync with docs/requirements.txt
-sphinx = { accepted = ">=1.6", installed = "5.3.0", canary = "sphinx-build" }
+sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
 sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 
 [avocado]
-- 
2.45.0

[PULL 6/6] docs: remove Sphinx 1.x compatibility code

[John Snow]

In general, the Use_SSI workaround is no longer needed, and neither is
the pre-1.6 logging shim for kerneldoc.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-id: 20240703175235.239004-3-jsnow@redhat.com
[rebased on top of origin/master. --js]
Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/hxtool.py    | 21 ++++-----------------
 docs/sphinx/kerneldoc.py | 38 ++++++++++++--------------------------
 docs/sphinx/kernellog.py | 28 ----------------------------
 docs/sphinx/qapidoc.py   | 33 +++------------------------------
 4 files changed, 19 insertions(+), 101 deletions(-)
 delete mode 100644 docs/sphinx/kernellog.py

diff --git a/docs/sphinx/hxtool.py b/docs/sphinx/hxtool.py
index 3729084a36c..a84723be19e 100644
--- a/docs/sphinx/hxtool.py
+++ b/docs/sphinx/hxtool.py
@@ -24,16 +24,10 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 from sphinx.errors import ExtensionError
+from sphinx.util.docutils import switch_source_input
 from sphinx.util.nodes import nested_parse_with_titles
 import sphinx
 
-# Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
-# use switch_source_input. Check borrowed from kerneldoc.py.
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
 
 __version__ = '1.0'
 
@@ -185,16 +179,9 @@ def run(self):
     # of title_styles and section_level that kerneldoc.py does,
     # because nested_parse_with_titles() does that for us.
     def do_parse(self, result, node):
-        if Use_SSI:
-            with switch_source_input(self.state, result):
-                nested_parse_with_titles(self.state, result, node)
-        else:
-            save = self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(result, self.state.memo.reporter)
-            try:
-                nested_parse_with_titles(self.state, result, node)
-            finally:
-                self.state.memo.reporter = save
+        with switch_source_input(self.state, result):
+            nested_parse_with_titles(self.state, result, node)
+
 
 def setup(app):
     """ Register hxtool-doc directive with Sphinx"""
diff --git a/docs/sphinx/kerneldoc.py b/docs/sphinx/kerneldoc.py
index 72c403a7379..3aa972f2e89 100644
--- a/docs/sphinx/kerneldoc.py
+++ b/docs/sphinx/kerneldoc.py
@@ -38,20 +38,14 @@
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 
-#
-# AutodocReporter is only good up to Sphinx 1.7
-#
 import sphinx
+from sphinx.util import logging
+from sphinx.util.docutils import switch_source_input
 
-Use_SSI = sphinx.__version__[:3] >= '1.7'
-if Use_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import AutodocReporter
-
-import kernellog
 
 __version__  = '1.0'
+logger = logging.getLogger('kerneldoc')
+
 
 class KernelDocDirective(Directive):
     """Extract kernel-doc comments from the specified file"""
@@ -111,8 +105,7 @@ def run(self):
         cmd += [filename]
 
         try:
-            kernellog.verbose(env.app,
-                              'calling kernel-doc \'%s\'' % (" ".join(cmd)))
+            logger.verbose('calling kernel-doc \'%s\'' % (" ".join(cmd)))
 
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             out, err = p.communicate()
@@ -122,8 +115,10 @@ def run(self):
             if p.returncode != 0:
                 sys.stderr.write(err)
 
-                kernellog.warn(env.app,
-                               'kernel-doc \'%s\' failed with return code %d' % (" ".join(cmd), p.returncode))
+                logger.warning(
+                    'kernel-doc \'%s\' failed with return code %d' %
+                    (" ".join(cmd), p.returncode)
+                )
                 return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
             elif env.config.kerneldoc_verbosity > 0:
                 sys.stderr.write(err)
@@ -149,22 +144,13 @@ def run(self):
             return node.children
 
         except Exception as e:  # pylint: disable=W0703
-            kernellog.warn(env.app, 'kernel-doc \'%s\' processing failed with: %s' %
+            logger.warning('kernel-doc \'%s\' processing failed with: %s' %
                            (" ".join(cmd), str(e)))
             return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
 
     def do_parse(self, result, node):
-        if Use_SSI:
-            with switch_source_input(self.state, result):
-                self.state.nested_parse(result, 0, node, match_titles=1)
-        else:
-            save = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(result, self.state.memo.reporter)
-            self.state.memo.title_styles, self.state.memo.section_level = [], 0
-            try:
-                self.state.nested_parse(result, 0, node, match_titles=1)
-            finally:
-                self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = save
+        with switch_source_input(self.state, result):
+            self.state.nested_parse(result, 0, node, match_titles=1)
 
 
 def setup(app):
diff --git a/docs/sphinx/kernellog.py b/docs/sphinx/kernellog.py
deleted file mode 100644
index af924f51a7d..00000000000
--- a/docs/sphinx/kernellog.py
+++ /dev/null
@@ -1,28 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-#
-# Sphinx has deprecated its older logging interface, but the replacement
-# only goes back to 1.6.  So here's a wrapper layer to keep around for
-# as long as we support 1.4.
-#
-import sphinx
-
-if sphinx.__version__[:3] >= '1.6':
-    UseLogging = True
-    from sphinx.util import logging
-    logger = logging.getLogger('kerneldoc')
-else:
-    UseLogging = False
-
-def warn(app, message):
-    if UseLogging:
-        logger.warning(message)
-    else:
-        app.warn(message)
-
-def verbose(app, message):
-    if UseLogging:
-        logger.verbose(message)
-    else:
-        app.verbose(message)
-
-
diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 2b06750a3cd..62b39833ca0 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -35,22 +35,11 @@
 from qapi.gen import QAPISchemaVisitor
 from qapi.schema import QAPISchema
 
-import sphinx
 from sphinx.errors import ExtensionError
+from sphinx.util.docutils import switch_source_input
 from sphinx.util.nodes import nested_parse_with_titles
 
 
-# Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
-# use switch_source_input. Check borrowed from kerneldoc.py.
-USE_SSI = sphinx.__version__[:3] >= "1.7"
-if USE_SSI:
-    from sphinx.util.docutils import switch_source_input
-else:
-    from sphinx.ext.autodoc import (  # pylint: disable=no-name-in-module
-        AutodocReporter,
-    )
-
-
 __version__ = "1.0"
 
 
@@ -539,24 +528,8 @@ def do_parse(self, rstlist, node):
         subheadings (titles) without confusing the rendering of
         anything else.
         """
-        # This is from kerneldoc.py -- it works around an API change in
-        # Sphinx between 1.6 and 1.7. Unlike kerneldoc.py, we use
-        # sphinx.util.nodes.nested_parse_with_titles() rather than the
-        # plain self.state.nested_parse(), and so we can drop the saving
-        # of title_styles and section_level that kerneldoc.py does,
-        # because nested_parse_with_titles() does that for us.
-        if USE_SSI:
-            with switch_source_input(self.state, rstlist):
-                nested_parse_with_titles(self.state, rstlist, node)
-        else:
-            save = self.state.memo.reporter
-            self.state.memo.reporter = AutodocReporter(
-                rstlist, self.state.memo.reporter
-            )
-            try:
-                nested_parse_with_titles(self.state, rstlist, node)
-            finally:
-                self.state.memo.reporter = save
+        with switch_source_input(self.state, rstlist):
+            nested_parse_with_titles(self.state, rstlist, node)
 
 
 def setup(app):
-- 
2.45.0

Re: [PULL 0/6] Python patches

[Richard Henderson]

On 7/16/24 03:32, John Snow wrote:
> The following changes since commit 4469bee2c529832d762af4a2f89468c926f02fe4:
> 
>    Merge tag 'nvme-next-pull-request' ofhttps://gitlab.com/birkelund/qemu  into staging (2024-07-11 14:32:51 -0700)
> 
> are available in the Git repository at:
> 
>    https://gitlab.com/jsnow/qemu.git  tags/python-pull-request
> 
> for you to fetch changes up to dd23f9ec519db9c424223cff8767715de5532718:
> 
>    docs: remove Sphinx 1.x compatibility code (2024-07-12 16:46:21 -0400)
> 
> ----------------------------------------------------------------
> Python: 3.13 compat & sphinx minver bump

Applied, thanks.  Please update https://wiki.qemu.org/ChangeLog/9.1 as appropriate.

r~