[PULL 0/5] Python patches

[PULL 0/5] Python patches

[John Snow]

The following changes since commit ecc9a58835f8d4ea4e3ed36832032a71ee08fbb2:

  Merge tag 'pull-9p-20221223' of https://github.com/cschoenebeck/qemu into staging (2023-01-04 14:53:59 +0000)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to 519f3cfce07a067971ff39d4a989b77e7100a947:

  python: add 3.11 to supported list (2023-01-04 13:46:05 -0500)

----------------------------------------------------------------
Python patch roundup

Mostly CI fixes and some small debugging improvements.

----------------------------------------------------------------

John Snow (5):
  python/machine: Add debug logging to key state changes
  python/machine: Handle termination cases without QMP
  Python: fix flake8 config
  iotests/check: Fix typing for sys.exit() value
  python: add 3.11 to supported list

 python/qemu/machine/machine.py | 31 +++++++++++++++++++++++++++++++
 python/setup.cfg               |  6 ++++--
 tests/qemu-iotests/check       |  2 +-
 3 files changed, 36 insertions(+), 3 deletions(-)

-- 
2.39.0

[PULL 1/5] python/machine: Add debug logging to key state changes

[John Snow]

When key decisions are made about the lifetime of the VM process being
managed, there's no log entry. Juxtaposed with the very verbose runstate
change logging of the QMP module, machine seems a bit too introverted
now.

Season the machine.py module with logging statements to taste to help
make a tastier soup.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/machine/machine.py | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py
index 37191f433b2..6f1374a7550 100644
--- a/python/qemu/machine/machine.py
+++ b/python/qemu/machine/machine.py
@@ -373,6 +373,7 @@ def _post_shutdown(self) -> None:
         Called to cleanup the VM instance after the process has exited.
         May also be called after a failed launch.
         """
+        LOG.debug("Cleaning up after VM process")
         try:
             self._close_qmp_connection()
         except Exception as err:  # pylint: disable=broad-except
@@ -497,6 +498,7 @@ def _early_cleanup(self) -> None:
         # for QEMU to exit, while QEMU is waiting for the socket to
         # become writable.
         if self._console_socket is not None:
+            LOG.debug("Closing console socket")
             self._console_socket.close()
             self._console_socket = None
 
@@ -507,6 +509,7 @@ def _hard_shutdown(self) -> None:
         :raise subprocess.Timeout: When timeout is exceeds 60 seconds
             waiting for the QEMU process to terminate.
         """
+        LOG.debug("Performing hard shutdown")
         self._early_cleanup()
         self._subp.kill()
         self._subp.wait(timeout=60)
@@ -523,8 +526,18 @@ def _soft_shutdown(self, timeout: Optional[int]) -> None:
         :raise subprocess.TimeoutExpired: When timeout is exceeded waiting for
             the QEMU process to terminate.
         """
+        LOG.debug("Attempting graceful termination")
+
         self._early_cleanup()
 
+        if self._quit_issued:
+            LOG.debug(
+                "Anticipating QEMU termination due to prior 'quit' command, "
+                "or explicit call to wait()"
+            )
+        else:
+            LOG.debug("Politely asking QEMU to terminate")
+
         if self._qmp_connection:
             try:
                 if not self._quit_issued:
@@ -536,6 +549,10 @@ def _soft_shutdown(self, timeout: Optional[int]) -> None:
                 self._close_qmp_connection()
 
         # May raise subprocess.TimeoutExpired
+        LOG.debug(
+            "Waiting (timeout=%s) for QEMU process (pid=%s) to terminate",
+            timeout, self._subp.pid
+        )
         self._subp.wait(timeout=timeout)
 
     def _do_shutdown(self, timeout: Optional[int]) -> None:
@@ -553,6 +570,10 @@ def _do_shutdown(self, timeout: Optional[int]) -> None:
         try:
             self._soft_shutdown(timeout)
         except Exception as exc:
+            if isinstance(exc, subprocess.TimeoutExpired):
+                LOG.debug("Timed out waiting for QEMU process to exit")
+            LOG.debug("Graceful shutdown failed", exc_info=True)
+            LOG.debug("Falling back to hard shutdown")
             self._hard_shutdown()
             raise AbnormalShutdown("Could not perform graceful shutdown") \
                 from exc
@@ -575,6 +596,10 @@ def shutdown(self,
         if not self._launched:
             return
 
+        LOG.debug("Shutting down VM appliance; timeout=%s", timeout)
+        if hard:
+            LOG.debug("Caller requests immediate termination of QEMU process.")
+
         try:
             if hard:
                 self._user_killed = True
-- 
2.39.0

[PULL 2/5] python/machine: Handle termination cases without QMP

[John Snow]

If we request a shutdown of a VM without a QMP console, we'll just hang
waiting. Not ideal.

Add in code that attempts graceful termination in these cases.  Tested
lightly; it appears to work and I doubt we rely on this case anywhere,
but it's a corner you're allowed to wedge yourself in, so it should be
handled.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/machine/machine.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py
index 6f1374a7550..748a0d807c9 100644
--- a/python/qemu/machine/machine.py
+++ b/python/qemu/machine/machine.py
@@ -547,6 +547,12 @@ def _soft_shutdown(self, timeout: Optional[int]) -> None:
             finally:
                 # Regardless, we want to quiesce the connection.
                 self._close_qmp_connection()
+        elif not self._quit_issued:
+            LOG.debug(
+                "Not anticipating QEMU quit and no QMP connection present, "
+                "issuing SIGTERM"
+            )
+            self._subp.terminate()
 
         # May raise subprocess.TimeoutExpired
         LOG.debug(
-- 
2.39.0

[PULL 3/5] Python: fix flake8 config

[John Snow]

Newer flake8 versions are a bit pickier about the config file, and my
in-line comment confuses the parser. Fix it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Message-id: 20221203005234.620788-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/setup.cfg | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/python/setup.cfg b/python/setup.cfg
index c2c61c75190..c0d7bab168e 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -71,7 +71,8 @@ console_scripts =
     qmp-tui = qemu.qmp.qmp_tui:main [tui]
 
 [flake8]
-extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
+# Prefer pylint's bare-except checks to flake8's
+extend-ignore = E722
 exclude = __pycache__,
 
 [mypy]
-- 
2.39.0

[PULL 4/5] iotests/check: Fix typing for sys.exit() value

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Message-id: 20221203005234.620788-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/check | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 75de1b4691e..9bdda1394e7 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -159,7 +159,7 @@ if __name__ == '__main__':
         if not tests:
             raise ValueError('No tests selected')
     except ValueError as e:
-        sys.exit(e)
+        sys.exit(str(e))
 
     if args.dry_run:
         print('\n'.join(tests))
-- 
2.39.0

[PULL 5/5] python: add 3.11 to supported list

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com>
Message-id: 20221203005234.620788-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/setup.cfg | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/python/setup.cfg b/python/setup.cfg
index c0d7bab168e..56418157065 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -19,6 +19,7 @@ classifiers =
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
     Programming Language :: Python :: 3.10
+    Programming Language :: Python :: 3.11
     Typing :: Typed
 
 [options]
@@ -159,7 +160,7 @@ multi_line_output=3
 # of python available on your system to run this test.
 
 [tox:tox]
-envlist = py36, py37, py38, py39, py310
+envlist = py36, py37, py38, py39, py310, py311
 skip_missing_interpreters = true
 
 [testenv]
-- 
2.39.0

Re: [PULL 0/5] Python patches

[Peter Maydell]

On Wed, 4 Jan 2023 at 21:05, John Snow <jsnow@redhat.com> wrote:
>
> The following changes since commit ecc9a58835f8d4ea4e3ed36832032a71ee08fbb2:
>
>   Merge tag 'pull-9p-20221223' of https://github.com/cschoenebeck/qemu into staging (2023-01-04 14:53:59 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>
> for you to fetch changes up to 519f3cfce07a067971ff39d4a989b77e7100a947:
>
>   python: add 3.11 to supported list (2023-01-04 13:46:05 -0500)
>
> ----------------------------------------------------------------
> Python patch roundup
>
> Mostly CI fixes and some small debugging improvements.
>


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/8.0
for any user-visible changes.

-- PMM