[PATCH v3 0/2] python: a few improvements to qmp-shell

[PATCH v3 0/2] python: a few improvements to qmp-shell

[Daniel P. Berrangé]

This makes the qmp-shell program a little more pleasant to use when you
are just trying to spawn a throw-away QEMU process to query some info
from.

First it introduces a 'qmp-shell-wrap' command that takes a QEMU command
line instead of QMP socket, and spawns QEMU automatically, so its life
is tied to that of the shell.

Second it adds ability to log QMP commands/responses to a file that can
be queried with 'jq' to extract information. This is good for commands
which return huge JSON docs.

In v3:

 - Add qmp-shell-wrap to setup.cfg entry points

In v2:

 - Unlink unix socket path on exit
 - Fix default command name
 - Deal with flake8/pylint warnings

Daniel P. Berrangé (2):
  python: introduce qmp-shell-wrap convenience tool
  python: support recording QMP session to a file

 python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
 python/setup.cfg              |  4 ++
 scripts/qmp/qmp-shell-wrap    | 11 +++++
 3 files changed, 96 insertions(+), 7 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap

-- 
2.34.1

[PATCH v3 1/2] python: introduce qmp-shell-wrap convenience tool

[Daniel P. Berrangé]

With the current 'qmp-shell' tool developers must first spawn QEMU with
a suitable -qmp arg and then spawn qmp-shell in a separate terminal
pointing to the right socket.

With 'qmp-shell-wrap' developers can ignore QMP sockets entirely and
just pass the QEMU command and arguments they want. The program will
listen on a UNIX socket and tell QEMU to connect QMP to that.

For example, this:

 # qmp-shell-wrap -- qemu-system-x86_64 -display none

Is roughly equivalent of running:

 # qemu-system-x86_64 -display none -qmp qmp-shell-1234 &
 # qmp-shell qmp-shell-1234

Except that 'qmp-shell-wrap' switches the socket peers around so that
it is the UNIX socket server and QEMU is the socket client. This makes
QEMU reliably go away when qmp-shell-wrap exits, closing the server
socket.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 python/qemu/aqmp/qmp_shell.py | 67 ++++++++++++++++++++++++++++++++---
 python/setup.cfg              |  1 +
 scripts/qmp/qmp-shell-wrap    | 11 ++++++
 3 files changed, 75 insertions(+), 4 deletions(-)
 create mode 100755 scripts/qmp/qmp-shell-wrap

diff --git a/python/qemu/aqmp/qmp_shell.py b/python/qemu/aqmp/qmp_shell.py
index d11bf54b00..185f5066ef 100644
--- a/python/qemu/aqmp/qmp_shell.py
+++ b/python/qemu/aqmp/qmp_shell.py
@@ -86,6 +86,7 @@
 import os
 import re
 import readline
+from subprocess import Popen
 import sys
 from typing import (
     Iterator,
@@ -167,8 +168,10 @@ class QMPShell(QEMUMonitorProtocol):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
         self._transmode = False
@@ -409,8 +412,10 @@ class HMPShell(QMPShell):
     :param verbose: Echo outgoing QMP messages to console.
     """
     def __init__(self, address: SocketAddrT,
-                 pretty: bool = False, verbose: bool = False):
-        super().__init__(address, pretty, verbose)
+                 pretty: bool = False,
+                 verbose: bool = False,
+                 server: bool = False):
+        super().__init__(address, pretty, verbose, server)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -533,5 +538,59 @@ def main() -> None:
             pass
 
 
+def main_wrap() -> None:
+    """
+    qmp-shell-wrap entry point: parse command line arguments and
+    start the REPL.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-H', '--hmp', action='store_true',
+                        help='Use HMP interface')
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        help='Verbose (echo commands sent and received)')
+    parser.add_argument('-p', '--pretty', action='store_true',
+                        help='Pretty-print JSON')
+
+    parser.add_argument('command', nargs=argparse.REMAINDER,
+                        help='QEMU command line to invoke')
+
+    args = parser.parse_args()
+
+    cmd = args.command
+    if len(cmd) != 0 and cmd[0] == '--':
+        cmd = cmd[1:]
+    if len(cmd) == 0:
+        cmd = ["qemu-system-x86_64"]
+
+    sockpath = "qmp-shell-wrap-%d" % os.getpid()
+    cmd += ["-qmp", "unix:%s" % sockpath]
+
+    shell_class = HMPShell if args.hmp else QMPShell
+
+    try:
+        address = shell_class.parse_address(sockpath)
+    except qmp.QMPBadPortError:
+        parser.error(f"Bad port number: {sockpath}")
+        return  # pycharm doesn't know error() is noreturn
+
+    try:
+        with shell_class(address, args.pretty, args.verbose, True) as qemu:
+            with Popen(cmd):
+
+                try:
+                    qemu.accept()
+                except qmp.QMPConnectError:
+                    die("Didn't get QMP greeting message")
+                except qmp.QMPCapabilitiesError:
+                    die("Couldn't negotiate capabilities")
+                except OSError as err:
+                    die(f"Couldn't connect to {sockpath}: {err!s}")
+
+                for _ in qemu.repl():
+                    pass
+    finally:
+        os.unlink(sockpath)
+
+
 if __name__ == '__main__':
     main()
diff --git a/python/setup.cfg b/python/setup.cfg
index 3fb18f845d..84c0f33baf 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -68,6 +68,7 @@ console_scripts =
     qom-fuse = qemu.utils.qom_fuse:QOMFuse.entry_point [fuse]
     qemu-ga-client = qemu.utils.qemu_ga_client:main
     qmp-shell = qemu.aqmp.qmp_shell:main
+    qmp-shell-wrap = qemu.aqmp.qmp_shell:main_wrap
     aqmp-tui = qemu.aqmp.aqmp_tui:main [tui]
 
 [flake8]
diff --git a/scripts/qmp/qmp-shell-wrap b/scripts/qmp/qmp-shell-wrap
new file mode 100755
index 0000000000..9e94da114f
--- /dev/null
+++ b/scripts/qmp/qmp-shell-wrap
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.qmp import qmp_shell
+
+
+if __name__ == '__main__':
+    qmp_shell.main_wrap()
-- 
2.34.1

[PATCH v3 2/2] python: support recording QMP session to a file

[Daniel P. Berrangé]

When running QMP commands with very large response payloads, it is often
not easy to spot the info you want. If we can save the response to a
file then tools like 'grep' or 'jq' can be used to extract information.

For convenience of processing, we merge the QMP command and response
dictionaries together:

  {
      "arguments": {},
      "execute": "query-kvm",
      "return": {
          "enabled": false,
          "present": true
      }
  }

Example usage

  $ ./scripts/qmp/qmp-shell-wrap -l q.log -p -- ./build/qemu-system-x86_64 -display none
  Welcome to the QMP low-level shell!
  Connected
  (QEMU) query-kvm
  {
      "return": {
          "enabled": false,
          "present": true
      }
  }
  (QEMU) query-mice
  {
      "return": [
          {
              "absolute": false,
              "current": true,
              "index": 2,
              "name": "QEMU PS/2 Mouse"
          }
      ]
  }

 $ jq --slurp '. | to_entries[] | select(.value.execute == "query-kvm") |
               .value.return.enabled' < q.log
   false

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 python/qemu/aqmp/qmp_shell.py | 29 ++++++++++++++++++++++-------
 python/setup.cfg              |  3 +++
 2 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/python/qemu/aqmp/qmp_shell.py b/python/qemu/aqmp/qmp_shell.py
index 185f5066ef..85988d9f2e 100644
--- a/python/qemu/aqmp/qmp_shell.py
+++ b/python/qemu/aqmp/qmp_shell.py
@@ -89,6 +89,7 @@
 from subprocess import Popen
 import sys
 from typing import (
+    IO,
     Iterator,
     List,
     NoReturn,
@@ -170,7 +171,8 @@ class QMPShell(QEMUMonitorProtocol):
     def __init__(self, address: SocketAddrT,
                  pretty: bool = False,
                  verbose: bool = False,
-                 server: bool = False):
+                 server: bool = False,
+                 logfile: Optional[str] = None):
         super().__init__(address, server=server)
         self._greeting: Optional[QMPMessage] = None
         self._completer = QMPCompleter()
@@ -180,6 +182,10 @@ def __init__(self, address: SocketAddrT,
                                       '.qmp-shell_history')
         self.pretty = pretty
         self.verbose = verbose
+        self.logfile = None
+
+        if logfile is not None:
+            self.logfile = open(logfile, "w", encoding='utf-8')
 
     def close(self) -> None:
         # Hook into context manager of parent to save shell history.
@@ -320,11 +326,11 @@ def _build_cmd(self, cmdline: str) -> Optional[QMPMessage]:
         self._cli_expr(cmdargs[1:], qmpcmd['arguments'])
         return qmpcmd
 
-    def _print(self, qmp_message: object) -> None:
+    def _print(self, qmp_message: object, fh: IO[str] = sys.stdout) -> None:
         jsobj = json.dumps(qmp_message,
                            indent=4 if self.pretty else None,
                            sort_keys=self.pretty)
-        print(str(jsobj))
+        print(str(jsobj), file=fh)
 
     def _execute_cmd(self, cmdline: str) -> bool:
         try:
@@ -347,6 +353,9 @@ def _execute_cmd(self, cmdline: str) -> bool:
             print('Disconnected')
             return False
         self._print(resp)
+        if self.logfile is not None:
+            cmd = {**qmpcmd, **resp}
+            self._print(cmd, fh=self.logfile)
         return True
 
     def connect(self, negotiate: bool = True) -> None:
@@ -414,8 +423,9 @@ class HMPShell(QMPShell):
     def __init__(self, address: SocketAddrT,
                  pretty: bool = False,
                  verbose: bool = False,
-                 server: bool = False):
-        super().__init__(address, pretty, verbose, server)
+                 server: bool = False,
+                 logfile: Optional[str] = None):
+        super().__init__(address, pretty, verbose, server, logfile)
         self._cpu_index = 0
 
     def _cmd_completion(self) -> None:
@@ -508,6 +518,8 @@ def main() -> None:
                         help='Verbose (echo commands sent and received)')
     parser.add_argument('-p', '--pretty', action='store_true',
                         help='Pretty-print JSON')
+    parser.add_argument('-l', '--logfile',
+                        help='Save log of all QMP messages to PATH')
 
     default_server = os.environ.get('QMP_SOCKET')
     parser.add_argument('qmp_server', action='store',
@@ -526,7 +538,7 @@ def main() -> None:
         parser.error(f"Bad port number: {args.qmp_server}")
         return  # pycharm doesn't know error() is noreturn
 
-    with shell_class(address, args.pretty, args.verbose) as qemu:
+    with shell_class(address, args.pretty, args.verbose, args.logfile) as qemu:
         try:
             qemu.connect(negotiate=not args.skip_negotiation)
         except ConnectError as err:
@@ -550,6 +562,8 @@ def main_wrap() -> None:
                         help='Verbose (echo commands sent and received)')
     parser.add_argument('-p', '--pretty', action='store_true',
                         help='Pretty-print JSON')
+    parser.add_argument('-l', '--logfile',
+                        help='Save log of all QMP messages to PATH')
 
     parser.add_argument('command', nargs=argparse.REMAINDER,
                         help='QEMU command line to invoke')
@@ -574,7 +588,8 @@ def main_wrap() -> None:
         return  # pycharm doesn't know error() is noreturn
 
     try:
-        with shell_class(address, args.pretty, args.verbose, True) as qemu:
+        with shell_class(address, args.pretty, args.verbose,
+                         True, args.logfile) as qemu:
             with Popen(cmd):
 
                 try:
diff --git a/python/setup.cfg b/python/setup.cfg
index 84c0f33baf..9aeb74aed4 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -114,7 +114,10 @@ ignore_missing_imports = True
 # no Warning level messages displayed, use "--disable=all --enable=classes
 # --disable=W".
 disable=consider-using-f-string,
+        consider-using-with,
+        too-many-arguments,
         too-many-function-args,  # mypy handles this with less false positives.
+        too-many-instance-attributes,
         no-member,  # mypy also handles this better.
 
 [pylint.basic]
-- 
2.34.1

Re: [PATCH v3 0/2] python: a few improvements to qmp-shell

[John Snow]

On Fri, Jan 28, 2022 at 11:12 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> This makes the qmp-shell program a little more pleasant to use when you
> are just trying to spawn a throw-away QEMU process to query some info
> from.
>
> First it introduces a 'qmp-shell-wrap' command that takes a QEMU command
> line instead of QMP socket, and spawns QEMU automatically, so its life
> is tied to that of the shell.
>
> Second it adds ability to log QMP commands/responses to a file that can
> be queried with 'jq' to extract information. This is good for commands
> which return huge JSON docs.
>
> In v3:
>
>  - Add qmp-shell-wrap to setup.cfg entry points
>
> In v2:
>
>  - Unlink unix socket path on exit
>  - Fix default command name
>  - Deal with flake8/pylint warnings
>
> Daniel P. Berrangé (2):
>   python: introduce qmp-shell-wrap convenience tool
>   python: support recording QMP session to a file
>
>  python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
>  python/setup.cfg              |  4 ++
>  scripts/qmp/qmp-shell-wrap    | 11 +++++
>  3 files changed, 96 insertions(+), 7 deletions(-)
>  create mode 100755 scripts/qmp/qmp-shell-wrap
>
> --
> 2.34.1
>
>

Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged these.

--js

(fwiw: using pip, it seems like the wrapper script works just fine. it
appears as though using 'python3 setup.py install' does indeed cause
issues here. I have a patch I'll send soon that discourages the direct
setup.py invocation to avoid frustration in the future.)

Re: [PATCH v3 0/2] python: a few improvements to qmp-shell

[Daniel P. Berrangé]

On Mon, Feb 07, 2022 at 04:05:47PM -0500, John Snow wrote:
> On Fri, Jan 28, 2022 at 11:12 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > This makes the qmp-shell program a little more pleasant to use when you
> > are just trying to spawn a throw-away QEMU process to query some info
> > from.
> >
> > First it introduces a 'qmp-shell-wrap' command that takes a QEMU command
> > line instead of QMP socket, and spawns QEMU automatically, so its life
> > is tied to that of the shell.
> >
> > Second it adds ability to log QMP commands/responses to a file that can
> > be queried with 'jq' to extract information. This is good for commands
> > which return huge JSON docs.
> >
> > In v3:
> >
> >  - Add qmp-shell-wrap to setup.cfg entry points
> >
> > In v2:
> >
> >  - Unlink unix socket path on exit
> >  - Fix default command name
> >  - Deal with flake8/pylint warnings
> >
> > Daniel P. Berrangé (2):
> >   python: introduce qmp-shell-wrap convenience tool
> >   python: support recording QMP session to a file
> >
> >  python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
> >  python/setup.cfg              |  4 ++
> >  scripts/qmp/qmp-shell-wrap    | 11 +++++
> >  3 files changed, 96 insertions(+), 7 deletions(-)
> >  create mode 100755 scripts/qmp/qmp-shell-wrap
> >
> > --
> > 2.34.1
> >
> >
> 
> Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged these.
> 
> --js
> 
> (fwiw: using pip, it seems like the wrapper script works just fine. it
> appears as though using 'python3 setup.py install' does indeed cause
> issues here. I have a patch I'll send soon that discourages the direct
> setup.py invocation to avoid frustration in the future.)

I've only ever used  pip to install from pypi or remote git archives.
How do you use it to install from your local git checkout

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 0/2] python: a few improvements to qmp-shell

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2668 bytes --]

On Tue, Feb 8, 2022, 3:56 AM Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Mon, Feb 07, 2022 at 04:05:47PM -0500, John Snow wrote:
> > On Fri, Jan 28, 2022 at 11:12 AM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> > >
> > > This makes the qmp-shell program a little more pleasant to use when you
> > > are just trying to spawn a throw-away QEMU process to query some info
> > > from.
> > >
> > > First it introduces a 'qmp-shell-wrap' command that takes a QEMU
> command
> > > line instead of QMP socket, and spawns QEMU automatically, so its life
> > > is tied to that of the shell.
> > >
> > > Second it adds ability to log QMP commands/responses to a file that can
> > > be queried with 'jq' to extract information. This is good for commands
> > > which return huge JSON docs.
> > >
> > > In v3:
> > >
> > >  - Add qmp-shell-wrap to setup.cfg entry points
> > >
> > > In v2:
> > >
> > >  - Unlink unix socket path on exit
> > >  - Fix default command name
> > >  - Deal with flake8/pylint warnings
> > >
> > > Daniel P. Berrangé (2):
> > >   python: introduce qmp-shell-wrap convenience tool
> > >   python: support recording QMP session to a file
> > >
> > >  python/qemu/aqmp/qmp_shell.py | 88 ++++++++++++++++++++++++++++++++---
> > >  python/setup.cfg              |  4 ++
> > >  scripts/qmp/qmp-shell-wrap    | 11 +++++
> > >  3 files changed, 96 insertions(+), 7 deletions(-)
> > >  create mode 100755 scripts/qmp/qmp-shell-wrap
> > >
> > > --
> > > 2.34.1
> > >
> > >
> >
> > Great, thanks! I rebased patch 1/2 myself as a courtesy and have staged
> these.
> >
> > --js
> >
> > (fwiw: using pip, it seems like the wrapper script works just fine. it
> > appears as though using 'python3 setup.py install' does indeed cause
> > issues here. I have a patch I'll send soon that discourages the direct
> > setup.py invocation to avoid frustration in the future.)
>
> I've only ever used  pip to install from pypi or remote git archives.
> How do you use it to install from your local git checkout
>

cd ~/src/qemu/python
pip install [-e] [--user] .

See python/README.rst:  https://gitlab.com/jsnow/qemu/-/tree/master/python

I've also sent a patch that disables the broken method (and explains why
it's broken), see

https://patchew.org/QEMU/20220207213039.2278569-1-jsnow@redhat.com/

thanks!
--js


> Regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 4714 bytes --]