[Qemu-devel] [RFC PATCH] qmp.py: Fix exception parsing partial JSON

[Qemu-devel] [RFC PATCH] qmp.py: Fix exception parsing partial JSON

[Philippe Mathieu-Daudé]

The readline() call returns partial data.
Keep appending until the JSON buffer is complete.

This fixes:

    $ scripts/qmp/qmp-shell -v -p /tmp/qmp.sock
    Traceback (most recent call last):
      File "scripts/qmp/qmp-shell", line 456, in <module>
        main()
      File "scripts/qmp/qmp-shell", line 441, in main
        qemu.connect(negotiate)
      File "scripts/qmp/qmp-shell", line 284, in connect
        self._greeting = super(QMPShell, self).connect(negotiate)
      File "scripts/qmp/qmp.py", line 143, in connect
        return self.__negotiate_capabilities()
      File "scripts/qmp/qmp.py", line 71, in __negotiate_capabilities
        greeting = self.__json_read()
      File "scripts/qmp/qmp.py", line 85, in __json_read
        resp = json.loads(data)
      File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
        obj, end = self.scan_once(s, idx)
    ValueError: Expecting object: line 1 column 3 (char 2)

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
Daniel suggested this is due to blocking i/o.

I'm sure there is a nicer/more pythonic way to do this, but this works for me,
sorry :)

 scripts/qmp/qmp.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
index 5c8cf6a056..e8b55dfcc4 100644
--- a/scripts/qmp/qmp.py
+++ b/scripts/qmp/qmp.py
@@ -78,11 +78,16 @@ class QEMUMonitorProtocol(object):
         raise QMPCapabilitiesError
 
     def __json_read(self, only_event=False):
+        data = ""
         while True:
-            data = self.__sockfile.readline()
-            if not data:
+            tmp = self.__sockfile.readline()
+            if not tmp:
                 return
-            resp = json.loads(data)
+            data += tmp
+            try:
+                resp = json.loads(data)
+            except ValueError:
+                continue
             if 'event' in resp:
                 self.logger.debug("<<< %s", resp)
                 self.__events.append(resp)
-- 
2.17.1

Re: [Qemu-devel] [RFC PATCH] qmp.py: Fix exception parsing partial JSON

[Daniel P. Berrangé]

On Wed, Jun 06, 2018 at 01:53:19PM -0300, Philippe Mathieu-Daudé wrote:
> The readline() call returns partial data.
> Keep appending until the JSON buffer is complete.
> 
> This fixes:
> 
>     $ scripts/qmp/qmp-shell -v -p /tmp/qmp.sock
>     Traceback (most recent call last):
>       File "scripts/qmp/qmp-shell", line 456, in <module>
>         main()
>       File "scripts/qmp/qmp-shell", line 441, in main
>         qemu.connect(negotiate)
>       File "scripts/qmp/qmp-shell", line 284, in connect
>         self._greeting = super(QMPShell, self).connect(negotiate)
>       File "scripts/qmp/qmp.py", line 143, in connect
>         return self.__negotiate_capabilities()
>       File "scripts/qmp/qmp.py", line 71, in __negotiate_capabilities
>         greeting = self.__json_read()
>       File "scripts/qmp/qmp.py", line 85, in __json_read
>         resp = json.loads(data)
>       File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
>         return _default_decoder.decode(s)
>       File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
>         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>       File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
>         obj, end = self.scan_once(s, idx)
>     ValueError: Expecting object: line 1 column 3 (char 2)
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> Daniel suggested this is due to blocking i/o.
> 
> I'm sure there is a nicer/more pythonic way to do this, but this works for me,
> sorry :)
> 
>  scripts/qmp/qmp.py | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
> index 5c8cf6a056..e8b55dfcc4 100644
> --- a/scripts/qmp/qmp.py
> +++ b/scripts/qmp/qmp.py
> @@ -78,11 +78,16 @@ class QEMUMonitorProtocol(object):
>          raise QMPCapabilitiesError
>  
>      def __json_read(self, only_event=False):
> +        data = ""
>          while True:
> -            data = self.__sockfile.readline()
> -            if not data:
> +            tmp = self.__sockfile.readline()
> +            if not tmp:
>                  return
> -            resp = json.loads(data)
> +            data += tmp
> +            try:
> +                resp = json.loads(data)
> +            except ValueError:
> +                continue

Downside is that we'll blindly loop if QEMU ever sends
malformed data, but given the use context of this QMP
shell that's not too serious.

>              if 'event' in resp:
>                  self.logger.debug("<<< %s", resp)
>                  self.__events.append(resp)

If we get an event, we'll continue on the loop and append
to 'data' again, so you must blank out data to "" after
json.loads() succeeeds.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|