Re: Does Libvirt's json parser support single quoted string in qmp json string?

Re: Does Libvirt's json parser support single quoted string in qmp json string?

[Eric Blake]

[adding Markus]

On 2/3/20 4:13 AM, Daniel P. Berrangé wrote:
> On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote:
>> On 1/31/20 4:38 AM, Peter Luo wrote:
>>
>>> error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text.
>>>
>>>             , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo
>>>
>>>                        (right here) ------^
>>>
>>
>> qemu's QMP language has an extension where it accepts 'string' in addition
>> to the JSON "string".  But it uses a hand-rolled parser, so it can do
>> whatever it wants.
> 
> Can we deprecate & remove this extension in QEMU ?

We could start a deprecation clock, if desired, but I don't know how 
many external users would be impacted (at least qemu's testsuite heavily 
relies on the extension of single quotes).

Are there any third-party libraries that parse JSON5? 
https://json5.org/ documents that one of the nice features of JSON5 is 
that single quotes work out of the box.  Right now, even though qemu 
does NOT parse the same thing as pure JSON (which is 
https://datatracker.ietf.org/doc/rfc8259/), it does appear to parse a 
subset of JSON5, where the additional features of JSON5 (such as 
allowing a trailing comma, escaped newlines within strings, allowing 
bareword key:"value" syntax, a larger set of accepted numeric values, 
allowing comments) don't seem that onerous.

> 
> If we're going to call QMP protocol JSON, then IMHO QEMU should follow
> the JSON spec as closely as possible, without extensions.

As it is, qemu explicitly does NOT parse all valid JSON - it rejects 
non-ASCII bytes (whether as UTF-8 or as \u escape sequences) in strings. 
  So the fact that it already extensions in some places and limitations 
in others is a burden for clients to be aware of.

> 
> On the flip side, if we're going to support extensions like single quoting,
> then we should make it clear to applications that this is not really JSON
> and that they need to provide an impl that is 100% matching QEMU's dialect.
> This effectively means they need just import a copy of QEMU's code.
> 
> Regards,
> Daniel
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Does Libvirt's json parser support single quoted string in qmp json string?

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> [adding Markus]
>
> On 2/3/20 4:13 AM, Daniel P. Berrangé wrote:
>> On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote:
>>> On 1/31/20 4:38 AM, Peter Luo wrote:
>>>
>>>> error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text.
>>>>
>>>>             , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo
>>>>
>>>>                        (right here) ------^
>>>>
>>>
>>> qemu's QMP language has an extension where it accepts 'string' in addition
>>> to the JSON "string".  But it uses a hand-rolled parser, so it can do
>>> whatever it wants.
>>
>> Can we deprecate & remove this extension in QEMU ?

I think deprecating the extension makes sense only if we can actually
kill it.

We could try to make the extension opt-in, and have only the intermal
users opt in.  Can't say offhand whether that's practical.

> We could start a deprecation clock, if desired, but I don't know how
> many external users would be impacted (at least qemu's testsuite
> heavily relies on the extension of single quotes).

Looks like this:

    rsp = qdict_from_jsonf_nofail("{ 'error': { 'class': %s, 'desc': %s } }",
                                  QapiErrorClass_str(error_get_class(err)),
                                  error_get_pretty(err));

Without the extension, we'd suffer from a mild case of leaning toothpick
syndrome:

    rsp = qdict_from_jsonf_nofail("{ \"error\": { \"class\": %s, \"desc\": %s } }",
                                  QapiErrorClass_str(error_get_class(err)),
                                  error_get_pretty(err));

I intentionally picked an example outside tests/ :)

> Are there any third-party libraries that parse JSON5?
> https://json5.org/ documents that one of the nice features of JSON5 is
> that single quotes work out of the box.  Right now, even though qemu
> does NOT parse the same thing as pure JSON (which is
> https://datatracker.ietf.org/doc/rfc8259/), it does appear to parse a
> subset of JSON5, where the additional features of JSON5 (such as
> allowing a trailing comma, escaped newlines within strings, allowing
> bareword key:"value" syntax, a larger set of accepted numeric values,
> allowing comments) don't seem that onerous.

JSON5 is rather niche, I'm afraid.

>> If we're going to call QMP protocol JSON, then IMHO QEMU should follow
>> the JSON spec as closely as possible, without extensions.
>
> As it is, qemu explicitly does NOT parse all valid JSON - it rejects
> non-ASCII bytes (whether as UTF-8 or as \u escape sequences) in
> strings. So the fact that it already extensions in some places and
> limitations in others is a burden for clients to be aware of.

Not true; I think you're confusing the QAPI schema parser
scripts/qapi/parser.py with the JSON parser qobject/json-*.[ch].

Quoting json-lexer.c:

 * Extensions over RFC 8259:
 * - Extra escape sequence in strings:
 *   0x27 (apostrophe) is recognized after escape, too
 * - Single-quoted strings:
 *   Like double-quoted strings, except they're delimited by %x27
 *   (apostrophe) instead of %x22 (quotation mark), and can't contain
 *   unescaped apostrophe, but can contain unescaped quotation mark.
 * - Interpolation, if enabled:
 *   The lexer accepts %[A-Za-z0-9]*, and leaves rejecting invalid
 *   ones to the parser.

Makes use of RFC 8259 section 9. Parsers: "A JSON parser MAY accept
non-JSON forms or extensions."

The QAPI schema parser indeed restricts strings to printable ASCII.  The
QAPI schema language is so bastardized, I'm not sure how much RFC 8259
still matters, but if you think it does, then consider section
9. Parsers: "An implementation may set limits on the length and
character contents of strings."

>> On the flip side, if we're going to support extensions like single quoting,
>> then we should make it clear to applications that this is not really JSON
>> and that they need to provide an impl that is 100% matching QEMU's dialect.
>> This effectively means they need just import a copy of QEMU's code.

To the best of my knowledge, the JSON parser interprets any valid strict
JSON input in accordance to RFC 8259.  In other words, you don't notice
the extensions unless you use them, or rely on invalid strict JSON to be
rejected.

Peter Luo's input uses one of QEMU's JSON parser's extensions like this:

    "base":'json:{...}'

This is not valid strict JSON.  Libvirt's JSON parser doesn't accept it.

The problem is not presence of extensions in QEMU, it's the use of these
extensions in input for libvirt.  Removing the extensions from QEMU will
not affect the error.  Removing their use from the input will.

Re: Does Libvirt's json parser support single quoted string in qmp json string?

[Daniel P. Berrangé]

On Tue, Feb 04, 2020 at 09:11:27AM +0100, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
> > [adding Markus]
> >
> > On 2/3/20 4:13 AM, Daniel P. Berrangé wrote:
> >> On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote:
> >>> On 1/31/20 4:38 AM, Peter Luo wrote:
> >>>
> >>>> error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text.
> >>>>
> >>>>             , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo
> >>>>
> >>>>                        (right here) ------^
> >>>>
> >>>
> >>> qemu's QMP language has an extension where it accepts 'string' in addition
> >>> to the JSON "string".  But it uses a hand-rolled parser, so it can do
> >>> whatever it wants.
> >>
> >> Can we deprecate & remove this extension in QEMU ?

[snip]

> >> On the flip side, if we're going to support extensions like single quoting,
> >> then we should make it clear to applications that this is not really JSON
> >> and that they need to provide an impl that is 100% matching QEMU's dialect.
> >> This effectively means they need just import a copy of QEMU's code.
> 
> To the best of my knowledge, the JSON parser interprets any valid strict
> JSON input in accordance to RFC 8259.  In other words, you don't notice
> the extensions unless you use them, or rely on invalid strict JSON to be
> rejected.
> 
> Peter Luo's input uses one of QEMU's JSON parser's extensions like this:
> 
>     "base":'json:{...}'
> 
> This is not valid strict JSON.  Libvirt's JSON parser doesn't accept it.
> 
> The problem is not presence of extensions in QEMU, it's the use of these
> extensions in input for libvirt.  Removing the extensions from QEMU will
> not affect the error.  Removing their use from the input will.

The issue that I see is that QEMU accepts this input string when it
parsers JSON provided by the user.  This in turn means the user has
an expectation that other tools based on QEMU will accept this exact
same document. This is not the case because the other tools are
using a stricter impl of JSON. 

IOW QEMU's extensions have mislead the users into believing their 
JSON input is valid for any tool based on QEMU. 

Thus overall I think it would be beneficial for any places where
QEMU accepts JSON from external users or apps, to be restricted to
common JSON syntax only, without any QEMU specific extensions.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Does Libvirt's json parser support single quoted string in qmp json string?

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Feb 04, 2020 at 09:11:27AM +0100, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>> > [adding Markus]
>> >
>> > On 2/3/20 4:13 AM, Daniel P. Berrangé wrote:
>> >> On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote:
>> >>> On 1/31/20 4:38 AM, Peter Luo wrote:
>> >>>
>> >>>> error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text.
>> >>>>
>> >>>>             , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo
>> >>>>
>> >>>>                        (right here) ------^
>> >>>>
>> >>>
>> >>> qemu's QMP language has an extension where it accepts 'string' in addition
>> >>> to the JSON "string".  But it uses a hand-rolled parser, so it can do
>> >>> whatever it wants.
>> >>
>> >> Can we deprecate & remove this extension in QEMU ?
>
> [snip]
>
>> >> On the flip side, if we're going to support extensions like single quoting,
>> >> then we should make it clear to applications that this is not really JSON
>> >> and that they need to provide an impl that is 100% matching QEMU's dialect.
>> >> This effectively means they need just import a copy of QEMU's code.
>> 
>> To the best of my knowledge, the JSON parser interprets any valid strict
>> JSON input in accordance to RFC 8259.  In other words, you don't notice
>> the extensions unless you use them, or rely on invalid strict JSON to be
>> rejected.
>> 
>> Peter Luo's input uses one of QEMU's JSON parser's extensions like this:
>> 
>>     "base":'json:{...}'
>> 
>> This is not valid strict JSON.  Libvirt's JSON parser doesn't accept it.
>> 
>> The problem is not presence of extensions in QEMU, it's the use of these
>> extensions in input for libvirt.  Removing the extensions from QEMU will
>> not affect the error.  Removing their use from the input will.
>
> The issue that I see is that QEMU accepts this input string when it
> parsers JSON provided by the user.  This in turn means the user has
> an expectation that other tools based on QEMU will accept this exact
> same document. This is not the case because the other tools are
> using a stricter impl of JSON. 
>
> IOW QEMU's extensions have mislead the users into believing their 
> JSON input is valid for any tool based on QEMU. 
>
> Thus overall I think it would be beneficial for any places where
> QEMU accepts JSON from external users or apps, to be restricted to
> common JSON syntax only, without any QEMU specific extensions.

I don't think the JSON extensions are much of a problem.  But I also
don't think they provide much value in the external interface.  Feel
free to post patches that deprecate them there.

Aside: deprecating stuff in QMP is awkward, because we don't have a good
way to tell users.  We do it anyway.

Re: Does Libvirt's json parser support single quoted string in qmp json string?

[Daniel P. Berrangé]

On Tue, Feb 04, 2020 at 09:11:27AM +0100, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
> > [adding Markus]
> >
> > On 2/3/20 4:13 AM, Daniel P. Berrangé wrote:
> >> On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote:
> >>> On 1/31/20 4:38 AM, Peter Luo wrote:
> >>>
> >>>> error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text.
> >>>>
> >>>>             , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo
> >>>>
> >>>>                        (right here) ------^
> >>>>
> >>>
> >>> qemu's QMP language has an extension where it accepts 'string' in addition
> >>> to the JSON "string".  But it uses a hand-rolled parser, so it can do
> >>> whatever it wants.
> >>
> >> Can we deprecate & remove this extension in QEMU ?
> 
> I think deprecating the extension makes sense only if we can actually
> kill it.
> 
> We could try to make the extension opt-in, and have only the intermal
> users opt in.  Can't say offhand whether that's practical.
> 
> > We could start a deprecation clock, if desired, but I don't know how
> > many external users would be impacted (at least qemu's testsuite
> > heavily relies on the extension of single quotes).
> 
> Looks like this:
> 
>     rsp = qdict_from_jsonf_nofail("{ 'error': { 'class': %s, 'desc': %s } }",
>                                   QapiErrorClass_str(error_get_class(err)),
>                                   error_get_pretty(err));
> 
> Without the extension, we'd suffer from a mild case of leaning toothpick
> syndrome:
> 
>     rsp = qdict_from_jsonf_nofail("{ \"error\": { \"class\": %s, \"desc\": %s } }",
>                                   QapiErrorClass_str(error_get_class(err)),
>                                   error_get_pretty(err));
> 
> I intentionally picked an example outside tests/ :)

This is purely QEMU internal code. That is fine to use whatever
extensions are desired, as it isn't exposed to end users or mgmt
apps. 

The harm from JSON extensions comes in places where our public API
accepts it - CLI, QMP, JSON inside disk images, etc, as those are
all places where interoperability with third party apps is relevant


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Does Libvirt's json parser support single quoted string in qmp json string?

[John Snow]

On 2/3/20 11:27 AM, Eric Blake wrote:
> [adding Markus]
> 
> On 2/3/20 4:13 AM, Daniel P. Berrangé wrote:
>> On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote:
>>> On 1/31/20 4:38 AM, Peter Luo wrote:
>>>
>>>> error: internal error: cannot parse json {"execute": "block-commit",
>>>> "arguments": { "device": "drive-virtio-disk2", "job-id": "job100",
>>>> "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}',
>>>> "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}:
>>>> lexical error: invalid char in json text.
>>>>
>>>>             , "job-id": "job100",
>>>> "base":'json:{"encrypt.key-secret":"vo
>>>>
>>>>                        (right here) ------^
>>>>
>>>
>>> qemu's QMP language has an extension where it accepts 'string' in
>>> addition
>>> to the JSON "string".  But it uses a hand-rolled parser, so it can do
>>> whatever it wants.
>>
>> Can we deprecate & remove this extension in QEMU ?
> 
> We could start a deprecation clock, if desired, but I don't know how
> many external users would be impacted (at least qemu's testsuite heavily
> relies on the extension of single quotes).
> 
> Are there any third-party libraries that parse JSON5? https://json5.org/
> documents that one of the nice features of JSON5 is that single quotes
> work out of the box.  Right now, even though qemu does NOT parse the
> same thing as pure JSON (which is
> https://datatracker.ietf.org/doc/rfc8259/), it does appear to parse a
> subset of JSON5, where the additional features of JSON5 (such as
> allowing a trailing comma, escaped newlines within strings, allowing
> bareword key:"value" syntax, a larger set of accepted numeric values,
> allowing comments) don't seem that onerous.
> 
>>
>> If we're going to call QMP protocol JSON, then IMHO QEMU should follow
>> the JSON spec as closely as possible, without extensions.
> 
> As it is, qemu explicitly does NOT parse all valid JSON - it rejects
> non-ASCII bytes (whether as UTF-8 or as \u escape sequences) in strings.
>  So the fact that it already extensions in some places and limitations
> in others is a burden for clients to be aware of.
> 
>>
>> On the flip side, if we're going to support extensions like single
>> quoting,
>> then we should make it clear to applications that this is not really JSON
>> and that they need to provide an impl that is 100% matching QEMU's
>> dialect.
>> This effectively means they need just import a copy of QEMU's code.
>>
>> Regards,
>> Daniel
>>
> 

More anecdata: qmp-shell will also accept either strict JSON or what it
calls "fuzzy" JSON, which allows single quotes, Python spellings for
boolean constants, and other deviations.

However, when using pretty-print, it will always show you the *strict*
json form of your input.

If we are confusing users, that's bad; but there's some use for the
single quotes in our code base. Following a standard would be nice.

JSON5 might be "niche", but so what? If we happen to more closely adhere
to JSON5 than vanilla JSON, we could still make sure we adhere to it and
we get to say that we accept "strict JSON5." This is at least unambiguous.

I'm more inclined to head towards JSON5 instead of heading back to
strict JSON.

--js