About query-qmp-schema

About query-qmp-schema

[Marc-André Lureau]

Hi

The query-qmp-schema returns a stripped-down version, removing types
unused by commands and events. It also renames types with integers.

That makes it impractical for a client to know which values are
acceptable for a generic qom-set commands: qom-list returns the real
"unmasked" type name, and it may be a type that is not even used by
qmp commands/events. A quick test shows the current schema is about
~243kb of json. With all types, it grows to ~273kb. Unmask it grows to
~308kb.

Could we send the full version? Eventually, we could have a "flavour":
"full" argument.

Another option is to compress the schema. A "full-gz" in baes64 would be 51kb.

-- 
Marc-André Lureau

Re: About query-qmp-schema

[Marc-André Lureau]

Hi

On Tue, Apr 21, 2026 at 12:11 PM Marc-André Lureau
<marcandre.lureau@gmail.com> wrote:
>
> Hi
>
> The query-qmp-schema returns a stripped-down version, removing types
> unused by commands and events. It also renames types with integers.
>
> That makes it impractical for a client to know which values are
> acceptable for a generic qom-set commands: qom-list returns the real
> "unmasked" type name, and it may be a type that is not even used by
> qmp commands/events. A quick test shows the current schema is about
> ~243kb of json. With all types, it grows to ~273kb. Unmask it grows to
> ~308kb.
>
> Could we send the full version? Eventually, we could have a "flavour":
> "full" argument.
>
> Another option is to compress the schema. A "full-gz" in baes64 would be 51kb.

After some experiments and some thoughts, here is what I propose:
- Replace QLit representation with a compressed JSON - the JSON is
built-evaluated at compile time to handle the #if conditions and
produce the per-target schema
- At runtime, query-qmp-schema decompresses it and parses the JSON to
keep compatibility with existing commands, applying filtering and
typename substitution
- Add a query-qmp-schema-data, which returns the unmodified compressed
JSON schema blob (~35kb in base64)
- additionally, I'd like a way for the client to query the
documentation. Having a similar query-qmp-schema-doc would result in
shipping ~135kb of gz content, which could be shared by all binaries
(since the doc doesn't have configuration).

Please comment, thanks

-- 
Marc-André Lureau

Re: About query-qmp-schema

[Markus Armbruster]

[-- Attachment #1: Type: text/plain, Size: 9720 bytes --]

Marc-André Lureau <marcandre.lureau@gmail.com> writes:

> Hi
>
> On Tue, Apr 21, 2026 at 12:11 PM Marc-André Lureau
> <marcandre.lureau@gmail.com> wrote:
>>
>> Hi
>>
>> The query-qmp-schema returns a stripped-down version, removing types
>> unused by commands and events. It also renames types with integers.
>>
>> That makes it impractical for a client to know which values are
>> acceptable for a generic qom-set commands:

Wanting to know which values are acceptable for qom-set is fair.  The
means we provide for that (QOM introspection) are flawed.

>>                                            qom-list returns the real
>> "unmasked" type name,

If you're lucky.  More on that below.

>>                       and it may be a type that is not even used by
>> qmp commands/events.

Yes, because query-qmp-schema returns the *QMP* schema.

>>                      A quick test shows the current schema is about
>> ~243kb of json. With all types, it grows to ~273kb. Unmask it grows to
>> ~308kb.
>>
>> Could we send the full version? Eventually, we could have a "flavour":
>> "full" argument.
>>
>> Another option is to compress the schema. A "full-gz" in baes64 would be 51kb.
>
> After some experiments and some thoughts, here is what I propose:
> - Replace QLit representation with a compressed JSON - the JSON is
> built-evaluated at compile time to handle the #if conditions and
> produce the per-target schema

I wouldn't mind QLit to go.  I don't want to lose the ability to inspect
the generated introspection value, though.

We used to generate JSON returned by query-qmp-schema as a string.  Easy
to read for me.

Commit 7d0f982bfb (qapi: generate a literal qobject for introspection)
replaced it with QLit.  Harder to read, but still workable, especially
with 8c643361eeb (qapi: Add comments to aid debugging generated
introspection).

Compressed JSON would be unreadable.  Generate a comment with the
uncompressed JSON?

> - At runtime, query-qmp-schema decompresses it and parses the JSON to
> keep compatibility with existing commands, applying filtering and
> typename substitution
> - Add a query-qmp-schema-data, which returns the unmodified compressed
> JSON schema blob (~35kb in base64)

If we expose QAPI schema type names externally, they become ABI.  The
design decision not to has enabled schema refactorings time and again.
Changing that feels like a bad idea.

Instead of making QAPI introspection worse to match QOM introspection,
let's make QOM introspection better to match QAPI introspection: use the
same masked type names.

> - additionally, I'd like a way for the client to query the
> documentation. Having a similar query-qmp-schema-doc would result in
> shipping ~135kb of gz content, which could be shared by all binaries
> (since the doc doesn't have configuration).

John Snow mentioned a desire to access QMP documentation via QMP.  I
figure it could use a bit more structure than dumping the generated HTML
or the ReST-with-extensions source.  We should talk, but let's put this
aside while we discuss QOM introspection.

> Please comment, thanks

First, the type names returned by qom-list and qom-list-properties are
problematic.  Here's the documentation:

    # @type: the type of the property.  This will typically come in one of
    #     four forms:
    #
    #     1) A primitive type such as 'u8', 'u16', 'bool', 'str', or
    #        'double'.  These types are mapped to the appropriate JSON
    #        type.
    #
    #     2) A child type in the form 'child<subtype>' where subtype is a
    #        qdev device type name.  Child properties create the
    #        composition tree.
    #
    #     3) A link type in the form 'link<subtype>' where subtype is a
    #        qdev device type name.  Link properties form the device model
    #        graph.

Pretty much every sentence is misleading or wrong:

* Type names "come in four forms", but we document just three.  The
  missing fourth form is a type that's neither "primitive", "child", nor
  "link".

* The primitive types are "defined" by listing a few examples, and
  waving hands towards JSON.  Srsly?!?

* A "child type" can refer to any object type, not just device types.

* Same for "link type".

So what are the *actual* types?  Find attached a rough Python script to
find most of the type names used for QOM properties in
qemu-system-x86_64.

"Most of" because it can't find properties created after object
initialization, and it needs to skip a few objects that crash on
introspection (will report separately).  Good enough for this memo.

Of type type names the script finds, the following are returned by
query-qmp-schema (checked with grep -E '= (name1|name2|...)'
qapi/qapi-introspect.c):

    BlockdevOnError
    DisplayGLMode
    GrabToggleKeys
    GuestPanicInformation
    GuestPanicInformationHyperV
    GuestPanicInformationS390
    GuestPanicInformationSev
    GuestPanicInformationTdx
    GuestPanicInformationType
    HostMemPolicy
    MigMode
    MultiFDCompression
    NetFilterDirection
    OnOffAuto
    QAuthZListPolicy
    QAuthZListRule
    QCryptoSecretFormat
    QCryptoTLSCredsEndpoint
    SgxEPCDeviceInfo
    SgxEPCDeviceInfoWrapper
    SocketAddress
    SocketAddressLegacy
    SocketAddressType
    StrOrNull
    ThrottleLimits
    ZeroPageDetection
    bool
    int
    str

The following aren't, but they exist in the schema (checked with grep
-Eh '# @(name1|name2|...)' qapi/*json):

    BiosAtaTranslation
    BootConfiguration
    CXLFixedMemoryWindowOptions
    FloppyDriveType
    GranuleMode
    IOThreadVirtQueueMapping
    LostTickPolicy
    MemorySizeConfiguration
    OffAutoPCIBAR
    PCIELinkSpeed
    PCIELinkWidth
    SMPConfiguration
    SgxEPC
    SgxEPCDeviceInfo
    SgxEPCDeviceInfoWrapper
    SgxEPCProperties
    SmpCachePropertiesWrapper
    VirtIOGPUOutput
    X86CPUFeatureWordInfo

The following are built-in QAPI types we intentionally do not expose in
query-qmp-schema:

    int32
    size
    uint16
    uint32
    uint64
    uint8

The following aren't QAPI types all:

    CXLFixedMemoryWindow
    NotifyVMexitOption
    PCSouthBridgeOption
    QemuUUID
    SpdmTransportType
    busnr
    guest statistics
    int32_t
    list
    on|off|split
    string
    uint32_t
    uint64_t

Some of them are C types, some of them aren't even that.

Nothing ensures the type names returned by qom-list and
qom-list-properties correspond to anything.  It's all convention and
developer diligence (or lack thereof).  This makes these type names
*unreliable*.

Nothing requires QOM properties to be of a QAPI type.  Developers are
free to use QAPI visitors as they see fit.

Your proposal does not address these issues.

Alan Perlis's epigram applies: "Beware of the Turing tar-pit in which
everything is possible but nothing of interest is easy."

We should make defining a QOM property of QAPI type easy.  Properties
defined that way should get the correct type name automatically.

Fixing all properties in one go feels impractical.  Therefore, type
names remain unreliable.  I propose to add another, reliable member next
to it: the masked QAPI name.  Present only for properties defined the
QAPI way.


Example run of my script:

$ PYTHONPATH=python python/qemu/utils/qom-type-names.py
216 child<>
982 link<>
103003 other
property types: ['BiosAtaTranslation', 'BlockdevOnError', 'BootConfiguration', 'CXLFixedMemoryWindow', 'DisplayGLMode', 'FloppyDriveType', 'GrabToggleKeys', 'GranuleMode', 'GuestPanicInformation', 'HostMemPolicy', 'IOThreadVirtQueueMapping', 'LostTickPolicy', 'MemorySizeConfiguration', 'MigMode', 'MultiFDCompression', 'NetFilterDirection', 'NotifyVMexitOption', 'OffAutoPCIBAR', 'OnOffAuto', 'PCIELinkSpeed', 'PCIELinkWidth', 'PCSouthBridgeOption', 'QAuthZListPolicy', 'QAuthZListRule', 'QCryptoSecretFormat', 'QCryptoTLSCredsEndpoint', 'QemuUUID', 'SMPConfiguration', 'SgxEPC', 'SmpCachePropertiesWrapper', 'SocketAddress', 'SpdmTransportType', 'StrOrNull', 'ThrottleLimits', 'VirtIOGPUOutput', 'X86CPUFeatureWordInfo', 'ZeroPageDetection', 'bool', 'busnr', 'guest statistics', 'int', 'int32', 'int32_t', 'list', 'on|off|split', 'size', 'str', 'string', 'uint16', 'uint32', 'uint32_t', 'uint64', 'uint64_t', 'uint8']
good refs: ['PCI', 'base-xhci', 'bus', 'can-bus', 'cfi.pflash01', 'confidential-guest-support', 'cryptodev-backend', 'cxl-upstream', 'device', 'esp', 'gpex-root', 'hotplug-handler', 'i2c-bus', 'i2c-ddc', 'i8042', 'iommufd', 'iothread', 'ipmi-bmc', 'irq', 'mc146818rtc', 'mch', 'memory-backend', 'memory-region', 'nitro-vsock-bus', 'nvme-subsys', 'piix3-ide', 'piix4-ide', 'pit-common', 'ps2-kbd', 'ps2-mouse', 'rng-backend', 'sdhci-bus', 'serial', 'thread-context', 'vhost-scsi', 'vhost-user-blk', 'vhost-user-fs-device', 'vhost-user-gpio-device', 'vhost-user-gpu', 'vhost-user-i2c-device', 'vhost-user-input', 'vhost-user-rng', 'vhost-user-scsi', 'vhost-user-snd', 'vhost-user-spi-device', 'vhost-user-test-device', 'vhost-user-vsock-device', 'vhost-vdpa-device', 'vhost-vsock-device', 'virtio-9p-device', 'virtio-balloon-device', 'virtio-blk-device', 'virtio-crypto-device', 'virtio-gpu-device', 'virtio-gpu-gl-device', 'virtio-input-host-device', 'virtio-iommu-device', 'virtio-keyboard-device', 'virtio-mem', 'virtio-mouse-device', 'virtio-multitouch-device', 'virtio-net-device', 'virtio-pmem', 'virtio-rng-device', 'virtio-scsi-device', 'virtio-serial-device', 'virtio-sound-device', 'virtio-tablet-device']
dangling refs: ['memory-backend-epc']


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: qom-type-names.py --]
[-- Type: text/x-python, Size: 1701 bytes --]

#!/usr/bin/env python3

import sys

from qemu.machine import QEMUMachine

def main():
    obj_types = set()
    refs = set()
    prop_types = set()
    nchild = nlink = nother = 0
    broken_obj_types = {
        'colo-compare',
        'cryptodev-backend-lkcf',
        'memory-region-portio-list',
        'qemu-fixed-text-console',
        'qemu-graphic-console',
        'qio-net-listener',
        'x-remote-object',
    }
    vm = QEMUMachine(binary='bld-x86/qemu-system-x86_64')
    vm.launch()

    resp = vm.qmp('qom-list-types', {'abstract': True})
    otinfos = resp['return']
    concrete_obj_types = set([info['name']
                              for info in otinfos if not info.get('abstract')])
    obj_types = set([info['name']
                     for info in otinfos])

    for ot in concrete_obj_types - broken_obj_types:
        resp = vm.qmp('qom-list-properties', {'typename': ot})
        opinfos = resp['return']
        for info in opinfos:
            pt = info['type']
            if pt.startswith('child<'):
                nchild += 1
                refs.add(pt[6:-1])
            elif pt.startswith('link<'):
                nlink += 1
                refs.add(pt[5:-1])
            else:
                if pt.endswith('List'):
                    pt = pt[:-4]
                nother += 1
                prop_types.add(pt)
                ref = None

    print(f'{nchild} child<>')
    print(f'{nlink} link<>')
    print(f'{nother} other')
    print(f'property types: {sorted(prop_types)}')
    print(f'good refs: {sorted(refs & obj_types)}')
    print(f'dangling refs: {sorted(refs - obj_types)}')

    vm.shutdown()

if __name__ == '__main__':
    sys.exit(main())

Re: About query-qmp-schema

[Daniel P. Berrangé]

On Wed, Apr 22, 2026 at 01:35:31PM +0200, Markus Armbruster wrote:
> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
> 

snip

> >>                       and it may be a type that is not even used by
> >> qmp commands/events.
> 
> Yes, because query-qmp-schema returns the *QMP* schema.
> 
> >>                      A quick test shows the current schema is about
> >> ~243kb of json. With all types, it grows to ~273kb. Unmask it grows to
> >> ~308kb.
> >>
> >> Could we send the full version? Eventually, we could have a "flavour":
> >> "full" argument.
> >>
> >> Another option is to compress the schema. A "full-gz" in baes64 would be 51kb.


> 
> > - At runtime, query-qmp-schema decompresses it and parses the JSON to
> > keep compatibility with existing commands, applying filtering and
> > typename substitution
> > - Add a query-qmp-schema-data, which returns the unmodified compressed
> > JSON schema blob (~35kb in base64)
> 
> If we expose QAPI schema type names externally, they become ABI.  The
> design decision not to has enabled schema refactorings time and again.
> Changing that feels like a bad idea.

Yeah the ability to change our type names has been surprisingly
useful and common, in fact because it allows more than just
type name changes. A single type exposed in QMP can map to
multiple types in QAPI, thanks to the inheritance from the
'base' keyword.  That inheritance is flattended in what we
expose in QMP.



With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|