[PATCH V1 0/6] fast qom tree get

[PATCH V1 0/6] fast qom tree get

[Steve Sistare]

Using qom-list and qom-get to get all the nodes and property values in a
QOM tree can take multiple seconds because it requires 1000's of individual
QOM requests.  Some managers fetch the entire tree or a large subset
of it when starting a new VM, and this cost is a substantial fraction of
start up time.

To reduce this cost, consider QAPI calls that fetch more information in
each call:
  * qom-list-get: given a path, return a list of properties and values.
  * qom-list-getv: given a list of paths, return a list of properties and
    values for each path.
  * qom-tree-get: given a path, return all descendant nodes rooted at that
    path, with properties and values for each.

In all cases, a returned property is represented by ObjectPropertyValue,
with fields name, type, value, and error.  If an error occurs when reading
a value, the value field is omitted, and the error message is returned in the
the error field.  Thus an error for one property will not cause a bulk fetch
operation to fail.

To evaluate each method, I modified scripts/qmp/qom-tree to use the method,
verified all methods produce the same output, and timed each using:

  qemu-system-x86_64 -display none \
    -chardev socket,id=monitor0,path=/tmp/vm1.sock,server=on,wait=off \
    -mon monitor0,mode=control &

  time qom-tree -s /tmp/vm1.sock > /dev/null

I only measured once per method, but the variation is low after a warm up run.
The 'real - user - sys' column is a proxy for QEMU CPU time.

method               real(s)   user(s)   sys(s)  (real - user - sys)(s)
qom-list / qom-get   2.048     0.932     0.057   1.059
qom-list-get         0.402     0.230     0.029   0.143
qom-list-getv        0.200     0.132     0.015   0.053
qom-tree-get         0.143     0.123     0.012   0.008

qom-tree-get is the clear winner, reducing elapsed time by a factor of 14X,
and reducing QEMU CPU time by 132X.

qom-list-getv is slower when fetching the entire tree, but can beat
qom-tree-get when only a subset of the tree needs to be fetched (not shown).
qom-list-get is shown for comparison only, and is not included in this series.

Steve Sistare (6):
  qom: qom_resolve_path
  qom: qom-tree-get
  python: use qom-tree-get
  tests/qtest/qom-test: unit test for qom-tree-get
  qom: qom-list-getv
  tests/qtest/qom-test: unit test for qom-list-getv

 python/qemu/utils/qom.py        |  36 +++++------
 python/qemu/utils/qom_common.py |  50 ++++++++++++++++
 qapi/qom.json                   |  93 ++++++++++++++++++++++++++++
 qom/qom-qmp-cmds.c              | 130 ++++++++++++++++++++++++++++++++++++++--
 tests/qtest/qom-test.c          | 116 +++++++++++++++++++++++++++++++++++
 5 files changed, 401 insertions(+), 24 deletions(-)

base-commit: 354925d42252f6f36a9e1e4a6b929aaafb2eaf45
-- 
1.8.3.1

[PATCH V1 1/6] qom: qom_resolve_path

[Steve Sistare]

Factor out a helper to resolve the user's path and print error messages.
No functional change.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 qom/qom-qmp-cmds.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/qom/qom-qmp-cmds.c b/qom/qom-qmp-cmds.c
index e866547..293755f 100644
--- a/qom/qom-qmp-cmds.c
+++ b/qom/qom-qmp-cmds.c
@@ -28,15 +28,11 @@
 #include "qom/object_interfaces.h"
 #include "qom/qom-qobject.h"
 
-ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
+static Object *qom_resolve_path(const char *path, Error **errp)
 {
-    Object *obj;
     bool ambiguous = false;
-    ObjectPropertyInfoList *props = NULL;
-    ObjectProperty *prop;
-    ObjectPropertyIterator iter;
+    Object *obj = object_resolve_path(path, &ambiguous);
 
-    obj = object_resolve_path(path, &ambiguous);
     if (obj == NULL) {
         if (ambiguous) {
             error_setg(errp, "Path '%s' is ambiguous", path);
@@ -44,6 +40,19 @@ ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
             error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
                       "Device '%s' not found", path);
         }
+    }
+    return obj;
+}
+
+ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
+{
+    Object *obj;
+    ObjectPropertyInfoList *props = NULL;
+    ObjectProperty *prop;
+    ObjectPropertyIterator iter;
+
+    obj = qom_resolve_path(path, errp);
+    if (obj == NULL) {
         return NULL;
     }
 
-- 
1.8.3.1

Re: [PATCH V1 1/6] qom: qom_resolve_path

[Philippe Mathieu-Daudé]

On 3/3/25 22:09, Steve Sistare wrote:
> Factor out a helper to resolve the user's path and print error messages.
> No functional change.
> 
> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
> ---
>   qom/qom-qmp-cmds.c | 21 +++++++++++++++------
>   1 file changed, 15 insertions(+), 6 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH V1 2/6] qom: qom-tree-get

[Steve Sistare]

Define the qom-tree-get QAPI command, which fetches an entire tree of
properties and values with a single QAPI call.  This is much faster
than using qom-list plus qom-get for every node and property of the
tree.  See qom.json for details.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 qapi/qom.json      | 59 ++++++++++++++++++++++++++++++++++++++++++++++
 qom/qom-qmp-cmds.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 128 insertions(+)

diff --git a/qapi/qom.json b/qapi/qom.json
index 28ce24c..646e3c6 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -46,6 +46,41 @@
             '*default-value': 'any' } }
 
 ##
+# @ObjectPropertyValue:
+#
+# @name: the name of the property
+#
+# @type: the type of the property, as described in @ObjectPropertyInfo
+#
+# @value: the value of the property
+#
+# @error: error message if value cannot be fetched
+#
+# Since 10.0
+##
+{ 'struct': 'ObjectPropertyValue',
+  'data': { 'name': 'str',
+            'type': 'str',
+            '*value': 'any',
+            '*error': 'str' } }
+
+##
+# @ObjectNode:
+#
+# @name: the name of the node
+#
+# @children: child nodes
+#
+# @properties: properties of the node
+#
+# Since 10.0
+##
+{ 'struct': 'ObjectNode',
+  'data': { 'name': 'str',
+            'children': [ 'ObjectNode' ],
+            'properties': [ 'ObjectPropertyValue' ] }}
+
+##
 # @qom-list:
 #
 # This command will list any properties of a object given a path in
@@ -126,6 +161,30 @@
   'allow-preconfig': true }
 
 ##
+# @qom-tree-get:
+#
+# This command returns a tree of objects and their properties,
+# rooted at the specified path.
+#
+# @path: The absolute or partial path within the object model, as
+#     described in @qom-get
+#
+# Errors:
+#     - If path is not valid or is ambiguous, returns an error.
+#     - If a property cannot be read, returns an error message in the
+#       corresponding @ObjectPropertyValue.
+#
+# Returns: A tree of @ObjectNode.  Each node contains its name, list
+#     of properties, and list of child nodes.
+#
+# Since 10.0
+##
+{ 'command': 'qom-tree-get',
+  'data': { 'path': 'str' },
+  'returns': 'ObjectNode',
+  'allow-preconfig': true }
+
+##
 # @qom-set:
 #
 # This command will set a property from a object model path.
diff --git a/qom/qom-qmp-cmds.c b/qom/qom-qmp-cmds.c
index 293755f..271f62d 100644
--- a/qom/qom-qmp-cmds.c
+++ b/qom/qom-qmp-cmds.c
@@ -69,6 +69,75 @@ ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
     return props;
 }
 
+static void qom_list_add_property_value(Object *obj, ObjectProperty *prop,
+                                        ObjectPropertyValueList **props)
+{
+    ObjectPropertyValue *item = g_new0(ObjectPropertyValue, 1);
+    Error *err = NULL;
+
+    QAPI_LIST_PREPEND(*props, item);
+
+    item->name = g_strdup(prop->name);
+    item->type = g_strdup(prop->type);
+    item->value = object_property_get_qobject(obj, prop->name, &err);
+
+    if (!item->value) {
+        item->error = g_strdup(error_get_pretty(err));
+        error_free(err);
+    }
+}
+
+static ObjectNode *qom_tree_get(const char *path, Error **errp)
+{
+    Object *obj;
+    ObjectProperty *prop;
+    ObjectNode *result, *child;
+    ObjectPropertyIterator iter;
+
+    obj = qom_resolve_path(path, errp);
+    if (obj == NULL) {
+        return NULL;
+    }
+
+    result = g_new0(ObjectNode, 1);
+
+    object_property_iter_init(&iter, obj);
+    while ((prop = object_property_iter_next(&iter))) {
+        if (strstart(prop->type, "child<", NULL)) {
+            g_autofree char *child_path = g_strdup_printf("%s/%s",
+                                                          path, prop->name);
+            child = qom_tree_get(child_path, errp);
+            if (!child) {
+                qapi_free_ObjectNode(result);
+                return NULL;
+            }
+            child->name = g_strdup(prop->name);
+            QAPI_LIST_PREPEND(result->children, child);
+        } else {
+            qom_list_add_property_value(obj, prop, &result->properties);
+        }
+    }
+
+    return result;
+}
+
+ObjectNode *qmp_qom_tree_get(const char *path, Error **errp)
+{
+    ObjectNode *result = qom_tree_get(path, errp);
+
+    if (result) {
+        /* Strip the path prefix if any */
+        const char *basename = strrchr(path, '/');
+
+        if (!basename || !basename[1]) {
+            result->name = g_strdup(path);
+        } else {
+            result->name = g_strdup(basename + 1);
+        }
+    }
+    return result;
+}
+
 void qmp_qom_set(const char *path, const char *property, QObject *value,
                  Error **errp)
 {
-- 
1.8.3.1

[PATCH V1 3/6] python: use qom-tree-get

[Steve Sistare]

Use qom-tree-get to speed up the qom-tree command.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 python/qemu/utils/qom.py        | 36 ++++++++++++++---------------
 python/qemu/utils/qom_common.py | 50 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+), 18 deletions(-)

diff --git a/python/qemu/utils/qom.py b/python/qemu/utils/qom.py
index 426a0f2..086a592 100644
--- a/python/qemu/utils/qom.py
+++ b/python/qemu/utils/qom.py
@@ -35,6 +35,7 @@
 from qemu.qmp import ExecuteError
 
 from .qom_common import QOMCommand
+from .qom_common import ObjectNode
 
 
 try:
@@ -224,28 +225,27 @@ def __init__(self, args: argparse.Namespace):
         super().__init__(args)
         self.path = args.path
 
-    def _list_node(self, path: str) -> None:
+    def _list_node(self, node: ObjectNode, path: str) -> None:
         print(path)
-        items = self.qom_list(path)
-        for item in items:
-            if item.child:
-                continue
-            try:
-                rsp = self.qmp.cmd('qom-get', path=path,
-                                   property=item.name)
-                print(f"  {item.name}: {rsp} ({item.type})")
-            except ExecuteError as err:
-                print(f"  {item.name}: <EXCEPTION: {err!s}> ({item.type})")
+
+        for item in node.properties:
+            value = item.value
+            if value == None:
+                value = f"<EXCEPTION: {item.error}>"
+            print(f"  {item.name}: {value} ({item.type})")
+
         print('')
-        for item in items:
-            if not item.child:
-                continue
-            if path == '/':
-                path = ''
-            self._list_node(f"{path}/{item.name}")
+        if path == '/':
+            path = ''
+
+        for child in node.children:
+            self._list_tree(child, f"{path}/{child.name}")
+
 
     def run(self) -> int:
-        self._list_node(self.path)
+        root = self.qom_tree_get(self.path)
+        self._list_node(root, self.path)
+
         return 0
 
 
diff --git a/python/qemu/utils/qom_common.py b/python/qemu/utils/qom_common.py
index dd2c8b1..0303863 100644
--- a/python/qemu/utils/qom_common.py
+++ b/python/qemu/utils/qom_common.py
@@ -65,6 +65,49 @@ def link(self) -> bool:
         return self.type.startswith('link<')
 
 
+class ObjectPropertyValue:
+    """
+    Represents a property return from e.g. qom-tree-get
+    """
+    def __init__(self, name: str, type_: str, value: object, error: str):
+        self.name = name
+        self.type = type_
+        self.value = value
+        self.error = error
+
+    @classmethod
+    def make(cls, value: Dict[str, Any]) -> 'ObjectPropertyValue':
+        """
+        Build an ObjectPropertyValue from a Dict with an unknown shape.
+        """
+        assert value.keys() >= {'name', 'type'}
+        assert value.keys() <= {'name', 'type', 'value', 'error'}
+        return cls(value['name'], value['type'],
+                   value.get('value'), value.get('error'))
+
+
+class ObjectNode:
+    """
+    Represents the return type from e.g. qom-tree-get
+    """
+    def __init__(self, name: str, children: List['ObjectNode'],
+                 properties: List[ObjectPropertyValue]):
+        self.name = name
+        self.children = children
+        self.properties = properties
+
+    @classmethod
+    def make(cls, value: Dict[str, Any]) -> 'ObjectNode':
+        """
+        Build an ObjectNode from a Dict with an unknown shape.
+        """
+        assert value.keys() == {'name', 'children', 'properties'}
+
+        props = [ObjectPropertyValue.make(x) for x in value['properties']]
+        children = [ObjectNode.make(x) for x in value['children']]
+        return cls(value['name'], children, props)
+
+
 CommandT = TypeVar('CommandT', bound='QOMCommand')
 
 
@@ -145,6 +188,13 @@ def qom_list(self, path: str) -> List[ObjectPropertyInfo]:
         assert isinstance(rsp, list)
         return [ObjectPropertyInfo.make(x) for x in rsp]
 
+    def qom_tree_get(self, path: str) -> ObjectNode:
+        """
+        :return: a strongly typed root node from the 'qom-tree-get' command.
+        """
+        rsp = self.qmp.cmd('qom-tree-get', path=path)
+        return ObjectNode.make(rsp)
+
     @classmethod
     def command_runner(
             cls: Type[CommandT],
-- 
1.8.3.1

[PATCH V1 4/6] tests/qtest/qom-test: unit test for qom-tree-get

[Steve Sistare]

Add a unit test for qom-tree-get

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 tests/qtest/qom-test.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/tests/qtest/qom-test.c b/tests/qtest/qom-test.c
index 27d70bc..c567c4c 100644
--- a/tests/qtest/qom-test.c
+++ b/tests/qtest/qom-test.c
@@ -16,6 +16,55 @@
 
 static int verbosity_level;
 
+static void test_tree_node(QDict *node)
+{
+    QDict *prop, *child;
+    QList *props, *children;
+    QListEntry *entry;
+
+    g_assert(qdict_haskey(node, "name"));
+    g_assert(qdict_haskey(node, "properties"));
+
+    if (verbosity_level >= 3) {
+        g_test_message("%s", qdict_get_str(node, "name"));
+    }
+
+    props = qobject_to(QList, qdict_get(node, "properties"));
+    QLIST_FOREACH_ENTRY(props, entry) {
+        prop = qobject_to(QDict, qlist_entry_obj(entry));
+        g_assert(qdict_haskey(prop, "name"));
+        g_assert(qdict_haskey(prop, "type"));
+        g_assert(qdict_haskey(prop, "value") || qdict_haskey(prop, "error"));
+    }
+
+    if (!qdict_haskey(node, "children")) {
+        return;
+    }
+
+    children = qobject_to(QList, qdict_get(node, "children"));
+    QLIST_FOREACH_ENTRY(children, entry) {
+        child = qobject_to(QDict, qlist_entry_obj(entry));
+        test_tree_node(child);
+    }
+}
+
+static void test_tree(QTestState *qts, const char *path)
+{
+    g_autoptr(QDict) response = NULL;
+    QDict *node;
+
+    if (verbosity_level >= 2) {
+        g_test_message("Obtaining tree at %s", path);
+    }
+    response = qtest_qmp(qts, "{ 'execute': 'qom-tree-get',"
+                              "  'arguments': { 'path': %s } }", path);
+    g_assert(response);
+
+    g_assert(qdict_haskey(response, "return"));
+    node = qobject_to(QDict, qdict_get(response, "return"));
+    test_tree_node(node);
+}
+
 static void test_properties(QTestState *qts, const char *path, bool recurse)
 {
     char *child_path;
@@ -100,6 +149,7 @@ static void test_machine(gconstpointer data)
     }
 
     test_properties(qts, "/machine", true);
+    test_tree(qts, "/machine");
 
     response = qtest_qmp(qts, "{ 'execute': 'quit' }");
     g_assert(qdict_haskey(response, "return"));
-- 
1.8.3.1

[PATCH V1 5/6] qom: qom-list-getv

[Steve Sistare]

Define the qom-list-getv command, which fetches all the properties and
values for a list of paths.  This is faster than qom-tree-get when
fetching a subset of the QOM tree.  See qom.json for details.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 qapi/qom.json      | 34 ++++++++++++++++++++++++++++++++++
 qom/qom-qmp-cmds.c | 40 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)

diff --git a/qapi/qom.json b/qapi/qom.json
index 646e3c6..c0c1839 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -65,6 +65,16 @@
             '*error': 'str' } }
 
 ##
+# @ObjectPropertiesValues:
+#
+# @properties: a list of properties.
+#
+# Since 10.0
+##
+{ 'struct': 'ObjectPropertiesValues',
+  'data': { 'properties': [ 'ObjectPropertyValue' ] }}
+
+##
 # @ObjectNode:
 #
 # @name: the name of the node
@@ -161,6 +171,30 @@
   'allow-preconfig': true }
 
 ##
+# @qom-list-getv:
+#
+# This command returns a list of properties and their values for
+# each object path in the input list.
+#
+# @paths: The absolute or partial path for each object, as described
+#     in @qom-get
+#
+# Errors:
+#     - If any path is not valid or is ambiguous, returns an error.
+#     - If a property cannot be read, returns an error message in the
+#       corresponding @ObjectPropertyValue.
+#
+# Returns: A list of @ObjectPropertiesValues.  Each element contains
+#     the properties of the corresponding element in @paths.
+#
+# Since 10.0
+##
+{ 'command': 'qom-list-getv',
+  'data': { 'paths': [ 'str' ] },
+  'returns': [ 'ObjectPropertiesValues' ],
+  'allow-preconfig': true }
+
+##
 # @qom-tree-get:
 #
 # This command returns a tree of objects and their properties,
diff --git a/qom/qom-qmp-cmds.c b/qom/qom-qmp-cmds.c
index 271f62d..b38f0d2 100644
--- a/qom/qom-qmp-cmds.c
+++ b/qom/qom-qmp-cmds.c
@@ -87,6 +87,46 @@ static void qom_list_add_property_value(Object *obj, ObjectProperty *prop,
     }
 }
 
+static ObjectPropertyValueList *qom_get_property_value_list(const char *path,
+                                                            Error **errp)
+{
+    Object *obj;
+    ObjectProperty *prop;
+    ObjectPropertyIterator iter;
+    ObjectPropertyValueList *props = NULL;
+
+    obj = qom_resolve_path(path, errp);
+    if (obj == NULL) {
+        return NULL;
+    }
+
+    object_property_iter_init(&iter, obj);
+    while ((prop = object_property_iter_next(&iter))) {
+        qom_list_add_property_value(obj, prop, &props);
+    }
+
+    return props;
+}
+
+ObjectPropertiesValuesList *qmp_qom_list_getv(strList *paths, Error **errp)
+{
+    ObjectPropertiesValuesList *head = NULL, **tail = &head;
+
+    for ( ; paths ; paths = paths->next) {
+        ObjectPropertiesValues *item = g_new0(ObjectPropertiesValues, 1);
+
+        QAPI_LIST_APPEND(tail, item);
+
+        item->properties = qom_get_property_value_list(paths->value, errp);
+        if (!item->properties) {
+            qapi_free_ObjectPropertiesValuesList(head);
+            return NULL;
+        }
+    }
+
+    return head;
+}
+
 static ObjectNode *qom_tree_get(const char *path, Error **errp)
 {
     Object *obj;
-- 
1.8.3.1

[PATCH V1 6/6] tests/qtest/qom-test: unit test for qom-list-getv

[Steve Sistare]

Add a unit test for qom-list-getv.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 tests/qtest/qom-test.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/tests/qtest/qom-test.c b/tests/qtest/qom-test.c
index c567c4c..8785ea3 100644
--- a/tests/qtest/qom-test.c
+++ b/tests/qtest/qom-test.c
@@ -11,6 +11,7 @@
 
 #include "qobject/qdict.h"
 #include "qobject/qlist.h"
+#include "qobject/qstring.h"
 #include "qemu/cutils.h"
 #include "libqtest.h"
 
@@ -65,6 +66,68 @@ static void test_tree(QTestState *qts, const char *path)
     test_tree_node(node);
 }
 
+static void test_getv(QTestState *qts, QList *paths)
+{
+    QListEntry *entry, *prop_entry, *path_entry;
+    g_autoptr(QDict) response = NULL;
+    QDict *args = qdict_new();
+    QDict *prop;
+    QList *return_list;
+
+    if (verbosity_level >= 2) {
+        g_test_message("Obtaining properties for paths:");
+        QLIST_FOREACH_ENTRY(paths, path_entry) {
+            QString *qstr = qobject_to(QString, qlist_entry_obj(path_entry));
+            g_test_message("  %s", qstring_get_str(qstr));
+        }
+    }
+
+    qdict_put_obj(args, "paths", QOBJECT(qlist_copy(paths)));
+    response = qtest_qmp(qts, "{ 'execute': 'qom-list-getv',"
+                              "  'arguments': %p }", args);
+    g_assert(response);
+    g_assert(qdict_haskey(response, "return"));
+    return_list = qobject_to(QList, qdict_get(response, "return"));
+
+    path_entry = QTAILQ_FIRST(&paths->head);
+    QLIST_FOREACH_ENTRY(return_list, entry) {
+        QDict *obj = qobject_to(QDict, qlist_entry_obj(entry));
+        g_assert(qdict_haskey(obj, "properties"));
+        QList *properties = qobject_to(QList, qdict_get(obj, "properties"));
+        bool has_child = false;
+
+        QLIST_FOREACH_ENTRY(properties, prop_entry) {
+            prop = qobject_to(QDict, qlist_entry_obj(prop_entry));
+            g_assert(qdict_haskey(prop, "name"));
+            g_assert(qdict_haskey(prop, "type"));
+            g_assert(qdict_haskey(prop, "value") ||
+                     qdict_haskey(prop, "error"));
+            has_child |= strstart(qdict_get_str(prop, "type"), "child<", NULL);
+        }
+
+        if (has_child) {
+            /* build a list of child paths */
+            QString *qstr = qobject_to(QString, qlist_entry_obj(path_entry));
+            const char *path = qstring_get_str(qstr);
+            g_autoptr(QList) child_paths = qlist_new();
+
+            QLIST_FOREACH_ENTRY(properties, prop_entry) {
+                prop = qobject_to(QDict, qlist_entry_obj(prop_entry));
+                if (strstart(qdict_get_str(prop, "type"), "child<", NULL)) {
+                    g_autofree char *child_path = g_strdup_printf(
+                        "%s/%s", path, qdict_get_str(prop, "name"));
+                    qlist_append_str(child_paths, child_path);
+                }
+            }
+
+            /* fetch props for all children with one qom-list-getv call */
+            test_getv(qts, child_paths);
+        }
+
+        path_entry = QTAILQ_NEXT(path_entry, next);
+    }
+}
+
 static void test_properties(QTestState *qts, const char *path, bool recurse)
 {
     char *child_path;
@@ -134,6 +197,7 @@ static void test_machine(gconstpointer data)
     const char *machine = data;
     QDict *response;
     QTestState *qts;
+    g_autoptr(QList) paths = qlist_new();
 
     qts = qtest_initf("-machine %s", machine);
 
@@ -150,6 +214,8 @@ static void test_machine(gconstpointer data)
 
     test_properties(qts, "/machine", true);
     test_tree(qts, "/machine");
+    qlist_append_str(paths, "/machine");
+    test_getv(qts, paths);
 
     response = qtest_qmp(qts, "{ 'execute': 'quit' }");
     g_assert(qdict_haskey(response, "return"));
-- 
1.8.3.1

Re: [PATCH V1 0/6] fast qom tree get

[Markus Armbruster]

Hi Steve, I apologize for the slow response.

Steve Sistare <steven.sistare@oracle.com> writes:

> Using qom-list and qom-get to get all the nodes and property values in a
> QOM tree can take multiple seconds because it requires 1000's of individual
> QOM requests.  Some managers fetch the entire tree or a large subset
> of it when starting a new VM, and this cost is a substantial fraction of
> start up time.

"Some managers"... could you name one?

> To reduce this cost, consider QAPI calls that fetch more information in
> each call:
>   * qom-list-get: given a path, return a list of properties and values.
>   * qom-list-getv: given a list of paths, return a list of properties and
>     values for each path.
>   * qom-tree-get: given a path, return all descendant nodes rooted at that
>     path, with properties and values for each.

Libvirt developers, would you be interested in any of these?

> In all cases, a returned property is represented by ObjectPropertyValue,
> with fields name, type, value, and error.  If an error occurs when reading
> a value, the value field is omitted, and the error message is returned in the
> the error field.  Thus an error for one property will not cause a bulk fetch
> operation to fail.

Returning errors this way is highly unusual.  Observation; I'm not
rejecting this out of hand.  Can you elaborate a bit on why it's useful?

> To evaluate each method, I modified scripts/qmp/qom-tree to use the method,
> verified all methods produce the same output, and timed each using:
>
>   qemu-system-x86_64 -display none \
>     -chardev socket,id=monitor0,path=/tmp/vm1.sock,server=on,wait=off \
>     -mon monitor0,mode=control &
>
>   time qom-tree -s /tmp/vm1.sock > /dev/null

Cool!

> I only measured once per method, but the variation is low after a warm up run.
> The 'real - user - sys' column is a proxy for QEMU CPU time.
>
> method               real(s)   user(s)   sys(s)  (real - user - sys)(s)
> qom-list / qom-get   2.048     0.932     0.057   1.059
> qom-list-get         0.402     0.230     0.029   0.143
> qom-list-getv        0.200     0.132     0.015   0.053
> qom-tree-get         0.143     0.123     0.012   0.008
>
> qom-tree-get is the clear winner, reducing elapsed time by a factor of 14X,
> and reducing QEMU CPU time by 132X.
>
> qom-list-getv is slower when fetching the entire tree, but can beat
> qom-tree-get when only a subset of the tree needs to be fetched (not shown).
>
> qom-list-get is shown for comparison only, and is not included in this series.

If we have qom-list-getv, then qom-list-get is not worth having.

Re: [PATCH V1 0/6] fast qom tree get

[Peter Krempa]

On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote:
> Hi Steve, I apologize for the slow response.
> 
> Steve Sistare <steven.sistare@oracle.com> writes:
> 
> > Using qom-list and qom-get to get all the nodes and property values in a
> > QOM tree can take multiple seconds because it requires 1000's of individual
> > QOM requests.  Some managers fetch the entire tree or a large subset
> > of it when starting a new VM, and this cost is a substantial fraction of
> > start up time.
> 
> "Some managers"... could you name one?

libvirt is at ~500 qom-get calls during an average startup ...

> > To reduce this cost, consider QAPI calls that fetch more information in
> > each call:
> >   * qom-list-get: given a path, return a list of properties and values.
> >   * qom-list-getv: given a list of paths, return a list of properties and
> >     values for each path.
> >   * qom-tree-get: given a path, return all descendant nodes rooted at that
> >     path, with properties and values for each.
> 
> Libvirt developers, would you be interested in any of these?

YES!!!

The getter with value could SO MUCH optimize the startup sequence of a
VM where libvirt needs to probe CPU flags:

(note the 'id' field in libvirt's monitor is sequential)

buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"}
buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"}

[...]

buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"}
buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"}
buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"}
buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}

First and last line's timestamps:

2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}

2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}

Libvirt spent ~170 ms probing cpu flags.

Re: [PATCH V1 0/6] fast qom tree get

[Daniel P. Berrangé]

On Wed, Apr 09, 2025 at 09:58:13AM +0200, Peter Krempa via Devel wrote:
> On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote:
> > Hi Steve, I apologize for the slow response.
> > 
> > Steve Sistare <steven.sistare@oracle.com> writes:
> > 
> > > Using qom-list and qom-get to get all the nodes and property values in a
> > > QOM tree can take multiple seconds because it requires 1000's of individual
> > > QOM requests.  Some managers fetch the entire tree or a large subset
> > > of it when starting a new VM, and this cost is a substantial fraction of
> > > start up time.
> > 
> > "Some managers"... could you name one?
> 
> libvirt is at ~500 qom-get calls during an average startup ...
> 
> > > To reduce this cost, consider QAPI calls that fetch more information in
> > > each call:
> > >   * qom-list-get: given a path, return a list of properties and values.
> > >   * qom-list-getv: given a list of paths, return a list of properties and
> > >     values for each path.
> > >   * qom-tree-get: given a path, return all descendant nodes rooted at that
> > >     path, with properties and values for each.
> > 
> > Libvirt developers, would you be interested in any of these?
> 
> YES!!!

Not neccessarily, see below... !!!! 

> 
> The getter with value could SO MUCH optimize the startup sequence of a
> VM where libvirt needs to probe CPU flags:
> 
> (note the 'id' field in libvirt's monitor is sequential)
> 
> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"}
> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"}
> 
> [...]
> 
> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"}
> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"}
> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"}
> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
> 
> First and last line's timestamps:
> 
> 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
> 
> 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
> 
> Libvirt spent ~170 ms probing cpu flags.

One thing I would point out is that qom-get can be considered an
"escape hatch" to get information when no better QMP command exists.
In this case, libvirt has made the assumption that every CPU feature
is a QOM property.

Adding qom-list-get doesn't appreciably change that, just makes the
usage more efficient.

Considering the bigger picture QMP design, when libvirt is trying to
understand QEMU's CPU feature flag expansion, I would ask why we don't
have something like a "query-cpu" command to tell us the current CPU
expansion, avoiding the need for poking at QOM properties directly.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Management applications and CPU feature flags (was: [PATCH V1 0/6] fast qom tree get)

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Apr 09, 2025 at 09:58:13AM +0200, Peter Krempa via Devel wrote:
>> On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote:
>> > Hi Steve, I apologize for the slow response.
>> > 
>> > Steve Sistare <steven.sistare@oracle.com> writes:
>> > 
>> > > Using qom-list and qom-get to get all the nodes and property values in a
>> > > QOM tree can take multiple seconds because it requires 1000's of individual
>> > > QOM requests.  Some managers fetch the entire tree or a large subset
>> > > of it when starting a new VM, and this cost is a substantial fraction of
>> > > start up time.
>> > 
>> > "Some managers"... could you name one?
>> 
>> libvirt is at ~500 qom-get calls during an average startup ...
>> 
>> > > To reduce this cost, consider QAPI calls that fetch more information in
>> > > each call:
>> > >   * qom-list-get: given a path, return a list of properties and values.
>> > >   * qom-list-getv: given a list of paths, return a list of properties and
>> > >     values for each path.
>> > >   * qom-tree-get: given a path, return all descendant nodes rooted at that
>> > >     path, with properties and values for each.
>> > 
>> > Libvirt developers, would you be interested in any of these?
>> 
>> YES!!!
>
> Not neccessarily, see below... !!!! 
>
>> 
>> The getter with value could SO MUCH optimize the startup sequence of a
>> VM where libvirt needs to probe CPU flags:
>> 
>> (note the 'id' field in libvirt's monitor is sequential)
>> 
>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"}
>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"}
>> 
>> [...]
>> 
>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"}
>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"}
>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"}
>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
>> 
>> First and last line's timestamps:
>> 
>> 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
>> 
>> 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
>> 
>> Libvirt spent ~170 ms probing cpu flags.
>
> One thing I would point out is that qom-get can be considered an
> "escape hatch" to get information when no better QMP command exists.
> In this case, libvirt has made the assumption that every CPU feature
> is a QOM property.
>
> Adding qom-list-get doesn't appreciably change that, just makes the
> usage more efficient.
>
> Considering the bigger picture QMP design, when libvirt is trying to
> understand QEMU's CPU feature flag expansion, I would ask why we don't
> have something like a "query-cpu" command to tell us the current CPU
> expansion, avoiding the need for poking at QOM properties directly.

How do the existing query-cpu-FOO fall short of what management
applications such as libvirt needs?

Re: Management applications and CPU feature flags (was: [PATCH V1 0/6] fast qom tree get)

[Daniel P. Berrangé]

On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Wed, Apr 09, 2025 at 09:58:13AM +0200, Peter Krempa via Devel wrote:
> >> On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote:
> >> > Hi Steve, I apologize for the slow response.
> >> > 
> >> > Steve Sistare <steven.sistare@oracle.com> writes:
> >> > 
> >> > > Using qom-list and qom-get to get all the nodes and property values in a
> >> > > QOM tree can take multiple seconds because it requires 1000's of individual
> >> > > QOM requests.  Some managers fetch the entire tree or a large subset
> >> > > of it when starting a new VM, and this cost is a substantial fraction of
> >> > > start up time.
> >> > 
> >> > "Some managers"... could you name one?
> >> 
> >> libvirt is at ~500 qom-get calls during an average startup ...
> >> 
> >> > > To reduce this cost, consider QAPI calls that fetch more information in
> >> > > each call:
> >> > >   * qom-list-get: given a path, return a list of properties and values.
> >> > >   * qom-list-getv: given a list of paths, return a list of properties and
> >> > >     values for each path.
> >> > >   * qom-tree-get: given a path, return all descendant nodes rooted at that
> >> > >     path, with properties and values for each.
> >> > 
> >> > Libvirt developers, would you be interested in any of these?
> >> 
> >> YES!!!
> >
> > Not neccessarily, see below... !!!! 
> >
> >> 
> >> The getter with value could SO MUCH optimize the startup sequence of a
> >> VM where libvirt needs to probe CPU flags:
> >> 
> >> (note the 'id' field in libvirt's monitor is sequential)
> >> 
> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"}
> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"}
> >> 
> >> [...]
> >> 
> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"}
> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"}
> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"}
> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
> >> 
> >> First and last line's timestamps:
> >> 
> >> 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
> >> 
> >> 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
> >> 
> >> Libvirt spent ~170 ms probing cpu flags.
> >
> > One thing I would point out is that qom-get can be considered an
> > "escape hatch" to get information when no better QMP command exists.
> > In this case, libvirt has made the assumption that every CPU feature
> > is a QOM property.
> >
> > Adding qom-list-get doesn't appreciably change that, just makes the
> > usage more efficient.
> >
> > Considering the bigger picture QMP design, when libvirt is trying to
> > understand QEMU's CPU feature flag expansion, I would ask why we don't
> > have something like a "query-cpu" command to tell us the current CPU
> > expansion, avoiding the need for poking at QOM properties directly.
> 
> How do the existing query-cpu-FOO fall short of what management
> applications such as libvirt needs?

It has been along while since I looked at them, but IIRC they were
returning static info about CPU models, whereas libvirt wanted info
on the currently requested '-cpu ARGS'

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Management applications and CPU feature flags

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> > On Wed, Apr 09, 2025 at 09:58:13AM +0200, Peter Krempa via Devel wrote:
>> >> On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote:
>> >> > Hi Steve, I apologize for the slow response.
>> >> > 
>> >> > Steve Sistare <steven.sistare@oracle.com> writes:
>> >> > 
>> >> > > Using qom-list and qom-get to get all the nodes and property values in a
>> >> > > QOM tree can take multiple seconds because it requires 1000's of individual
>> >> > > QOM requests.  Some managers fetch the entire tree or a large subset
>> >> > > of it when starting a new VM, and this cost is a substantial fraction of
>> >> > > start up time.
>> >> > 
>> >> > "Some managers"... could you name one?
>> >> 
>> >> libvirt is at ~500 qom-get calls during an average startup ...
>> >> 
>> >> > > To reduce this cost, consider QAPI calls that fetch more information in
>> >> > > each call:
>> >> > >   * qom-list-get: given a path, return a list of properties and values.
>> >> > >   * qom-list-getv: given a list of paths, return a list of properties and
>> >> > >     values for each path.
>> >> > >   * qom-tree-get: given a path, return all descendant nodes rooted at that
>> >> > >     path, with properties and values for each.
>> >> > 
>> >> > Libvirt developers, would you be interested in any of these?
>> >> 
>> >> YES!!!
>> >
>> > Not neccessarily, see below... !!!! 
>> >
>> >> 
>> >> The getter with value could SO MUCH optimize the startup sequence of a
>> >> VM where libvirt needs to probe CPU flags:
>> >> 
>> >> (note the 'id' field in libvirt's monitor is sequential)
>> >> 
>> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
>> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"}
>> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"}
>> >> 
>> >> [...]
>> >> 
>> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"}
>> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"}
>> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"}
>> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
>> >> 
>> >> First and last line's timestamps:
>> >> 
>> >> 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
>> >> 
>> >> 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
>> >> 
>> >> Libvirt spent ~170 ms probing cpu flags.
>> >
>> > One thing I would point out is that qom-get can be considered an
>> > "escape hatch" to get information when no better QMP command exists.
>> > In this case, libvirt has made the assumption that every CPU feature
>> > is a QOM property.
>> >
>> > Adding qom-list-get doesn't appreciably change that, just makes the
>> > usage more efficient.
>> >
>> > Considering the bigger picture QMP design, when libvirt is trying to
>> > understand QEMU's CPU feature flag expansion, I would ask why we don't
>> > have something like a "query-cpu" command to tell us the current CPU
>> > expansion, avoiding the need for poking at QOM properties directly.
>> 
>> How do the existing query-cpu-FOO fall short of what management
>> applications such as libvirt needs?
>
> It has been along while since I looked at them, but IIRC they were
> returning static info about CPU models, whereas libvirt wanted info
> on the currently requested '-cpu ARGS'

Libvirt developers, please work with us on design of new commands or
improvements to existing ones to better meet libvirt's needs in this
area.

Re: Management applications and CPU feature flags

[David Hildenbrand]

On 11.04.25 13:43, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
>> On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote:
>>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>>
>>>> On Wed, Apr 09, 2025 at 09:58:13AM +0200, Peter Krempa via Devel wrote:
>>>>> On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote:
>>>>>> Hi Steve, I apologize for the slow response.
>>>>>>
>>>>>> Steve Sistare <steven.sistare@oracle.com> writes:
>>>>>>
>>>>>>> Using qom-list and qom-get to get all the nodes and property values in a
>>>>>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>>>>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>>>>>> of it when starting a new VM, and this cost is a substantial fraction of
>>>>>>> start up time.
>>>>>>
>>>>>> "Some managers"... could you name one?
>>>>>
>>>>> libvirt is at ~500 qom-get calls during an average startup ...
>>>>>
>>>>>>> To reduce this cost, consider QAPI calls that fetch more information in
>>>>>>> each call:
>>>>>>>    * qom-list-get: given a path, return a list of properties and values.
>>>>>>>    * qom-list-getv: given a list of paths, return a list of properties and
>>>>>>>      values for each path.
>>>>>>>    * qom-tree-get: given a path, return all descendant nodes rooted at that
>>>>>>>      path, with properties and values for each.
>>>>>>
>>>>>> Libvirt developers, would you be interested in any of these?
>>>>>
>>>>> YES!!!
>>>>
>>>> Not neccessarily, see below... !!!!
>>>>
>>>>>
>>>>> The getter with value could SO MUCH optimize the startup sequence of a
>>>>> VM where libvirt needs to probe CPU flags:
>>>>>
>>>>> (note the 'id' field in libvirt's monitor is sequential)
>>>>>
>>>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
>>>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"}
>>>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"}
>>>>>
>>>>> [...]
>>>>>
>>>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"}
>>>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"}
>>>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"}
>>>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
>>>>>
>>>>> First and last line's timestamps:
>>>>>
>>>>> 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"}
>>>>>
>>>>> 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"}
>>>>>
>>>>> Libvirt spent ~170 ms probing cpu flags.
>>>>
>>>> One thing I would point out is that qom-get can be considered an
>>>> "escape hatch" to get information when no better QMP command exists.
>>>> In this case, libvirt has made the assumption that every CPU feature
>>>> is a QOM property.
>>>>
>>>> Adding qom-list-get doesn't appreciably change that, just makes the
>>>> usage more efficient.
>>>>
>>>> Considering the bigger picture QMP design, when libvirt is trying to
>>>> understand QEMU's CPU feature flag expansion, I would ask why we don't
>>>> have something like a "query-cpu" command to tell us the current CPU
>>>> expansion, avoiding the need for poking at QOM properties directly.
>>>
>>> How do the existing query-cpu-FOO fall short of what management
>>> applications such as libvirt needs?
>>
>> It has been along while since I looked at them, but IIRC they were
>> returning static info about CPU models, whereas libvirt wanted info
>> on the currently requested '-cpu ARGS'

Not sure what the exact requirements and other archs, but at least on 
s390x I think that's exactly what we do.

If you expand a non-static model (e.g., z14) you'd get the expansion as 
if you would specify "-cpu z14" on the cmdline for a specific QEMU machine.

Looking at CPU properties is really a nasty hack.

> 
> Libvirt developers, please work with us on design of new commands or
> improvements to existing ones to better meet libvirt's needs in this
> area.

Yes, knowing about requirements and why the existing APIs don't work 
would be great.

-- 
Cheers,

David / dhildenb

Re: Management applications and CPU feature flags

[Jiri Denemark]

On Fri, Apr 11, 2025 at 13:43:39 +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> > On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote:
> >> Daniel P. Berrangé <berrange@redhat.com> writes:
> >> > Considering the bigger picture QMP design, when libvirt is trying to
> >> > understand QEMU's CPU feature flag expansion, I would ask why we don't
> >> > have something like a "query-cpu" command to tell us the current CPU
> >> > expansion, avoiding the need for poking at QOM properties directly.
> >> 
> >> How do the existing query-cpu-FOO fall short of what management
> >> applications such as libvirt needs?
> >
> > It has been along while since I looked at them, but IIRC they were
> > returning static info about CPU models, whereas libvirt wanted info
> > on the currently requested '-cpu ARGS'
> 
> Libvirt developers, please work with us on design of new commands or
> improvements to existing ones to better meet libvirt's needs in this
> area.

The existing commands (query-cpu-definitions, query-cpu-model-expansion)
are useful for probing before starting a domain. But what we use qom-get
for is to get a view of the currently instantiated virtual CPU created
by QEMU according to -cpu when we're starting a domain. In other words,
we start QEMU with -S and before starting vCPUs we need to know exactly
what features were enabled and if any feature we requested was disabled
by QEMU. Currently we query QOM for CPU properties as that's what we
were advised to use ages ago.

The reason behind querying such info is ensuring stable guest ABI during
migration. Asking QEMU for a specific CPU model and features does not
mean we'll get exactly what we asked for (this is not a bug) so we need
to record the differences so that we can start QEMU for incoming
migration with a CPU matching exactly the one provided on the source.

As Peter said, the current way is terribly inefficient as it requires
several hundreds of QMP commands so the goal is to have a single QMP
command that would tell us all we need to know about the virtual CPU.
That is all enabled features and all features that could not be enabled
even though we asked for them.

Jirka

Re: Management applications and CPU feature flags

[Cornelia Huck]

On Fri, Apr 11 2025, Jiri Denemark <jdenemar@redhat.com> wrote:

> On Fri, Apr 11, 2025 at 13:43:39 +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> > On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote:
>> >> Daniel P. Berrangé <berrange@redhat.com> writes:
>> >> > Considering the bigger picture QMP design, when libvirt is trying to
>> >> > understand QEMU's CPU feature flag expansion, I would ask why we don't
>> >> > have something like a "query-cpu" command to tell us the current CPU
>> >> > expansion, avoiding the need for poking at QOM properties directly.
>> >> 
>> >> How do the existing query-cpu-FOO fall short of what management
>> >> applications such as libvirt needs?
>> >
>> > It has been along while since I looked at them, but IIRC they were
>> > returning static info about CPU models, whereas libvirt wanted info
>> > on the currently requested '-cpu ARGS'
>> 
>> Libvirt developers, please work with us on design of new commands or
>> improvements to existing ones to better meet libvirt's needs in this
>> area.
>
> The existing commands (query-cpu-definitions, query-cpu-model-expansion)
> are useful for probing before starting a domain. But what we use qom-get
> for is to get a view of the currently instantiated virtual CPU created
> by QEMU according to -cpu when we're starting a domain. In other words,
> we start QEMU with -S and before starting vCPUs we need to know exactly
> what features were enabled and if any feature we requested was disabled
> by QEMU. Currently we query QOM for CPU properties as that's what we
> were advised to use ages ago.
>
> The reason behind querying such info is ensuring stable guest ABI during
> migration. Asking QEMU for a specific CPU model and features does not
> mean we'll get exactly what we asked for (this is not a bug) so we need
> to record the differences so that we can start QEMU for incoming
> migration with a CPU matching exactly the one provided on the source.
>
> As Peter said, the current way is terribly inefficient as it requires
> several hundreds of QMP commands so the goal is to have a single QMP
> command that would tell us all we need to know about the virtual CPU.
> That is all enabled features and all features that could not be enabled
> even though we asked for them.

Wandering in here from the still-very-much-in-progress Arm perspective
(current but not yet posted QEMU code at
https://gitlab.com/cohuck/qemu/-/tree/arm-cpu-model-rfcv3?ref_type=heads):

We're currently operating at the "writable ID register fields" level
with the idea of providing features (FEAT_xxx) as an extra layer on top
(as they model a subset of what we actually need) and have yet to come
up with a good way to do named models for KVM. The
query-cpu-model-expansion command will yield a list of all writable ID
register fields and their values (as for now, for the 'host' model.) IIUC
you want to query (a) what is actually available for configuration
(before starting a domain) and (b) what you actually got (when starting
a domain). Would a dump of the current state of the ID register fields
before starting the vcpus work for (b)? Or is that too different from
what other archs need/want? How much wriggle room do we have for special
handling (different commands, different output, ...?)

Re: Management applications and CPU feature flags

[Jiří Denemark]

On Fri, Apr 11, 2025 at 15:58:54 +0200, Cornelia Huck wrote:
> On Fri, Apr 11 2025, Jiri Denemark <jdenemar@redhat.com> wrote:
> 
> > On Fri, Apr 11, 2025 at 13:43:39 +0200, Markus Armbruster wrote:
> >> Daniel P. Berrangé <berrange@redhat.com> writes:
> >> > On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote:
> >> >> Daniel P. Berrangé <berrange@redhat.com> writes:
> >> >> > Considering the bigger picture QMP design, when libvirt is trying to
> >> >> > understand QEMU's CPU feature flag expansion, I would ask why we don't
> >> >> > have something like a "query-cpu" command to tell us the current CPU
> >> >> > expansion, avoiding the need for poking at QOM properties directly.
> >> >> 
> >> >> How do the existing query-cpu-FOO fall short of what management
> >> >> applications such as libvirt needs?
> >> >
> >> > It has been along while since I looked at them, but IIRC they were
> >> > returning static info about CPU models, whereas libvirt wanted info
> >> > on the currently requested '-cpu ARGS'
> >> 
> >> Libvirt developers, please work with us on design of new commands or
> >> improvements to existing ones to better meet libvirt's needs in this
> >> area.
> >
> > The existing commands (query-cpu-definitions, query-cpu-model-expansion)
> > are useful for probing before starting a domain. But what we use qom-get
> > for is to get a view of the currently instantiated virtual CPU created
> > by QEMU according to -cpu when we're starting a domain. In other words,
> > we start QEMU with -S and before starting vCPUs we need to know exactly
> > what features were enabled and if any feature we requested was disabled
> > by QEMU. Currently we query QOM for CPU properties as that's what we
> > were advised to use ages ago.
> >
> > The reason behind querying such info is ensuring stable guest ABI during
> > migration. Asking QEMU for a specific CPU model and features does not
> > mean we'll get exactly what we asked for (this is not a bug) so we need
> > to record the differences so that we can start QEMU for incoming
> > migration with a CPU matching exactly the one provided on the source.
> >
> > As Peter said, the current way is terribly inefficient as it requires
> > several hundreds of QMP commands so the goal is to have a single QMP
> > command that would tell us all we need to know about the virtual CPU.
> > That is all enabled features and all features that could not be enabled
> > even though we asked for them.
> 
> Wandering in here from the still-very-much-in-progress Arm perspective
> (current but not yet posted QEMU code at
> https://gitlab.com/cohuck/qemu/-/tree/arm-cpu-model-rfcv3?ref_type=heads):
> 
> We're currently operating at the "writable ID register fields" level
> with the idea of providing features (FEAT_xxx) as an extra layer on top
> (as they model a subset of what we actually need) and have yet to come
> up with a good way to do named models for KVM. The
> query-cpu-model-expansion command will yield a list of all writable ID
> register fields and their values (as for now, for the 'host' model.) IIUC
> you want to query (a) what is actually available for configuration
> (before starting a domain) and (b) what you actually got (when starting
> a domain).

I guess it will be possible for QEMU to actually set something different
from what we tell it to do (for example dependency of a specific
settings on something else which was not set, etc)? If so, we indeed
need both (a) and (b).

> Would a dump of the current state of the ID register fields before
> starting the vcpus work for (b)?

I guess so. Originally for x86_64 we got a dump of CPUID data, but that
changed when some features started to be described by MSRs.

> Or is that too different from what other archs need/want?

Each arch has some specifics in CPU configuration and the way we talk
with QEMU about it. So having the same QMP interface is not a
requirement. It depends how well the existing interface maps to details
that need to be expressed. That said a common interface is better if it
makes sense.

Jirka

Re: [PATCH V1 0/6] fast qom tree get

[Steven Sistare]

On 4/9/2025 3:39 AM, Markus Armbruster wrote:
> Hi Steve, I apologize for the slow response.
> 
> Steve Sistare <steven.sistare@oracle.com> writes:
> 
>> Using qom-list and qom-get to get all the nodes and property values in a
>> QOM tree can take multiple seconds because it requires 1000's of individual
>> QOM requests.  Some managers fetch the entire tree or a large subset
>> of it when starting a new VM, and this cost is a substantial fraction of
>> start up time.
> 
> "Some managers"... could you name one?

My personal experience is with Oracle's OCI, but likely others could benefit.

>> To reduce this cost, consider QAPI calls that fetch more information in
>> each call:
>>    * qom-list-get: given a path, return a list of properties and values.
>>    * qom-list-getv: given a list of paths, return a list of properties and
>>      values for each path.
>>    * qom-tree-get: given a path, return all descendant nodes rooted at that
>>      path, with properties and values for each.
> 
> Libvirt developers, would you be interested in any of these?
> 
>> In all cases, a returned property is represented by ObjectPropertyValue,
>> with fields name, type, value, and error.  If an error occurs when reading
>> a value, the value field is omitted, and the error message is returned in the
>> the error field.  Thus an error for one property will not cause a bulk fetch
>> operation to fail.
> 
> Returning errors this way is highly unusual.  Observation; I'm not
> rejecting this out of hand.  Can you elaborate a bit on why it's useful?

It is considered an error to read some properties if they are not valid for
the configuration.  And some properties are write-only and return an error
if they are read.  Examples:

   legacy-i8042: <EXCEPTION: Property 'vmmouse.legacy-i8042' is not readable> (str)
   legacy-memory: <EXCEPTION: Property 'qemu64-x86_64-cpu.legacy-memory' is not readable> (str)
   crash-information: <EXCEPTION: No crash occurred> (GuestPanicInformation)

With conventional error handling, if any of these poison pills falls in the
scope of a bulk get operation, the entire operation fails.

>> To evaluate each method, I modified scripts/qmp/qom-tree to use the method,
>> verified all methods produce the same output, and timed each using:
>>
>>    qemu-system-x86_64 -display none \
>>      -chardev socket,id=monitor0,path=/tmp/vm1.sock,server=on,wait=off \
>>      -mon monitor0,mode=control &
>>
>>    time qom-tree -s /tmp/vm1.sock > /dev/null
> 
> Cool!
> 
>> I only measured once per method, but the variation is low after a warm up run.
>> The 'real - user - sys' column is a proxy for QEMU CPU time.
>>
>> method               real(s)   user(s)   sys(s)  (real - user - sys)(s)
>> qom-list / qom-get   2.048     0.932     0.057   1.059
>> qom-list-get         0.402     0.230     0.029   0.143
>> qom-list-getv        0.200     0.132     0.015   0.053
>> qom-tree-get         0.143     0.123     0.012   0.008
>>
>> qom-tree-get is the clear winner, reducing elapsed time by a factor of 14X,
>> and reducing QEMU CPU time by 132X.
>>
>> qom-list-getv is slower when fetching the entire tree, but can beat
>> qom-tree-get when only a subset of the tree needs to be fetched (not shown).
>>
>> qom-list-get is shown for comparison only, and is not included in this series.
> 
> If we have qom-list-getv, then qom-list-get is not worth having.

Exactly.

- Steve

Re: [PATCH V1 0/6] fast qom tree get

[Markus Armbruster]

Steven Sistare <steven.sistare@oracle.com> writes:

> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>> Hi Steve, I apologize for the slow response.
>> 
>> Steve Sistare <steven.sistare@oracle.com> writes:
>> 
>>> Using qom-list and qom-get to get all the nodes and property values in a
>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>> of it when starting a new VM, and this cost is a substantial fraction of
>>> start up time.
>> 
>> "Some managers"... could you name one?
>
> My personal experience is with Oracle's OCI, but likely others could benefit.

Peter Krempa tells us libvirt would benefit.

>>> To reduce this cost, consider QAPI calls that fetch more information in
>>> each call:
>>>    * qom-list-get: given a path, return a list of properties and values.
>>>    * qom-list-getv: given a list of paths, return a list of properties and
>>>      values for each path.
>>>    * qom-tree-get: given a path, return all descendant nodes rooted at that
>>>      path, with properties and values for each.
>> 
>> Libvirt developers, would you be interested in any of these?
>> 
>>> In all cases, a returned property is represented by ObjectPropertyValue,
>>> with fields name, type, value, and error.  If an error occurs when reading
>>> a value, the value field is omitted, and the error message is returned in the
>>> the error field.  Thus an error for one property will not cause a bulk fetch
>>> operation to fail.
>> 
>> Returning errors this way is highly unusual.  Observation; I'm not
>> rejecting this out of hand.  Can you elaborate a bit on why it's useful?
>
> It is considered an error to read some properties if they are not valid for
> the configuration.  And some properties are write-only and return an error
> if they are read.  Examples:
>
>    legacy-i8042: <EXCEPTION: Property 'vmmouse.legacy-i8042' is not readable> (str)
>    legacy-memory: <EXCEPTION: Property 'qemu64-x86_64-cpu.legacy-memory' is not readable> (str)
>    crash-information: <EXCEPTION: No crash occurred> (GuestPanicInformation)
>
> With conventional error handling, if any of these poison pills falls in the
> scope of a bulk get operation, the entire operation fails.

I suspect many of these poison pills are design mistakes.

If a property is not valid for the configuration, why does it exist?
QOM is by design dynamic.  I wish it wasn't, but as long as it is
dynamic, I can't see why we should create properties we know to be
unusable.

Why is reading crash-information an error when no crash occured?  This
is the *normal* case.  Errors are for the abnormal.

Anyway, asking you to fix design mistakes all over the place wouldn't be
fair.  So I'm asking you something else instead: do you actually need
the error information?

[...]

Re: [PATCH V1 0/6] fast qom tree get

[Steven Sistare]

On 4/9/2025 9:34 AM, Markus Armbruster wrote:
> Steven Sistare <steven.sistare@oracle.com> writes:
>> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>>> Hi Steve, I apologize for the slow response.
>>>
>>> Steve Sistare <steven.sistare@oracle.com> writes:
>>>
>>>> Using qom-list and qom-get to get all the nodes and property values in a
>>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>>> of it when starting a new VM, and this cost is a substantial fraction of
>>>> start up time.
>>>
>>> "Some managers"... could you name one?
>>
>> My personal experience is with Oracle's OCI, but likely others could benefit.
> 
> Peter Krempa tells us libvirt would benefit.
> 
>>>> To reduce this cost, consider QAPI calls that fetch more information in
>>>> each call:
>>>>     * qom-list-get: given a path, return a list of properties and values.
>>>>     * qom-list-getv: given a list of paths, return a list of properties and
>>>>       values for each path.
>>>>     * qom-tree-get: given a path, return all descendant nodes rooted at that
>>>>       path, with properties and values for each.
>>>
>>> Libvirt developers, would you be interested in any of these?
>>>
>>>> In all cases, a returned property is represented by ObjectPropertyValue,
>>>> with fields name, type, value, and error.  If an error occurs when reading
>>>> a value, the value field is omitted, and the error message is returned in the
>>>> the error field.  Thus an error for one property will not cause a bulk fetch
>>>> operation to fail.
>>>
>>> Returning errors this way is highly unusual.  Observation; I'm not
>>> rejecting this out of hand.  Can you elaborate a bit on why it's useful?
>>
>> It is considered an error to read some properties if they are not valid for
>> the configuration.  And some properties are write-only and return an error
>> if they are read.  Examples:
>>
>>     legacy-i8042: <EXCEPTION: Property 'vmmouse.legacy-i8042' is not readable> (str)
>>     legacy-memory: <EXCEPTION: Property 'qemu64-x86_64-cpu.legacy-memory' is not readable> (str)
>>     crash-information: <EXCEPTION: No crash occurred> (GuestPanicInformation)
>>
>> With conventional error handling, if any of these poison pills falls in the
>> scope of a bulk get operation, the entire operation fails.
> 
> I suspect many of these poison pills are design mistakes.
> 
> If a property is not valid for the configuration, why does it exist?
> QOM is by design dynamic.  I wish it wasn't, but as long as it is
> dynamic, I can't see why we should create properties we know to be
> unusable.
> 
> Why is reading crash-information an error when no crash occured?  This
> is the *normal* case.  Errors are for the abnormal.
> 
> Anyway, asking you to fix design mistakes all over the place wouldn't be
> fair.  So I'm asking you something else instead: do you actually need
> the error information?

I don't need the specific error message.

I could return a boolean meaning "property not available" instead of returning
the exact error message, as long as folks are OK with the output of the qom-tree
script changing for these properties.

- Steve

Re: [PATCH V1 0/6] fast qom tree get

[Markus Armbruster]

Steven Sistare <steven.sistare@oracle.com> writes:

> On 4/9/2025 9:34 AM, Markus Armbruster wrote:
>> Steven Sistare <steven.sistare@oracle.com> writes:
>>> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>>>> Hi Steve, I apologize for the slow response.
>>>>
>>>> Steve Sistare <steven.sistare@oracle.com> writes:
>>>>
>>>>> Using qom-list and qom-get to get all the nodes and property values in a
>>>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>>>> of it when starting a new VM, and this cost is a substantial fraction of
>>>>> start up time.
>>>>
>>>> "Some managers"... could you name one?
>>>
>>> My personal experience is with Oracle's OCI, but likely others could benefit.
>> 
>> Peter Krempa tells us libvirt would benefit.
>> 
>>>>> To reduce this cost, consider QAPI calls that fetch more information in
>>>>> each call:
>>>>>     * qom-list-get: given a path, return a list of properties and values.
>>>>>     * qom-list-getv: given a list of paths, return a list of properties and
>>>>>       values for each path.
>>>>>     * qom-tree-get: given a path, return all descendant nodes rooted at that
>>>>>       path, with properties and values for each.
>>>>
>>>> Libvirt developers, would you be interested in any of these?
>>>>
>>>>> In all cases, a returned property is represented by ObjectPropertyValue,
>>>>> with fields name, type, value, and error.  If an error occurs when reading
>>>>> a value, the value field is omitted, and the error message is returned in the
>>>>> the error field.  Thus an error for one property will not cause a bulk fetch
>>>>> operation to fail.
>>>>
>>>> Returning errors this way is highly unusual.  Observation; I'm not
>>>> rejecting this out of hand.  Can you elaborate a bit on why it's useful?
>>>
>>> It is considered an error to read some properties if they are not valid for
>>> the configuration.  And some properties are write-only and return an error
>>> if they are read.  Examples:
>>>
>>>     legacy-i8042: <EXCEPTION: Property 'vmmouse.legacy-i8042' is not readable> (str)
>>>     legacy-memory: <EXCEPTION: Property 'qemu64-x86_64-cpu.legacy-memory' is not readable> (str)
>>>     crash-information: <EXCEPTION: No crash occurred> (GuestPanicInformation)
>>>
>>> With conventional error handling, if any of these poison pills falls in the
>>> scope of a bulk get operation, the entire operation fails.
>> 
>> I suspect many of these poison pills are design mistakes.
>> 
>> If a property is not valid for the configuration, why does it exist?
>> QOM is by design dynamic.  I wish it wasn't, but as long as it is
>> dynamic, I can't see why we should create properties we know to be
>> unusable.
>> 
>> Why is reading crash-information an error when no crash occured?  This
>> is the *normal* case.  Errors are for the abnormal.
>> 
>> Anyway, asking you to fix design mistakes all over the place wouldn't be
>> fair.  So I'm asking you something else instead: do you actually need
>> the error information?
>
> I don't need the specific error message.
>
> I could return a boolean meaning "property not available" instead of returning
> the exact error message, as long as folks are OK with the output of the qom-tree
> script changing for these properties.

Let's put aside the qom-tree script for a moment.

In your patches, the queries return an object's properties as a list of
ObjectPropertyValue, defined as

    { 'struct': 'ObjectPropertyValue',
      'data': { 'name': 'str',
                'type': 'str',
                '*value': 'any',
                '*error': 'str' } }

As far as I understand, exactly one of @value and @error are present.

The list has no duplicates, i.e. no two elements have the same value of
"name".

Say we're interested in property "foo".  Three cases:

* The list has an element with "name": "foo", and the element has member
  "value": the property exists and "value" has its value.

* The list has an element with "name": "foo", and the element does not
  have member "value": the property exists, but its value cannot be
  gotten; member "error" has the error message.

* The list has no element with "name": "foo": the property does not
  exist.

If we simply drop ObjectPropertyValue member @error, we lose 'member
"error" has the error message'.  That's all.

If a need for more error information should arise later, we could add
member @error.  Or something else entirely.  Or tell people to qom-get
any properties qom-tree-get couldn't get for error information.  My
point is: dropping @error now does not tie our hands as far as I can
tell.

Back to qom-tree.  I believe this script is a development aid that
exists because qom-get is painful to use for humans.  Your qom-tree
command would completely obsolete it.  I wouldn't worry about it.
If you think I'm wrong there, please speak up!

Re: [PATCH V1 0/6] fast qom tree get

[Steven Sistare]

On 4/9/2025 10:44 AM, Markus Armbruster wrote:
> Steven Sistare <steven.sistare@oracle.com> writes:
> 
>> On 4/9/2025 9:34 AM, Markus Armbruster wrote:
>>> Steven Sistare <steven.sistare@oracle.com> writes:
>>>> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>>>>> Hi Steve, I apologize for the slow response.
>>>>>
>>>>> Steve Sistare <steven.sistare@oracle.com> writes:
>>>>>
>>>>>> Using qom-list and qom-get to get all the nodes and property values in a
>>>>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>>>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>>>>> of it when starting a new VM, and this cost is a substantial fraction of
>>>>>> start up time.
>>>>>
>>>>> "Some managers"... could you name one?
>>>>
>>>> My personal experience is with Oracle's OCI, but likely others could benefit.
>>>
>>> Peter Krempa tells us libvirt would benefit.
>>>
>>>>>> To reduce this cost, consider QAPI calls that fetch more information in
>>>>>> each call:
>>>>>>      * qom-list-get: given a path, return a list of properties and values.
>>>>>>      * qom-list-getv: given a list of paths, return a list of properties and
>>>>>>        values for each path.
>>>>>>      * qom-tree-get: given a path, return all descendant nodes rooted at that
>>>>>>        path, with properties and values for each.
>>>>>
>>>>> Libvirt developers, would you be interested in any of these?
>>>>>
>>>>>> In all cases, a returned property is represented by ObjectPropertyValue,
>>>>>> with fields name, type, value, and error.  If an error occurs when reading
>>>>>> a value, the value field is omitted, and the error message is returned in the
>>>>>> the error field.  Thus an error for one property will not cause a bulk fetch
>>>>>> operation to fail.
>>>>>
>>>>> Returning errors this way is highly unusual.  Observation; I'm not
>>>>> rejecting this out of hand.  Can you elaborate a bit on why it's useful?
>>>>
>>>> It is considered an error to read some properties if they are not valid for
>>>> the configuration.  And some properties are write-only and return an error
>>>> if they are read.  Examples:
>>>>
>>>>      legacy-i8042: <EXCEPTION: Property 'vmmouse.legacy-i8042' is not readable> (str)
>>>>      legacy-memory: <EXCEPTION: Property 'qemu64-x86_64-cpu.legacy-memory' is not readable> (str)
>>>>      crash-information: <EXCEPTION: No crash occurred> (GuestPanicInformation)
>>>>
>>>> With conventional error handling, if any of these poison pills falls in the
>>>> scope of a bulk get operation, the entire operation fails.
>>>
>>> I suspect many of these poison pills are design mistakes.
>>>
>>> If a property is not valid for the configuration, why does it exist?
>>> QOM is by design dynamic.  I wish it wasn't, but as long as it is
>>> dynamic, I can't see why we should create properties we know to be
>>> unusable.
>>>
>>> Why is reading crash-information an error when no crash occured?  This
>>> is the *normal* case.  Errors are for the abnormal.
>>>
>>> Anyway, asking you to fix design mistakes all over the place wouldn't be
>>> fair.  So I'm asking you something else instead: do you actually need
>>> the error information?
>>
>> I don't need the specific error message.
>>
>> I could return a boolean meaning "property not available" instead of returning
>> the exact error message, as long as folks are OK with the output of the qom-tree
>> script changing for these properties.
> 
> Let's put aside the qom-tree script for a moment.
> 
> In your patches, the queries return an object's properties as a list of
> ObjectPropertyValue, defined as
> 
>      { 'struct': 'ObjectPropertyValue',
>        'data': { 'name': 'str',
>                  'type': 'str',
>                  '*value': 'any',
>                  '*error': 'str' } }
> 
> As far as I understand, exactly one of @value and @error are present.
> 
> The list has no duplicates, i.e. no two elements have the same value of
> "name".
> 
> Say we're interested in property "foo".  Three cases:
> 
> * The list has an element with "name": "foo", and the element has member
>    "value": the property exists and "value" has its value.
> 
> * The list has an element with "name": "foo", and the element does not
>    have member "value": the property exists, but its value cannot be
>    gotten; member "error" has the error message.
> 
> * The list has no element with "name": "foo": the property does not
>    exist.
> 
> If we simply drop ObjectPropertyValue member @error, we lose 'member
> "error" has the error message'.  That's all.
> 
> If a need for more error information should arise later, we could add
> member @error.  Or something else entirely.  Or tell people to qom-get
> any properties qom-tree-get couldn't get for error information.  My
> point is: dropping @error now does not tie our hands as far as I can
> tell.

Agreed.  I forgot that I had defined value as an optional parameter,
so simply omitting it means "property not available".

> Back to qom-tree.  I believe this script is a development aid that
> exists because qom-get is painful to use for humans.  Your qom-tree
> command would completely obsolete it.  I wouldn't worry about it.
> If you think I'm wrong there, please speak up!

Regarding dropping the error messages, I agree, I was just pointing it out
in case anyone objected.

Yes, the new command plus a formatter like jq obsoletes the qom-tree script.
Just to be clear, I do not propose to delete the script, since folks are
accustomed to it being available, and are accustomed to its output.  It also
serves as a nice example for how to use the new command.

Do you want to review any code and specification now, or wait for me to send
V2 that deletes the error member?  The changes will be minor.

- Steve

Re: [PATCH V1 0/6] fast qom tree get

[Markus Armbruster]

Steven Sistare <steven.sistare@oracle.com> writes:

> On 4/9/2025 10:44 AM, Markus Armbruster wrote:
>> Steven Sistare <steven.sistare@oracle.com> writes:
>> 
>>> On 4/9/2025 9:34 AM, Markus Armbruster wrote:
>>>> Steven Sistare <steven.sistare@oracle.com> writes:
>>>>> On 4/9/2025 3:39 AM, Markus Armbruster wrote:

[...]

>>>> Anyway, asking you to fix design mistakes all over the place wouldn't be
>>>> fair.  So I'm asking you something else instead: do you actually need
>>>> the error information?
>>>
>>> I don't need the specific error message.
>>>
>>> I could return a boolean meaning "property not available" instead of returning
>>> the exact error message, as long as folks are OK with the output of the qom-tree
>>> script changing for these properties.
>> 
>> Let's put aside the qom-tree script for a moment.

[...]

>> Back to qom-tree.  I believe this script is a development aid that
>> exists because qom-get is painful to use for humans.  Your qom-tree
>> command would completely obsolete it.  I wouldn't worry about it.
>> If you think I'm wrong there, please speak up!
>
> Regarding dropping the error messages, I agree, I was just pointing it out
> in case anyone objected.

Appreciated.

> Yes, the new command plus a formatter like jq obsoletes the qom-tree script.
> Just to be clear, I do not propose to delete the script, since folks are
> accustomed to it being available, and are accustomed to its output.  It also
> serves as a nice example for how to use the new command.

I have little use for scripts/qmp/ myself.  Since nothing there adds to
my maintenance load appreciably, I don't mind keeping the scripts.
qom-fuse is rather cute.

> Do you want to review any code and specification now, or wait for me to send
> V2 that deletes the error member?  The changes will be minor.

v1 should do for review.  Thanks!

Re: [PATCH V1 0/6] fast qom tree get

[Markus Armbruster]

Steven Sistare <steven.sistare@oracle.com> writes:

> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>> Hi Steve, I apologize for the slow response.
>>
>> Steve Sistare <steven.sistare@oracle.com> writes:
>> 
>>> Using qom-list and qom-get to get all the nodes and property values in a
>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>> of it when starting a new VM, and this cost is a substantial fraction of
>>> start up time.
>>
>> "Some managers"... could you name one?
>
> My personal experience is with Oracle's OCI, but likely others could benefit.

Elsewhere in this thread, we examined libvirt's use qom-get.  Its use of
qom-get is also noticably slow, and your work could speed it up.
However, most of its use is for working around QMP interface
shortcomings around probing CPU flags.  Addressing these would help it
even more.

This makes me wonder what questions Oracle's OCI answers with the help
of qom-get.  Can you briefly describe them?

Even if OCI would likewise be helped more by better QMP queries, your
fast qom tree get work might still be useful.

Re: [PATCH V1 0/6] fast qom tree get

[Steven Sistare]

On 4/28/2025 4:04 AM, Markus Armbruster wrote:
> Steven Sistare <steven.sistare@oracle.com> writes:
> 
>> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>>> Hi Steve, I apologize for the slow response.
>>>
>>> Steve Sistare <steven.sistare@oracle.com> writes:
>>>
>>>> Using qom-list and qom-get to get all the nodes and property values in a
>>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>>> of it when starting a new VM, and this cost is a substantial fraction of
>>>> start up time.
>>>
>>> "Some managers"... could you name one?
>>
>> My personal experience is with Oracle's OCI, but likely others could benefit.
> 
> Elsewhere in this thread, we examined libvirt's use qom-get.  Its use of
> qom-get is also noticably slow, and your work could speed it up.
> However, most of its use is for working around QMP interface
> shortcomings around probing CPU flags.  Addressing these would help it
> even more.
> 
> This makes me wonder what questions Oracle's OCI answers with the help
> of qom-get.  Can you briefly describe them?
> 
> Even if OCI would likewise be helped more by better QMP queries, your
> fast qom tree get work might still be useful.

We already optimized our queries as a first step, but what remains is still
significant, which is why I submitted this RFE.

- Steve

Re: [PATCH V1 0/6] fast qom tree get

[Markus Armbruster]

Steven Sistare <steven.sistare@oracle.com> writes:

> On 4/28/2025 4:04 AM, Markus Armbruster wrote:
>> Steven Sistare <steven.sistare@oracle.com> writes:
>> 
>>> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>>>> Hi Steve, I apologize for the slow response.
>>>>
>>>> Steve Sistare <steven.sistare@oracle.com> writes:
>>>>
>>>>> Using qom-list and qom-get to get all the nodes and property values in a
>>>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>>>> of it when starting a new VM, and this cost is a substantial fraction of
>>>>> start up time.
>>>>
>>>> "Some managers"... could you name one?
>>>
>>> My personal experience is with Oracle's OCI, but likely others could benefit.
>> 
>> Elsewhere in this thread, we examined libvirt's use qom-get.  Its use of
>> qom-get is also noticably slow, and your work could speed it up.
>> However, most of its use is for working around QMP interface
>> shortcomings around probing CPU flags.  Addressing these would help it
>> even more.
>> 
>> This makes me wonder what questions Oracle's OCI answers with the help
>> of qom-get.  Can you briefly describe them?
>> 
>> Even if OCI would likewise be helped more by better QMP queries, your
>> fast qom tree get work might still be useful.
>
> We already optimized our queries as a first step, but what remains is still
> significant, which is why I submitted this RFE.

I understand your motivation.  I'd like to learn more on what OCI
actually needs from QMP, to be able to better serve it and potentially
other management applications.

Re: [PATCH V1 0/6] fast qom tree get

[Steven Sistare]

On 4/29/2025 2:02 AM, Markus Armbruster wrote:
> Steven Sistare <steven.sistare@oracle.com> writes:
> 
>> On 4/28/2025 4:04 AM, Markus Armbruster wrote:
>>> Steven Sistare <steven.sistare@oracle.com> writes:
>>>
>>>> On 4/9/2025 3:39 AM, Markus Armbruster wrote:
>>>>> Hi Steve, I apologize for the slow response.
>>>>>
>>>>> Steve Sistare <steven.sistare@oracle.com> writes:
>>>>>
>>>>>> Using qom-list and qom-get to get all the nodes and property values in a
>>>>>> QOM tree can take multiple seconds because it requires 1000's of individual
>>>>>> QOM requests.  Some managers fetch the entire tree or a large subset
>>>>>> of it when starting a new VM, and this cost is a substantial fraction of
>>>>>> start up time.
>>>>>
>>>>> "Some managers"... could you name one?
>>>>
>>>> My personal experience is with Oracle's OCI, but likely others could benefit.
>>>
>>> Elsewhere in this thread, we examined libvirt's use qom-get.  Its use of
>>> qom-get is also noticably slow, and your work could speed it up.
>>> However, most of its use is for working around QMP interface
>>> shortcomings around probing CPU flags.  Addressing these would help it
>>> even more.
>>>
>>> This makes me wonder what questions Oracle's OCI answers with the help
>>> of qom-get.  Can you briefly describe them?
>>>
>>> Even if OCI would likewise be helped more by better QMP queries, your
>>> fast qom tree get work might still be useful.
>>
>> We already optimized our queries as a first step, but what remains is still
>> significant, which is why I submitted this RFE.
> 
> I understand your motivation.  I'd like to learn more on what OCI
> actually needs from QMP, to be able to better serve it and potentially
> other management applications.

OCI uses qemu as the single source of truth during hot plug operations,
comparing what is actually there to what the user desires, and as such
must request many nodes and their properties from qemu.

Regarding being potentially useful to other management applications, my
proposed interface is quite general and easy to use, and I don't believe
we will improve it by examining more use cases in detail.  I'd appreciate
it if we could continue reviewing the code.  I made a concerted effort to dot
all the i's and cross all the t's, based on what you look for in our past
discussions of other patches.

- Steve