[PATCH 0/5] enhance debugging with qtest framework

[PATCH 0/5] enhance debugging with qtest framework

[qemu_oss--- via]

This series is a follow-up of the following previous series:
https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg02251.html
The 9p patches of the previous series have already been merged.

This series consists of 2 parts:

1. libqos patch 1 removes a limitation of the qtest/libqos subsystem:
   support for more than one device using the same (official) QEMU device
   name.

   Like discussed in the previous series, if nobody finds this patch useful
   then just ignore it. I needed it in the previou series before but
   eventually decided for a different approach and personally don't need it
   in near future.

2. Patches 2 to 5 enhance debugging issues with the qtest framework. I would
   appreciate if they got merged, because I still find them useful while
   working on new test cases.

Changes of these patches from derived series:

  * Squashed previous patches 1 & 2 -> [patch 1].

  * Dropped ANSI color escape sequences [patch 3].

  * Squashed previous patches 4 & 5 -> [patch 3].

  * Extended commit log to provide more details about purpose [patch 4].

Christian Schoenebeck (5):
  libqos/qgraph: add qos_node_create_driver_named()
  libqos/qgraph_internal: add qos_printf() and qos_printf_literal()
  tests/qtest/qos-test: dump qos graph if verbose
  tests/qtest/qos-test: dump environment variables if verbose
  tests/qtest/qos-test: dump QEMU command if verbose

 tests/qtest/libqos/qgraph.c          | 99 +++++++++++++++++++++++++++-
 tests/qtest/libqos/qgraph.h          | 36 ++++++++++
 tests/qtest/libqos/qgraph_internal.h | 12 ++++
 tests/qtest/qos-test.c               | 15 ++++-
 4 files changed, 158 insertions(+), 4 deletions(-)

-- 
2.20.1

[PATCH 1/5] libqos/qgraph: add qos_node_create_driver_named()

[qemu_oss--- via]

So far the qos subsystem of the qtest framework had the limitation
that only one instance of the same official QEMU (QMP) driver name
could be created for qtests. That's because a) the created qos
node names must always be unique, b) the node name must match the
official QEMU driver name being instantiated and c) all nodes are
in a global space shared by all tests.

This patch removes this limitation by introducing a new function
qos_node_create_driver_named() which allows test case authors to
specify a node name being different from the actual associated
QEMU driver name. It fills the new 'qemu_name' field of
QOSGraphNode for that purpose.

Adjust build_driver_cmd_line() and qos_graph_node_set_availability()
to correctly deal with either accessing node name vs. node's
qemu_name correctly.

Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
---
 tests/qtest/libqos/qgraph.c          | 54 ++++++++++++++++++++++++++--
 tests/qtest/libqos/qgraph.h          | 16 +++++++++
 tests/qtest/libqos/qgraph_internal.h |  1 +
 3 files changed, 68 insertions(+), 3 deletions(-)

diff --git a/tests/qtest/libqos/qgraph.c b/tests/qtest/libqos/qgraph.c
index fc49cfa879..61faf6b27d 100644
--- a/tests/qtest/libqos/qgraph.c
+++ b/tests/qtest/libqos/qgraph.c
@@ -153,6 +153,7 @@ static QOSGraphNode *create_node(const char *name, QOSNodeType type)
 static void destroy_node(void *val)
 {
     QOSGraphNode *node = val;
+    g_free(node->qemu_name);
     g_free(node->command_line);
     g_free(node);
 }
@@ -286,7 +287,8 @@ static void build_machine_cmd_line(QOSGraphNode *node, const char *args)
  */
 static void build_driver_cmd_line(QOSGraphNode *node)
 {
-    node->command_line = g_strconcat(" -device ", node->name, NULL);
+    const char *name = node->qemu_name ?: node->name;
+    node->command_line = g_strconcat(" -device ", name, NULL);
 }
 
 /* qos_print_cb(): callback prints all path found by the DFS algorithm. */
@@ -631,6 +633,15 @@ void qos_node_create_driver(const char *name, QOSCreateDriverFunc function)
     node->u.driver.constructor = function;
 }
 
+void qos_node_create_driver_named(const char *name, const char *qemu_name,
+                                  QOSCreateDriverFunc function)
+{
+    QOSGraphNode *node = create_node(name, QNODE_DRIVER);
+    node->qemu_name = g_strdup(qemu_name);
+    build_driver_cmd_line(node);
+    node->u.driver.constructor = function;
+}
+
 void qos_node_contains(const char *container, const char *contained,
                        QOSGraphEdgeOptions *opts, ...)
 {
@@ -663,7 +674,7 @@ void qos_node_consumes(const char *consumer, const char *interface,
     add_edge(interface, consumer, QEDGE_CONSUMED_BY, opts);
 }
 
-void qos_graph_node_set_availability(const char *node, bool av)
+static void qos_graph_node_set_availability_explicit(const char *node, bool av)
 {
     QOSGraphEdgeList *elist;
     QOSGraphNode *n = search_node(node);
@@ -678,9 +689,46 @@ void qos_graph_node_set_availability(const char *node, bool av)
     }
     QSLIST_FOREACH_SAFE(e, elist, edge_list, next) {
         if (e->type == QEDGE_CONTAINS || e->type == QEDGE_PRODUCES) {
-            qos_graph_node_set_availability(e->dest, av);
+            qos_graph_node_set_availability_explicit(e->dest, av);
+        }
+    }
+}
+
+/*
+ * Behaves as qos_graph_node_set_availability_explicit(), except that the
+ * former always matches by node name only, whereas this function matches both
+ * by node name and node's optional 'qemu_name' field.
+ */
+void qos_graph_node_set_availability(const char *node, bool av)
+{
+    GList *l;
+    QOSGraphEdgeList *elist;
+    QOSGraphEdge *e, *next;
+    QOSGraphNode *n;
+    GList *keys = g_hash_table_get_keys(node_table);
+
+    for (l = keys; l != NULL; l = l->next) {
+        const gchar *key = l->data;
+        n = g_hash_table_lookup(node_table, key);
+        /*
+         * node's 'qemu_name' is set if there is more than one device with
+         * the same QEMU (QMP) device name
+         */
+        const char *node_name = n->qemu_name ?: n->name;
+        if (g_strcmp0(node_name, node) == 0) {
+            n->available = av;
+            elist = get_edgelist(n->name);
+            if (elist) {
+                QSLIST_FOREACH_SAFE(e, elist, edge_list, next) {
+                    if (e->type == QEDGE_CONTAINS || e->type == QEDGE_PRODUCES)
+                    {
+                        qos_graph_node_set_availability_explicit(e->dest, av);
+                    }
+                }
+            }
         }
     }
+    g_list_free(keys);
 }
 
 void qos_graph_foreach_test_path(QOSTestCallback fn)
diff --git a/tests/qtest/libqos/qgraph.h b/tests/qtest/libqos/qgraph.h
index 5f63d352ca..f472949f68 100644
--- a/tests/qtest/libqos/qgraph.h
+++ b/tests/qtest/libqos/qgraph.h
@@ -452,6 +452,22 @@ void qos_node_create_machine_args(const char *name,
  */
 void qos_node_create_driver(const char *name, QOSCreateDriverFunc function);
 
+/**
+ * Behaves as qos_node_create_driver() with the extension of allowing to
+ * specify a different node name vs. associated QEMU device name.
+ *
+ * Use this function instead of qos_node_create_driver() if you need to create
+ * several instances of the same QEMU device. You are free to choose a custom
+ * node name, however the chosen node name must always be unique.
+ *
+ * @param name: custom, unique name of the node to be created
+ * @param qemu_name: actual (official) QEMU driver name the node shall be
+ *                   associated with
+ * @param function: driver constructor
+ */
+void qos_node_create_driver_named(const char *name, const char *qemu_name,
+                                  QOSCreateDriverFunc function);
+
 /**
  * qos_node_contains(): creates one or more edges of type QEDGE_CONTAINS
  * and adds them to the edge list mapped to @container in the
diff --git a/tests/qtest/libqos/qgraph_internal.h b/tests/qtest/libqos/qgraph_internal.h
index 968fa69450..974985dce9 100644
--- a/tests/qtest/libqos/qgraph_internal.h
+++ b/tests/qtest/libqos/qgraph_internal.h
@@ -56,6 +56,7 @@ struct QOSGraphNode {
     bool available;     /* set by QEMU via QMP, used during graph walk */
     bool visited;       /* used during graph walk */
     char *name;         /* used to identify the node */
+    char *qemu_name;    /* optional: see qos_node_create_driver_named() */
     char *command_line; /* used to start QEMU at test execution */
     union {
         struct {
-- 
2.20.1

[PATCH 2/5] libqos/qgraph_internal: add qos_printf() and qos_printf_literal()

[qemu_oss--- via]

These two are macros wrapping regular printf() call. They are intended
to be used instead of calling printf() directly in order to avoid
breaking TAP output format.

TAP output format is enabled by using --tap command line argument.
Starting with glib 2.62 it is enabled by default.

Unfortunately there is currently no public glib API available to check
whether TAP output format is enabled. For that reason qos_printf()
simply always prepends a '#' character for now.

Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
---
 tests/qtest/libqos/qgraph_internal.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/qtest/libqos/qgraph_internal.h b/tests/qtest/libqos/qgraph_internal.h
index 974985dce9..c0025f5ab9 100644
--- a/tests/qtest/libqos/qgraph_internal.h
+++ b/tests/qtest/libqos/qgraph_internal.h
@@ -255,4 +255,15 @@ void qos_delete_cmd_line(const char *name);
  */
 void qos_graph_node_set_availability(const char *node, bool av);
 
+/*
+ * Prepends a '#' character in front for not breaking TAP output format.
+ */
+#define qos_printf(...) printf("# " __VA_ARGS__)
+
+/*
+ * Intended for printing something literally, i.e. for appending text as is
+ * to a line already been started by qos_printf() before.
+ */
+#define qos_printf_literal printf
+
 #endif
-- 
2.20.1

Re: [PATCH 2/5] libqos/qgraph_internal: add qos_printf() and qos_printf_literal()

[Thomas Huth]

On 27/01/2021 00.04, Christian Schoenebeck wrote:
> These two are macros wrapping regular printf() call. They are intended
> to be used instead of calling printf() directly in order to avoid
> breaking TAP output format.
> 
> TAP output format is enabled by using --tap command line argument.
> Starting with glib 2.62 it is enabled by default.
> 
> Unfortunately there is currently no public glib API available to check
> whether TAP output format is enabled. For that reason qos_printf()
> simply always prepends a '#' character for now.
> 
> Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
> ---
>   tests/qtest/libqos/qgraph_internal.h | 11 +++++++++++
>   1 file changed, 11 insertions(+)
> 
> diff --git a/tests/qtest/libqos/qgraph_internal.h b/tests/qtest/libqos/qgraph_internal.h
> index 974985dce9..c0025f5ab9 100644
> --- a/tests/qtest/libqos/qgraph_internal.h
> +++ b/tests/qtest/libqos/qgraph_internal.h
> @@ -255,4 +255,15 @@ void qos_delete_cmd_line(const char *name);
>    */
>   void qos_graph_node_set_availability(const char *node, bool av);
>   
> +/*
> + * Prepends a '#' character in front for not breaking TAP output format.
> + */
> +#define qos_printf(...) printf("# " __VA_ARGS__)
> +
> +/*
> + * Intended for printing something literally, i.e. for appending text as is
> + * to a line already been started by qos_printf() before.
> + */
> +#define qos_printf_literal printf

I'd maybe rather name it qos_printf_append ... but that's just a matter of 
taste.

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/5] libqos/qgraph_internal: add qos_printf() and qos_printf_literal()

[Christian Schoenebeck]

On Donnerstag, 28. Januar 2021 11:13:27 CET Thomas Huth wrote:
> On 27/01/2021 00.04, Christian Schoenebeck wrote:
> > These two are macros wrapping regular printf() call. They are intended
> > to be used instead of calling printf() directly in order to avoid
> > breaking TAP output format.
> > 
> > TAP output format is enabled by using --tap command line argument.
> > Starting with glib 2.62 it is enabled by default.
> > 
> > Unfortunately there is currently no public glib API available to check
> > whether TAP output format is enabled. For that reason qos_printf()
> > simply always prepends a '#' character for now.
> > 
> > Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
> > ---
> > 
> >   tests/qtest/libqos/qgraph_internal.h | 11 +++++++++++
> >   1 file changed, 11 insertions(+)
> > 
> > diff --git a/tests/qtest/libqos/qgraph_internal.h
> > b/tests/qtest/libqos/qgraph_internal.h index 974985dce9..c0025f5ab9
> > 100644
> > --- a/tests/qtest/libqos/qgraph_internal.h
> > +++ b/tests/qtest/libqos/qgraph_internal.h
> > @@ -255,4 +255,15 @@ void qos_delete_cmd_line(const char *name);
> > 
> >    */
> >   
> >   void qos_graph_node_set_availability(const char *node, bool av);
> > 
> > +/*
> > + * Prepends a '#' character in front for not breaking TAP output format.
> > + */
> > +#define qos_printf(...) printf("# " __VA_ARGS__)
> > +
> > +/*
> > + * Intended for printing something literally, i.e. for appending text as
> > is + * to a line already been started by qos_printf() before.
> > + */
> > +#define qos_printf_literal printf
> 
> I'd maybe rather name it qos_printf_append ... but that's just a matter of
> taste.
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>

PING

Best regards,
Christian Schoenebeck

[PATCH 3/5] tests/qtest/qos-test: dump qos graph if verbose

[qemu_oss--- via]

If qtests were run in verbose mode (i.e. if --verbose CL argument was
provided) then dump the generated qos graph (all nodes and edges,
along with their current individual availability status) to stdout,
which allows to identify problems in the created qos graph e.g. when
writing new qos tests.

See API doc comment on function qos_dump_graph() for details.

Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
---
 tests/qtest/libqos/qgraph.c | 45 +++++++++++++++++++++++++++++++++++++
 tests/qtest/libqos/qgraph.h | 20 +++++++++++++++++
 tests/qtest/qos-test.c      |  3 +++
 3 files changed, 68 insertions(+)

diff --git a/tests/qtest/libqos/qgraph.c b/tests/qtest/libqos/qgraph.c
index 61faf6b27d..b3b1a31f81 100644
--- a/tests/qtest/libqos/qgraph.c
+++ b/tests/qtest/libqos/qgraph.c
@@ -805,3 +805,48 @@ void qos_delete_cmd_line(const char *name)
         node->command_line = NULL;
     }
 }
+
+void qos_dump_graph(void)
+{
+    GList *keys;
+    GList *l;
+    QOSGraphEdgeList *list;
+    QOSGraphEdge *e, *next;
+    QOSGraphNode *dest_node, *node;
+
+    qos_printf("ALL QGRAPH EDGES: {\n");
+    keys = g_hash_table_get_keys(edge_table);
+    for (l = keys; l != NULL; l = l->next) {
+        const gchar *key = l->data;
+        qos_printf("\t src='%s'\n", key);
+        list = get_edgelist(key);
+        QSLIST_FOREACH_SAFE(e, list, edge_list, next) {
+            dest_node = g_hash_table_lookup(node_table, e->dest);
+            qos_printf("\t\t|-> dest='%s' type=%d (node=%p)",
+                       e->dest, e->type, dest_node);
+            if (!dest_node) {
+                qos_printf_literal(" <------- ERROR !");
+            }
+            qos_printf_literal("\n");
+        }
+    }
+    g_list_free(keys);
+    qos_printf("}\n");
+
+    qos_printf("ALL QGRAPH NODES: {\n");
+    keys = g_hash_table_get_keys(node_table);
+    for (l = keys; l != NULL; l = l->next) {
+        const gchar *key = l->data;
+        node = g_hash_table_lookup(node_table, key);
+        qos_printf("\t name='%s' ", key);
+        if (node->qemu_name) {
+            qos_printf_literal("qemu_name='%s' ", node->qemu_name);
+        }
+        qos_printf_literal("type=%d cmd_line='%s' [%s]\n",
+                           node->type, node->command_line,
+                           node->available ? "available" : "UNAVAILBLE"
+        );
+    }
+    g_list_free(keys);
+    qos_printf("}\n");
+}
diff --git a/tests/qtest/libqos/qgraph.h b/tests/qtest/libqos/qgraph.h
index f472949f68..07a32535f1 100644
--- a/tests/qtest/libqos/qgraph.h
+++ b/tests/qtest/libqos/qgraph.h
@@ -586,5 +586,25 @@ QOSGraphObject *qos_machine_new(QOSGraphNode *node, QTestState *qts);
 QOSGraphObject *qos_driver_new(QOSGraphNode *node, QOSGraphObject *parent,
                                QGuestAllocator *alloc, void *arg);
 
+/**
+ * Just for debugging purpose: prints all currently existing nodes and
+ * edges to stdout.
+ *
+ * All qtests add themselves to the overall qos graph by calling qgraph
+ * functions that add device nodes and edges between the individual graph
+ * nodes for tests. As the actual graph is assmbled at runtime by the qos
+ * subsystem, it is sometimes not obvious how the overall graph looks like.
+ * E.g. when writing new tests it may happen that those new tests are simply
+ * ignored by the qtest framework.
+ *
+ * This function allows to identify problems in the created qgraph. Keep in
+ * mind: only tests with a path down from the actual test case node (leaf) up
+ * to the graph's root node are actually executed by the qtest framework. And
+ * the qtest framework uses QMP to automatically check which QEMU drivers are
+ * actually currently available, and accordingly qos marks certain pathes as
+ * 'unavailable' in such cases (e.g. when QEMU was compiled without support for
+ * a certain feature).
+ */
+void qos_dump_graph(void);
 
 #endif
diff --git a/tests/qtest/qos-test.c b/tests/qtest/qos-test.c
index 8fdf87b183..d98ef78613 100644
--- a/tests/qtest/qos-test.c
+++ b/tests/qtest/qos-test.c
@@ -322,6 +322,9 @@ int main(int argc, char **argv)
     qos_set_machines_devices_available();
 
     qos_graph_foreach_test_path(walk_path);
+    if (g_test_verbose()) {
+        qos_dump_graph();
+    }
     g_test_run();
     qtest_end();
     qos_graph_destroy();
-- 
2.20.1

[PATCH 4/5] tests/qtest/qos-test: dump environment variables if verbose

[qemu_oss--- via]

If qtests are run in verbose mode (i.e. if --verbose CL argument
was provided) then print all environment variables to stdout
before running the individual tests.

It is common nowadays, at least being able to output all config
vectors in a build chain, especially if it is required to
investigate build- and test-issues on foreign/remote machines,
which includes environment variables. In the context of writing
new test cases this is also useful for finding out whether there
are already some existing options for common questions like is
there a preferred location for writing test files to? Is there
a maximum size for test data? Is there a deadline for running
tests?

Use qos_printf() instead of g_test_message() to avoid the latter
cluttering the output.

Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
---
 tests/qtest/qos-test.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/tests/qtest/qos-test.c b/tests/qtest/qos-test.c
index d98ef78613..b279b6f816 100644
--- a/tests/qtest/qos-test.c
+++ b/tests/qtest/qos-test.c
@@ -313,9 +313,16 @@ static void walk_path(QOSGraphNode *orig_path, int len)
  *   machine/drivers/test objects
  * - Cleans up everything
  */
-int main(int argc, char **argv)
+int main(int argc, char **argv, char** envp)
 {
     g_test_init(&argc, &argv, NULL);
+    if (g_test_verbose()) {
+        qos_printf("ENVIRONMENT VARIABLES: {\n");
+        for (char **env = envp; *env != 0; env++) {
+            qos_printf("\t%s\n", *env);
+        }
+        qos_printf("}\n");
+    }
     qos_graph_init();
     module_call_init(MODULE_INIT_QOM);
     module_call_init(MODULE_INIT_LIBQOS);
-- 
2.20.1

[PATCH 5/5] tests/qtest/qos-test: dump QEMU command if verbose

[qemu_oss--- via]

If qtests are run in verbose mode (i.e. if --verbose CL argument
was provided) then print the assembled qemu command line for each
test.

Use qos_printf() instead of g_test_message() to avoid the latter
cluttering the output.

Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
---
 tests/qtest/qos-test.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/qtest/qos-test.c b/tests/qtest/qos-test.c
index b279b6f816..f97d0a08fd 100644
--- a/tests/qtest/qos-test.c
+++ b/tests/qtest/qos-test.c
@@ -89,6 +89,9 @@ static void qos_set_machines_devices_available(void)
 
 static void restart_qemu_or_continue(char *path)
 {
+    if (g_test_verbose()) {
+        qos_printf("Run QEMU with: '%s'\n", path);
+    }
     /* compares the current command line with the
      * one previously executed: if they are the same,
      * don't restart QEMU, if they differ, stop previous
-- 
2.20.1

Re: [PATCH 0/5] enhance debugging with qtest framework

[Paolo Bonzini]

On 27/01/21 00:36, Christian Schoenebeck wrote:
> This series is a follow-up of the following previous series:
> https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg02251.html
> The 9p patches of the previous series have already been merged.
> 
> This series consists of 2 parts:
> 
> 1. libqos patch 1 removes a limitation of the qtest/libqos subsystem:
>     support for more than one device using the same (official) QEMU device
>     name.
> 
>     Like discussed in the previous series, if nobody finds this patch useful
>     then just ignore it. I needed it in the previou series before but
>     eventually decided for a different approach and personally don't need it
>     in near future.
> 
> 2. Patches 2 to 5 enhance debugging issues with the qtest framework. I would
>     appreciate if they got merged, because I still find them useful while
>     working on new test cases.
> 
> Changes of these patches from derived series:
> 
>    * Squashed previous patches 1 & 2 -> [patch 1].
> 
>    * Dropped ANSI color escape sequences [patch 3].
> 
>    * Squashed previous patches 4 & 5 -> [patch 3].
> 
>    * Extended commit log to provide more details about purpose [patch 4].
> 
> Christian Schoenebeck (5):
>    libqos/qgraph: add qos_node_create_driver_named()
>    libqos/qgraph_internal: add qos_printf() and qos_printf_literal()
>    tests/qtest/qos-test: dump qos graph if verbose
>    tests/qtest/qos-test: dump environment variables if verbose
>    tests/qtest/qos-test: dump QEMU command if verbose
> 
>   tests/qtest/libqos/qgraph.c          | 99 +++++++++++++++++++++++++++-
>   tests/qtest/libqos/qgraph.h          | 36 ++++++++++
>   tests/qtest/libqos/qgraph_internal.h | 12 ++++
>   tests/qtest/qos-test.c               | 15 ++++-
>   4 files changed, 158 insertions(+), 4 deletions(-)
> 

Queued, thanks.

Paolo