[Qemu-devel] [PATCH v3 0/2] Fixes/tests for hmp_object_del()

[Qemu-devel] [PATCH v3 0/2] Fixes/tests for hmp_object_del()

[Michael Roth]

hmp_object_del() followed by a subsequent hmp_object_add() can trigger a
duplicate ID error if the previous object shared the same ID and was added
via the command-line. Please see patch 2/2 for more details.

This patchset fixes the issue in question and adds some general unit tests
for object created via -object, which we later extend to verify the fix in
question.

Changes since v3:
  - Moved the generic unit tests ahead of the fix patch, with a FIXME
    in place of the actual check for the failure addressed in patch
    2/2 (Daniel/Markus)
  - Dropped check for existence of objects' QemuOptsList (Markus)
  - Dropped unintended whitespace removal in PATCH 1/2
  - Slight rewording of commit messages to reflect the changes and fix
    minor grammar errors.

Changes since v2:

  - Moved QemuOpt cleanup out of {qmp,hmp}_object_del() and into common
    user_creatable_del() path (Daniel, David)
  - Added corresponding test case in check-qom-proplist

[Qemu-devel] [PATCH v3 1/2] tests: check-qom-proplist: add checks for cmdline-created objects

[Michael Roth]

check-qom-proplist originally added tests for verifying that
object-creation helpers object_new_with_{props,propv} behaved in
similar fashion to the "traditional" method involving setting each
individual property separately after object creation rather than
via a single call.

Another similar "helper" for creating Objects exists in the form of
objects specified via -object command-line parameters. By that
rationale, we extend check-qom-proplist to include similar checks
for command-line-created objects by employing the same
qemu_opts_parse()-based parsing the vl.c employs.

This parser has a side-effect of parsing the object's options into
a QemuOpt structure and registering this in the global QemuOptsList
using the Object's ID. This can conflict with future Object instances
that attempt to use the same ID if we don't ensure this is cleaned
up as part of Object finalization, so we include a FIXME stub to test
for this case, which will then be resolved in a subsequent patch.

Suggested-by: Daniel Berrange <berrange@redhat.com>
Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Daniel Berrange <berrange@redhat.com>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
 tests/check-qom-proplist.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/tests/check-qom-proplist.c b/tests/check-qom-proplist.c
index a16cefc..e3f56ca 100644
--- a/tests/check-qom-proplist.c
+++ b/tests/check-qom-proplist.c
@@ -23,6 +23,9 @@
 #include "qapi/error.h"
 #include "qom/object.h"
 #include "qemu/module.h"
+#include "qemu/option.h"
+#include "qemu/config-file.h"
+#include "qom/object_interfaces.h"
 
 
 #define TYPE_DUMMY "qemu-dummy"
@@ -162,6 +165,10 @@ static const TypeInfo dummy_info = {
     .instance_finalize = dummy_finalize,
     .class_size = sizeof(DummyObjectClass),
     .class_init = dummy_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_USER_CREATABLE },
+        { }
+    }
 };
 
 
@@ -320,6 +327,14 @@ static const TypeInfo dummy_backend_info = {
     .class_size = sizeof(DummyBackendClass),
 };
 
+static QemuOptsList qemu_object_opts = {
+    .name = "object",
+    .implied_opt_name = "qom-type",
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
+    .desc = {
+        { }
+    },
+};
 
 
 static void test_dummy_createv(void)
@@ -388,6 +403,45 @@ static void test_dummy_createlist(void)
     object_unparent(OBJECT(dobj));
 }
 
+static void test_dummy_createcmdl(void)
+{
+    QemuOpts *opts;
+    DummyObject *dobj;
+    Error *err = NULL;
+    const char *params = TYPE_DUMMY \
+                         ",id=dev0," \
+                         "bv=yes,sv=Hiss hiss hiss,av=platypus";
+
+    qemu_add_opts(&qemu_object_opts);
+    opts = qemu_opts_parse(&qemu_object_opts, params, true, &err);
+    g_assert(err == NULL);
+    g_assert(opts);
+
+    dobj = DUMMY_OBJECT(user_creatable_add_opts(opts, &err));
+    g_assert(err == NULL);
+    g_assert(dobj);
+    g_assert_cmpstr(dobj->sv, ==, "Hiss hiss hiss");
+    g_assert(dobj->bv == true);
+    g_assert(dobj->av == DUMMY_PLATYPUS);
+
+    user_creatable_del("dev0", &err);
+    g_assert(err == NULL);
+    error_free(err);
+
+    /* cmdline-parsing via qemu_opts_parse() results in a QemuOpts entry
+     * corresponding to the Object's ID to be added to the QemuOptsList
+     * for objects. To avoid having this entry conflict with future
+     * Objects using the same ID (which can happen in cases where
+     * qemu_opts_parse() is used to parse the object params, such as
+     * with hmp_object_add() at the time of this comment), we need to
+     * check for this in user_creatable_del() and remove the QemuOpts if
+     * it is present.
+     *
+     * FIXME: add an assert to verify that the QemuOpts is cleaned up
+     * once the corresponding cleanup code is added.
+     */
+}
+
 static void test_dummy_badenum(void)
 {
     Error *err = NULL;
@@ -525,6 +579,7 @@ int main(int argc, char **argv)
 
     g_test_add_func("/qom/proplist/createlist", test_dummy_createlist);
     g_test_add_func("/qom/proplist/createv", test_dummy_createv);
+    g_test_add_func("/qom/proplist/createcmdline", test_dummy_createcmdl);
     g_test_add_func("/qom/proplist/badenum", test_dummy_badenum);
     g_test_add_func("/qom/proplist/getenum", test_dummy_getenum);
     g_test_add_func("/qom/proplist/iterator", test_dummy_iterator);
-- 
1.9.1

[Qemu-devel] [PATCH v3 2/2] monitor: fix object_del for command-line-created objects

[Michael Roth]

Currently objects specified on the command-line are only partially
cleaned up when 'object_del' is issued in either HMP or QMP: the
object itself is fully finalized, but the QemuOpts are not removed.
This results in the following behavior:

  x86_64-softmmu/qemu-system-x86_64 -monitor stdio \
    -object memory-backend-ram,id=ram1,size=256M

  QEMU 2.7.91 monitor - type 'help' for more information
  (qemu) object_del ram1
  (qemu) object_del ram1
  object 'ram1' not found
  (qemu) object_add memory-backend-ram,id=ram1,size=256M
  Duplicate ID 'ram1' for object
  Try "help object_add" for more information

which can be an issue for use-cases like memory hotplug.

This happens on the HMP side because hmp_object_add() attempts to
create a temporary QemuOpts entry with ID 'ram1', which ends up
conflicting with the command-line-created entry, since it was never
cleaned up during the previous hmp_object_del() call.

We address this by adding a check in user_creatable_del(), which
is called by both qmp_object_del() and hmp_object_del() to handle
the actual object cleanup, to determine whether an option group entry
matching the object's ID is present and removing it if it is.

Note that qmp_object_add() never attempts to create a temporary
QemuOpts entry, so it does not encounter the duplicate ID error,
which is why this isn't generally visible in libvirt.

Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Daniel Berrange <berrange@redhat.com>
Cc: qemu-stable@nongnu.org
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
---
 qom/object_interfaces.c    | 7 +++++++
 tests/check-qom-proplist.c | 4 ++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/qom/object_interfaces.c b/qom/object_interfaces.c
index ded4d84..b462a74 100644
--- a/qom/object_interfaces.c
+++ b/qom/object_interfaces.c
@@ -5,6 +5,7 @@
 #include "qapi-visit.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/opts-visitor.h"
+#include "qemu/config-file.h"
 
 void user_creatable_complete(Object *obj, Error **errp)
 {
@@ -209,6 +210,12 @@ void user_creatable_del(const char *id, Error **errp)
         error_setg(errp, "object '%s' is in use, can not be deleted", id);
         return;
     }
+
+    /* if object was defined on the command-line, remove its corresponding
+     * option group entry
+     */
+    qemu_opts_del(qemu_opts_find(qemu_find_opts_err("object", NULL), id));
+
     object_unparent(obj);
 }
 
diff --git a/tests/check-qom-proplist.c b/tests/check-qom-proplist.c
index e3f56ca..c5d4b60 100644
--- a/tests/check-qom-proplist.c
+++ b/tests/check-qom-proplist.c
@@ -437,9 +437,9 @@ static void test_dummy_createcmdl(void)
      * check for this in user_creatable_del() and remove the QemuOpts if
      * it is present.
      *
-     * FIXME: add an assert to verify that the QemuOpts is cleaned up
-     * once the corresponding cleanup code is added.
+     * The below check ensures this works as expected.
      */
+    g_assert(qemu_opts_find(&qemu_object_opts, "dev0") == NULL);
 }
 
 static void test_dummy_badenum(void)
-- 
1.9.1

Re: [Qemu-devel] [PATCH v3 2/2] monitor: fix object_del for command-line-created objects

[Markus Armbruster]

Michael Roth <mdroth@linux.vnet.ibm.com> writes:

> Currently objects specified on the command-line are only partially
> cleaned up when 'object_del' is issued in either HMP or QMP: the
> object itself is fully finalized, but the QemuOpts are not removed.
> This results in the following behavior:
>
>   x86_64-softmmu/qemu-system-x86_64 -monitor stdio \
>     -object memory-backend-ram,id=ram1,size=256M
>
>   QEMU 2.7.91 monitor - type 'help' for more information
>   (qemu) object_del ram1
>   (qemu) object_del ram1
>   object 'ram1' not found
>   (qemu) object_add memory-backend-ram,id=ram1,size=256M
>   Duplicate ID 'ram1' for object
>   Try "help object_add" for more information
>
> which can be an issue for use-cases like memory hotplug.
>
> This happens on the HMP side because hmp_object_add() attempts to
> create a temporary QemuOpts entry with ID 'ram1', which ends up
> conflicting with the command-line-created entry, since it was never
> cleaned up during the previous hmp_object_del() call.
>
> We address this by adding a check in user_creatable_del(), which
> is called by both qmp_object_del() and hmp_object_del() to handle
> the actual object cleanup, to determine whether an option group entry
> matching the object's ID is present and removing it if it is.
>
> Note that qmp_object_add() never attempts to create a temporary
> QemuOpts entry, so it does not encounter the duplicate ID error,
> which is why this isn't generally visible in libvirt.
>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Cc: Markus Armbruster <armbru@redhat.com>
> Cc: Eric Blake <eblake@redhat.com>
> Cc: Daniel Berrange <berrange@redhat.com>
> Cc: qemu-stable@nongnu.org
> Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
> Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
> ---
>  qom/object_interfaces.c    | 7 +++++++
>  tests/check-qom-proplist.c | 4 ++--
>  2 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/qom/object_interfaces.c b/qom/object_interfaces.c
> index ded4d84..b462a74 100644
> --- a/qom/object_interfaces.c
> +++ b/qom/object_interfaces.c
> @@ -5,6 +5,7 @@
>  #include "qapi-visit.h"
>  #include "qapi/qobject-output-visitor.h"
>  #include "qapi/opts-visitor.h"
> +#include "qemu/config-file.h"
>  
>  void user_creatable_complete(Object *obj, Error **errp)
>  {
> @@ -209,6 +210,12 @@ void user_creatable_del(const char *id, Error **errp)
>          error_setg(errp, "object '%s' is in use, can not be deleted", id);
>          return;
>      }
> +
> +    /* if object was defined on the command-line, remove its corresponding
> +     * option group entry
> +     */

Consider adding wings to both ends of multi-line comments.

> +    qemu_opts_del(qemu_opts_find(qemu_find_opts_err("object", NULL), id));

qemu_find_opts_err() can't fail.  If it did, qemu_opts_find() would
crash.  Okay.

Elsewhere, we use qemu_opts_find(qemu_find_opts(NAME)), which would
report an error to stderr / monitor before crashing.  Also okay.

Perhaps the cleanest (but not the tersest) way to skin this cat:
qemu_opts_find(qemu_find_opts_err(&error_abort)).

Your choice.

> +
>      object_unparent(obj);
>  }
>  
> diff --git a/tests/check-qom-proplist.c b/tests/check-qom-proplist.c
> index e3f56ca..c5d4b60 100644
> --- a/tests/check-qom-proplist.c
> +++ b/tests/check-qom-proplist.c
> @@ -437,9 +437,9 @@ static void test_dummy_createcmdl(void)
>       * check for this in user_creatable_del() and remove the QemuOpts if
>       * it is present.
>       *
> -     * FIXME: add an assert to verify that the QemuOpts is cleaned up
> -     * once the corresponding cleanup code is added.
> +     * The below check ensures this works as expected.
>       */
> +    g_assert(qemu_opts_find(&qemu_object_opts, "dev0") == NULL);

There's also g_assert_null(), which could result in a more helpful
message when the assertion fails.  Your choice.

>  }
>  
>  static void test_dummy_badenum(void)

Re: [Qemu-devel] [PATCH v3 0/2] Fixes/tests for hmp_object_del()

[Markus Armbruster]

Michael Roth <mdroth@linux.vnet.ibm.com> writes:

> hmp_object_del() followed by a subsequent hmp_object_add() can trigger a
> duplicate ID error if the previous object shared the same ID and was added
> via the command-line. Please see patch 2/2 for more details.
>
> This patchset fixes the issue in question and adds some general unit tests
> for object created via -object, which we later extend to verify the fix in
> question.

Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [Qemu-devel] [PATCH v3 2/2] monitor: fix object_del for command-line-created objects

[Michael Roth]

Quoting Markus Armbruster (2016-12-12 09:53:20)
> Michael Roth <mdroth@linux.vnet.ibm.com> writes:
> 
> > Currently objects specified on the command-line are only partially
> > cleaned up when 'object_del' is issued in either HMP or QMP: the
> > object itself is fully finalized, but the QemuOpts are not removed.
> > This results in the following behavior:
> >
> >   x86_64-softmmu/qemu-system-x86_64 -monitor stdio \
> >     -object memory-backend-ram,id=ram1,size=256M
> >
> >   QEMU 2.7.91 monitor - type 'help' for more information
> >   (qemu) object_del ram1
> >   (qemu) object_del ram1
> >   object 'ram1' not found
> >   (qemu) object_add memory-backend-ram,id=ram1,size=256M
> >   Duplicate ID 'ram1' for object
> >   Try "help object_add" for more information
> >
> > which can be an issue for use-cases like memory hotplug.
> >
> > This happens on the HMP side because hmp_object_add() attempts to
> > create a temporary QemuOpts entry with ID 'ram1', which ends up
> > conflicting with the command-line-created entry, since it was never
> > cleaned up during the previous hmp_object_del() call.
> >
> > We address this by adding a check in user_creatable_del(), which
> > is called by both qmp_object_del() and hmp_object_del() to handle
> > the actual object cleanup, to determine whether an option group entry
> > matching the object's ID is present and removing it if it is.
> >
> > Note that qmp_object_add() never attempts to create a temporary
> > QemuOpts entry, so it does not encounter the duplicate ID error,
> > which is why this isn't generally visible in libvirt.
> >
> > Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > Cc: Markus Armbruster <armbru@redhat.com>
> > Cc: Eric Blake <eblake@redhat.com>
> > Cc: Daniel Berrange <berrange@redhat.com>
> > Cc: qemu-stable@nongnu.org
> > Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
> > Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
> > ---
> >  qom/object_interfaces.c    | 7 +++++++
> >  tests/check-qom-proplist.c | 4 ++--
> >  2 files changed, 9 insertions(+), 2 deletions(-)
> >
> > diff --git a/qom/object_interfaces.c b/qom/object_interfaces.c
> > index ded4d84..b462a74 100644
> > --- a/qom/object_interfaces.c
> > +++ b/qom/object_interfaces.c
> > @@ -5,6 +5,7 @@
> >  #include "qapi-visit.h"
> >  #include "qapi/qobject-output-visitor.h"
> >  #include "qapi/opts-visitor.h"
> > +#include "qemu/config-file.h"
> >  
> >  void user_creatable_complete(Object *obj, Error **errp)
> >  {
> > @@ -209,6 +210,12 @@ void user_creatable_del(const char *id, Error **errp)
> >          error_setg(errp, "object '%s' is in use, can not be deleted", id);
> >          return;
> >      }
> > +
> > +    /* if object was defined on the command-line, remove its corresponding
> > +     * option group entry
> > +     */
> 
> Consider adding wings to both ends of multi-line comments.
> 
> > +    qemu_opts_del(qemu_opts_find(qemu_find_opts_err("object", NULL), id));
> 
> qemu_find_opts_err() can't fail.  If it did, qemu_opts_find() would
> crash.  Okay.
> 
> Elsewhere, we use qemu_opts_find(qemu_find_opts(NAME)), which would
> report an error to stderr / monitor before crashing.  Also okay.
> 
> Perhaps the cleanest (but not the tersest) way to skin this cat:
> qemu_opts_find(qemu_find_opts_err(&error_abort)).
> 
> Your choice.
> 
> > +
> >      object_unparent(obj);
> >  }
> >  
> > diff --git a/tests/check-qom-proplist.c b/tests/check-qom-proplist.c
> > index e3f56ca..c5d4b60 100644
> > --- a/tests/check-qom-proplist.c
> > +++ b/tests/check-qom-proplist.c
> > @@ -437,9 +437,9 @@ static void test_dummy_createcmdl(void)
> >       * check for this in user_creatable_del() and remove the QemuOpts if
> >       * it is present.
> >       *
> > -     * FIXME: add an assert to verify that the QemuOpts is cleaned up
> > -     * once the corresponding cleanup code is added.
> > +     * The below check ensures this works as expected.
> >       */
> > +    g_assert(qemu_opts_find(&qemu_object_opts, "dev0") == NULL);
> 
> There's also g_assert_null(), which could result in a more helpful
> message when the assertion fails.  Your choice.

Thanks for the suggestions; I've incorporated all of them into v4.

> 
> >  }
> >  
> >  static void test_dummy_badenum(void)
>