[Qemu-devel] [PULL 0/4] QAPI patches for 2018-03-23, for 2.12-rc1

[Qemu-devel] [PULL 0/4] QAPI patches for 2018-03-23, for 2.12-rc1

[Eric Blake]

The following changes since commit 4c2c1015905fa1d616750dfe024b4c0b35875950:

  Merge remote-tracking branch 'remotes/borntraeger/tags/s390x-20180323' into staging (2018-03-23 10:20:54 +0000)

are available in the Git repository at:

  git://repo.or.cz/qemu/ericb.git tags/pull-qapi-2018-03-23

for you to fetch changes up to 39615354fc07af34e04ab5efb5b6d478b0d24e32:

  qapi: Force UTF8 encoding when parsing qapi files (2018-03-23 12:29:07 -0500)

There may be another rc1 pull request on Monday, but let's get
these in sooner rather than later to minimize the window where
iotests are broken during bisects.

----------------------------------------------------------------
qapi patches for 2018-03-12, 2.12-rc1

- Peter Xu: 0/4 Turn OOB off for 2.12-rc1, revert OOB tests
- Eric Blake: qapi: Force UTF8 encoding when parsing qapi files

----------------------------------------------------------------
Eric Blake (1):
      qapi: Force UTF8 encoding when parsing qapi files

Peter Xu (3):
      Revert "tests: qmp-test: add oob test"
      Revert "tests: qmp-test: verify command batching"
      Revert "monitor: enable IO thread for (qmp & !mux) typed"

 monitor.c              |  5 +--
 tests/qmp-test.c       | 97 +-------------------------------------------------
 tests/Makefile.include |  6 ++--
 3 files changed, 5 insertions(+), 103 deletions(-)

-- 
2.14.3

[Qemu-devel] [PULL 1/4] Revert "tests: qmp-test: add oob test"

[Eric Blake]

From: Peter Xu <peterx@redhat.com>

This reverts commit d003f7a8f9cafe50119975844fa01afc2baf41fb.

Enabling OOB caused several iotests failures; due to the imminent
2.12 release, the safest action is to disable OOB, but first we
have to revert tests that rely on OOB.

Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180323140821.28957-3-peterx@redhat.com>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
[eblake: reorder commits, enhance commit message]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qmp-test.c | 65 --------------------------------------------------------
 1 file changed, 65 deletions(-)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 07c0b87e27a..2e4b599a4c8 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -164,70 +164,6 @@ static void test_qmp_protocol(void)
     qtest_quit(qts);
 }

-/* Tests for Out-Of-Band support. */
-static void test_qmp_oob(void)
-{
-    QDict *resp;
-    int acks = 0;
-    const char *cmd_id;
-
-    global_qtest = qtest_init_without_qmp_handshake(common_args);
-
-    /* Ignore the greeting message. */
-    resp = qmp_receive();
-    g_assert(qdict_get_qdict(resp, "QMP"));
-    QDECREF(resp);
-
-    /* Try a fake capability, it should fail. */
-    resp = qmp("{ 'execute': 'qmp_capabilities', "
-               "  'arguments': { 'enable': [ 'cap-does-not-exist' ] } }");
-    g_assert(qdict_haskey(resp, "error"));
-    QDECREF(resp);
-
-    /* Now, enable OOB in current QMP session, it should succeed. */
-    resp = qmp("{ 'execute': 'qmp_capabilities', "
-               "  'arguments': { 'enable': [ 'oob' ] } }");
-    g_assert(qdict_haskey(resp, "return"));
-    QDECREF(resp);
-
-    /*
-     * Try any command that does not support OOB but with OOB flag. We
-     * should get failure.
-     */
-    resp = qmp("{ 'execute': 'query-cpus',"
-               "  'control': { 'run-oob': true } }");
-    g_assert(qdict_haskey(resp, "error"));
-    QDECREF(resp);
-
-    /*
-     * First send the "x-oob-test" command with lock=true and
-     * oob=false, it should hang the dispatcher and main thread;
-     * later, we send another lock=false with oob=true to continue
-     * that thread processing.  Finally we should receive replies from
-     * both commands.
-     */
-    qmp_async("{ 'execute': 'x-oob-test',"
-              "  'arguments': { 'lock': true }, "
-              "  'id': 'lock-cmd'}");
-    qmp_async("{ 'execute': 'x-oob-test', "
-              "  'arguments': { 'lock': false }, "
-              "  'control': { 'run-oob': true }, "
-              "  'id': 'unlock-cmd' }");
-
-    /* Ignore all events.  Wait for 2 acks */
-    while (acks < 2) {
-        resp = qmp_receive();
-        cmd_id = qdict_get_str(resp, "id");
-        if (!g_strcmp0(cmd_id, "lock-cmd") ||
-            !g_strcmp0(cmd_id, "unlock-cmd")) {
-            acks++;
-        }
-        QDECREF(resp);
-    }
-
-    qtest_end();
-}
-
 static int query_error_class(const char *cmd)
 {
     static struct {
@@ -412,7 +348,6 @@ int main(int argc, char *argv[])
     g_test_init(&argc, &argv, NULL);

     qtest_add_func("qmp/protocol", test_qmp_protocol);
-    qtest_add_func("qmp/oob", test_qmp_oob);
     qmp_schema_init(&schema);
     add_query_tests(&schema);

-- 
2.14.3

[Qemu-devel] [PULL 2/4] Revert "tests: qmp-test: verify command batching"

[Eric Blake]

From: Peter Xu <peterx@redhat.com>

This reverts commit 91ad45061af0fe44ac5dadb5bedaf4d7a08077c8.

Enabling OOB caused several iotests failures; due to the imminent
2.12 release, the safest action is to disable OOB, but first we
have to revert tests that rely on OOB.

Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180323140821.28957-4-peterx@redhat.com>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
[eblake: reorder commits, enhance commit message]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qmp-test.c | 22 ----------------------
 1 file changed, 22 deletions(-)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 2e4b599a4c8..d1fa1cb2177 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -82,7 +82,6 @@ static void test_qmp_protocol(void)
     QTestState *qts;
     const QListEntry *entry;
     QString *qstr;
-    int i;

     qts = qtest_init_without_qmp_handshake(common_args);

@@ -140,27 +139,6 @@ static void test_qmp_protocol(void)
     g_assert_cmpint(qdict_get_int(resp, "id"), ==, 2);
     QDECREF(resp);

-    /*
-     * Test command batching.  In current test OOB is not enabled, we
-     * should be able to run as many commands in batch as we like.
-     * Using 16 (>8, which is OOB queue length) to make sure OOB won't
-     * break existing clients.  Note: this test does not control the
-     * scheduling of QEMU's QMP command processing threads so it may
-     * not really trigger batching inside QEMU.  This is just a
-     * best-effort test.
-     */
-    for (i = 0; i < 16; i++) {
-        qtest_async_qmp(qts, "{ 'execute': 'query-version' }");
-    }
-    /* Verify the replies to make sure no command is dropped. */
-    for (i = 0; i < 16; i++) {
-        resp = qtest_qmp_receive(qts);
-        /* It should never be dropped.  Each of them should be a reply. */
-        g_assert(qdict_haskey(resp, "return"));
-        g_assert(!qdict_haskey(resp, "event"));
-        QDECREF(resp);
-    }
-
     qtest_quit(qts);
 }

-- 
2.14.3

[Qemu-devel] [PULL 3/4] Revert "monitor: enable IO thread for (qmp & !mux) typed"

[Eric Blake]

From: Peter Xu <peterx@redhat.com>

This reverts commit 3fd2457d18edf5736f713dfe1ada9c87a9badab1.

Enabling OOB caused several iotests failures; due to the imminent
2.12 release, the safest action is to disable OOB for now.  If
other patches fix the issues that iotests exposed, it may be turned
back on in time for the release, otherwise it will be 2.13 material;
either way, the framework changes not reverted now do not hurt if
they remain as part of the 2.12 release.

Additionally, revert the tests in the patch 02130314d8 ("qmp: introduce
QMPCapability", 2018-03-19), as both parts must be reverted at once
to keep 'make check' passing.

Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180323140821.28957-2-peterx@redhat.com>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
[eblake: reorder/squash commits, enhance commit message]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 monitor.c        |  5 +----
 tests/qmp-test.c | 10 +---------
 2 files changed, 2 insertions(+), 13 deletions(-)

diff --git a/monitor.c b/monitor.c
index 6ccd2fc0895..77f4c41cfa6 100644
--- a/monitor.c
+++ b/monitor.c
@@ -36,7 +36,6 @@
 #include "net/slirp.h"
 #include "chardev/char-fe.h"
 #include "chardev/char-io.h"
-#include "chardev/char-mux.h"
 #include "ui/qemu-spice.h"
 #include "sysemu/numa.h"
 #include "monitor/monitor.h"
@@ -4537,10 +4536,8 @@ static void monitor_qmp_setup_handlers_bh(void *opaque)
 void monitor_init(Chardev *chr, int flags)
 {
     Monitor *mon = g_malloc(sizeof(*mon));
-    /* Enable IOThread for QMPs that are not using MUX chardev backends. */
-    bool use_io_thr = (!CHARDEV_IS_MUX(chr)) && (flags & MONITOR_USE_CONTROL);

-    monitor_data_init(mon, false, use_io_thr);
+    monitor_data_init(mon, false, false);

     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index d1fa1cb2177..558e83540cc 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -80,8 +80,6 @@ static void test_qmp_protocol(void)
     QDict *resp, *q, *ret;
     QList *capabilities;
     QTestState *qts;
-    const QListEntry *entry;
-    QString *qstr;

     qts = qtest_init_without_qmp_handshake(common_args);

@@ -91,13 +89,7 @@ static void test_qmp_protocol(void)
     g_assert(q);
     test_version(qdict_get(q, "version"));
     capabilities = qdict_get_qlist(q, "capabilities");
-    g_assert(capabilities);
-    entry = qlist_first(capabilities);
-    g_assert(entry);
-    qstr = qobject_to(QString, entry->value);
-    g_assert(qstr);
-    g_assert_cmpstr(qstring_get_str(qstr), ==, "oob");
-    QDECREF(resp);
+    g_assert(capabilities && qlist_empty(capabilities));

     /* Test valid command before handshake */
     resp = qtest_qmp(qts, "{ 'execute': 'query-version' }");
-- 
2.14.3

[Qemu-devel] [PULL 4/4] qapi: Force UTF8 encoding when parsing qapi files

[Eric Blake]

Commit d4e5ec877 already fixed things to work around Python 3's
lame bug of having LC_ALL=C not be 8-bit clean, when parsing the
main QMP qapi files; but failed to do likewise in the tests
directory.  As a result, running 'LC_ALL=C make check' fails on
escape-too-big and unicode-str when using python 3 with a nasty
stack trace instead of the intended graceful error message that
QAPI doesn't yet support 8-bit data (the two tests contain
Unicode é, when parsed in UTF-8; they represent something
different when parsed in a proper single-byte C locale, but that
doesn't matter to the error message printed out, provided that
brain-dead Python hasn't first choked on the input instead of
being 8-bit clean).

Ideally, we'd teach the qapi generator scripts to automatically
slurp things in using UTF-8 regardless of locale, and to honor
content that is not limited to 7 bit data rather than gracefully
erroring out; but until then, since our graceful error depends
on python parsing 8-bit data (even if nothing we generate uses
8-bit data), our quick fix is to use the right locale when
running these tests.

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180319205040.1113423-1-eblake@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/Makefile.include | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 0b277036df1..eb218a9539d 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -669,13 +669,13 @@ tests/test-qapi-events.c tests/test-qapi-events.h \
 tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \
 tests/test-qapi-gen-timestamp ;
 tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+	$(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
 		-o tests -p "test-" $<, \
 		"GEN","$(@:%-timestamp=%)")
 	@>$@

 tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+	$(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
 		-o tests/qapi-schema -p "doc-good-" $<, \
 		"GEN","$@")
 	@mv tests/qapi-schema/doc-good-qapi-doc.texi $@
@@ -927,7 +927,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		$(PYTHON_UTF8) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
 		$^ >$*.test.out 2>$*.test.err; \
 		echo $$? >$*.test.exit, \
 		"TEST","$*.out")
-- 
2.14.3

Re: [Qemu-devel] [PULL 0/4] QAPI patches for 2018-03-23, for 2.12-rc1

[Peter Maydell]

On 23 March 2018 at 17:37, Eric Blake <eblake@redhat.com> wrote:
> The following changes since commit 4c2c1015905fa1d616750dfe024b4c0b35875950:
>
>   Merge remote-tracking branch 'remotes/borntraeger/tags/s390x-20180323' into staging (2018-03-23 10:20:54 +0000)
>
> are available in the Git repository at:
>
>   git://repo.or.cz/qemu/ericb.git tags/pull-qapi-2018-03-23
>
> for you to fetch changes up to 39615354fc07af34e04ab5efb5b6d478b0d24e32:
>
>   qapi: Force UTF8 encoding when parsing qapi files (2018-03-23 12:29:07 -0500)
>
> There may be another rc1 pull request on Monday, but let's get
> these in sooner rather than later to minimize the window where
> iotests are broken during bisects.
>
> ----------------------------------------------------------------
> qapi patches for 2018-03-12, 2.12-rc1
>
> - Peter Xu: 0/4 Turn OOB off for 2.12-rc1, revert OOB tests
> - Eric Blake: qapi: Force UTF8 encoding when parsing qapi files
>

Applied, thanks.

-- PMM