[Qemu-devel] [PATCH v3 0/4] block: Allow JSON filenames

[Qemu-devel] [PATCH v3 0/4] block: Allow JSON filenames

[Max Reitz]

This series acts as some kind of alternative or v6 to the "block/json:
Add JSON protocol driver" series. It makes bdrv_open() parse filenames
prefixed by "json:" as JSON objects (discarding the prefix beforehand)
and then use the resulting QDict as the options for the block device to
be opened with a NULL filename.

The purpose of this is that it may sometimes be desirable to specify
options for a block device where only a filename can be given, e.g., for
backing files. Using this should obviously be the exception, but it is
nice to have if actually needed.


After having written this series, I do indeed agree that it is much
nicer than the old version of having a dedicated "virtual" block driver
for this purpose.

Patches 1 and 2 are taken directly from the block/json v3, patch 4 has
been enriched by an additional scenario (options given both directly and
through the filename) and its comments have been adapted to the new
implementation.


v3:
 - Prioritize options given directly over those in the filename, as the
   use case for this series is being able to implicitly specify options
   e.g. for a backing file; a user should be able to override those,
   however [Kevin]
 - Adapted the test case in patch 4 accordingly



git-backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/4:[----] [--] 'qdict: Add qdict_join()'
002/4:[----] [--] 'check-qdict: Add test for qdict_join()'
003/4:[0010] [FC] 'block: Allow JSON filenames'
004/4:[0015] [FC] 'iotests: Add test for the JSON protocol'



Max Reitz (4):
  qdict: Add qdict_join()
  check-qdict: Add test for qdict_join()
  block: Allow JSON filenames
  iotests: Add test for the JSON protocol

 block.c                    |  41 ++++++++++++++
 include/qapi/qmp/qdict.h   |   3 ++
 qobject/qdict.c            |  32 +++++++++++
 tests/check-qdict.c        |  87 ++++++++++++++++++++++++++++++
 tests/qemu-iotests/089     | 130 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/089.out |  54 +++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 7 files changed, 348 insertions(+)
 create mode 100755 tests/qemu-iotests/089
 create mode 100644 tests/qemu-iotests/089.out

-- 
1.9.2

[Qemu-devel] [PATCH v3 1/4] qdict: Add qdict_join()

[Max Reitz]

This function joins two QDicts by absorbing one into the other.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 include/qapi/qmp/qdict.h |  3 +++
 qobject/qdict.c          | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/include/qapi/qmp/qdict.h b/include/qapi/qmp/qdict.h
index 1ddf97b..d68f4eb 100644
--- a/include/qapi/qmp/qdict.h
+++ b/include/qapi/qmp/qdict.h
@@ -16,6 +16,7 @@
 #include "qapi/qmp/qobject.h"
 #include "qapi/qmp/qlist.h"
 #include "qemu/queue.h"
+#include <stdbool.h>
 #include <stdint.h>
 
 #define QDICT_BUCKET_MAX 512
@@ -70,4 +71,6 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 
+void qdict_join(QDict *dest, QDict *src, bool overwrite);
+
 #endif /* QDICT_H */
diff --git a/qobject/qdict.c b/qobject/qdict.c
index 42ec4c0..ea239f0 100644
--- a/qobject/qdict.c
+++ b/qobject/qdict.c
@@ -665,3 +665,35 @@ void qdict_array_split(QDict *src, QList **dst)
         qlist_append_obj(*dst, subqobj ?: QOBJECT(subqdict));
     }
 }
+
+/**
+ * qdict_join(): Absorb the src QDict into the dest QDict, that is, move all
+ * elements from src to dest.
+ *
+ * If an element from src has a key already present in dest, it will not be
+ * moved unless overwrite is true.
+ *
+ * If overwrite is true, the conflicting values in dest will be discarded and
+ * replaced by the corresponding values from src.
+ *
+ * Therefore, with overwrite being true, the src QDict will always be empty when
+ * this function returns. If overwrite is false, the src QDict will be empty
+ * iff there were no conflicts.
+ */
+void qdict_join(QDict *dest, QDict *src, bool overwrite)
+{
+    const QDictEntry *entry, *next;
+
+    entry = qdict_first(src);
+    while (entry) {
+        next = qdict_next(src, entry);
+
+        if (overwrite || !qdict_haskey(dest, entry->key)) {
+            qobject_incref(entry->value);
+            qdict_put_obj(dest, entry->key, entry->value);
+            qdict_del(src, entry->key);
+        }
+
+        entry = next;
+    }
+}
-- 
1.9.2

[Qemu-devel] [PATCH v3 2/4] check-qdict: Add test for qdict_join()

[Max Reitz]

Add some test cases for qdict_join().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
---
 tests/check-qdict.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

diff --git a/tests/check-qdict.c b/tests/check-qdict.c
index 2ad0f78..a9296f0 100644
--- a/tests/check-qdict.c
+++ b/tests/check-qdict.c
@@ -444,6 +444,92 @@ static void qdict_array_split_test(void)
     QDECREF(test_dict);
 }
 
+static void qdict_join_test(void)
+{
+    QDict *dict1, *dict2;
+    bool overwrite = false;
+    int i;
+
+    dict1 = qdict_new();
+    dict2 = qdict_new();
+
+
+    /* Test everything once without overwrite and once with */
+    do
+    {
+        /* Test empty dicts */
+        qdict_join(dict1, dict2, overwrite);
+
+        g_assert(qdict_size(dict1) == 0);
+        g_assert(qdict_size(dict2) == 0);
+
+
+        /* First iteration: Test movement */
+        /* Second iteration: Test empty source and non-empty destination */
+        qdict_put(dict2, "foo", qint_from_int(42));
+
+        for (i = 0; i < 2; i++) {
+            qdict_join(dict1, dict2, overwrite);
+
+            g_assert(qdict_size(dict1) == 1);
+            g_assert(qdict_size(dict2) == 0);
+
+            g_assert(qdict_get_int(dict1, "foo") == 42);
+        }
+
+
+        /* Test non-empty source and destination without conflict */
+        qdict_put(dict2, "bar", qint_from_int(23));
+
+        qdict_join(dict1, dict2, overwrite);
+
+        g_assert(qdict_size(dict1) == 2);
+        g_assert(qdict_size(dict2) == 0);
+
+        g_assert(qdict_get_int(dict1, "foo") == 42);
+        g_assert(qdict_get_int(dict1, "bar") == 23);
+
+
+        /* Test conflict */
+        qdict_put(dict2, "foo", qint_from_int(84));
+
+        qdict_join(dict1, dict2, overwrite);
+
+        g_assert(qdict_size(dict1) == 2);
+        g_assert(qdict_size(dict2) == !overwrite);
+
+        g_assert(qdict_get_int(dict1, "foo") == overwrite ? 84 : 42);
+        g_assert(qdict_get_int(dict1, "bar") == 23);
+
+        if (!overwrite) {
+            g_assert(qdict_get_int(dict2, "foo") == 84);
+        }
+
+
+        /* Check the references */
+        g_assert(qdict_get(dict1, "foo")->refcnt == 1);
+        g_assert(qdict_get(dict1, "bar")->refcnt == 1);
+
+        if (!overwrite) {
+            g_assert(qdict_get(dict2, "foo")->refcnt == 1);
+        }
+
+
+        /* Clean up */
+        qdict_del(dict1, "foo");
+        qdict_del(dict1, "bar");
+
+        if (!overwrite) {
+            qdict_del(dict2, "foo");
+        }
+    }
+    while (overwrite ^= true);
+
+
+    QDECREF(dict1);
+    QDECREF(dict2);
+}
+
 /*
  * Errors test-cases
  */
@@ -584,6 +670,7 @@ int main(int argc, char **argv)
     g_test_add_func("/public/iterapi", qdict_iterapi_test);
     g_test_add_func("/public/flatten", qdict_flatten_test);
     g_test_add_func("/public/array_split", qdict_array_split_test);
+    g_test_add_func("/public/join", qdict_join_test);
 
     g_test_add_func("/errors/put_exists", qdict_put_exists_test);
     g_test_add_func("/errors/get_not_exists", qdict_get_not_exists_test);
-- 
1.9.2

[Qemu-devel] [PATCH v3 3/4] block: Allow JSON filenames

[Max Reitz]

If the filename given to bdrv_open() is prefixed with "json:", parse the
rest as a JSON object and merge the result into the options QDict. If
there are conflicts, report one of them to the user and abort.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/block.c b/block.c
index b749d31..6ac0f84 100644
--- a/block.c
+++ b/block.c
@@ -1275,6 +1275,33 @@ out:
     g_free(tmp_filename);
 }
 
+static QDict *parse_json_filename(const char *filename, Error **errp)
+{
+    QObject *options_obj;
+    QDict *options;
+    int ret;
+
+    ret = strstart(filename, "json:", &filename);
+    assert(ret);
+
+    options_obj = qobject_from_json(filename);
+    if (!options_obj) {
+        error_setg(errp, "Could not parse the JSON options");
+        return NULL;
+    }
+
+    if (qobject_type(options_obj) != QTYPE_QDICT) {
+        qobject_decref(options_obj);
+        error_setg(errp, "Invalid JSON object given");
+        return NULL;
+    }
+
+    options = qobject_to_qdict(options_obj);
+    qdict_flatten(options);
+
+    return options;
+}
+
 /*
  * Opens a disk image (raw, qcow2, vmdk, ...)
  *
@@ -1337,6 +1364,20 @@ int bdrv_open(BlockDriverState **pbs, const char *filename,
         options = qdict_new();
     }
 
+    if (filename && g_str_has_prefix(filename, "json:")) {
+        QDict *json_options = parse_json_filename(filename, &local_err);
+        if (local_err) {
+            ret = -EINVAL;
+            goto fail;
+        }
+
+        /* Options given in the filename have lower priority than options
+         * specified directly */
+        qdict_join(options, json_options, false);
+        QDECREF(json_options);
+        filename = NULL;
+    }
+
     bs->options = options;
     options = qdict_clone_shallow(options);
 
-- 
1.9.2

Re: [Qemu-devel] [PATCH v3 3/4] block: Allow JSON filenames

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

On 05/08/2014 12:12 PM, Max Reitz wrote:
> If the filename given to bdrv_open() is prefixed with "json:", parse the
> rest as a JSON object and merge the result into the options QDict. If
> there are conflicts, report one of them to the user and abort.

Commit message is stale compared to change in the code.

> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  block.c | 41 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 41 insertions(+)

Code is find, though, so:

Reviewed-by: Eric Blake <eblake@redhat.com>


> +
> +        /* Options given in the filename have lower priority than options
> +         * specified directly */
> +        qdict_join(options, json_options, false);
> +        QDECREF(json_options);
> +        filename = NULL;
> +    }
> +
>      bs->options = options;
>      options = qdict_clone_shallow(options);
>  
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH v3 3/4] block: Allow JSON filenames

[Kevin Wolf]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

Am 14.05.2014 um 00:12 hat Eric Blake geschrieben:
> On 05/08/2014 12:12 PM, Max Reitz wrote:
> > If the filename given to bdrv_open() is prefixed with "json:", parse the
> > rest as a JSON object and merge the result into the options QDict. If
> > there are conflicts, report one of them to the user and abort.
> 
> Commit message is stale compared to change in the code.

Changed the last sentence as follows:

"If there are conflicts, the options QDict takes precedence."

Kevin

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

[Qemu-devel] [PATCH v3 4/4] iotests: Add test for the JSON protocol

[Max Reitz]

Add a test for the JSON protocol driver.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/089     | 130 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/089.out |  54 +++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 185 insertions(+)
 create mode 100755 tests/qemu-iotests/089
 create mode 100644 tests/qemu-iotests/089.out

diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
new file mode 100755
index 0000000..ef47601
--- /dev/null
+++ b/tests/qemu-iotests/089
@@ -0,0 +1,130 @@
+#!/bin/bash
+#
+# Test case for support of JSON filenames
+#
+# Copyright (C) 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mreitz@redhat.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+here="$PWD"
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# Using an image filename containing quotation marks will render the JSON data
+# below invalid. In that case, we have little choice but simply not to run this
+# test.
+case $TEST_IMG in
+    *'"'*)
+        _notrun "image filename may not contain quotation marks"
+        ;;
+esac
+
+IMG_SIZE=64M
+
+# Taken from test 072
+echo
+echo "=== Testing nested image formats ==="
+echo
+
+TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
+
+$QEMU_IO -c 'write -P 42 0 512' -c 'write -P 23 512 512' \
+         -c 'write -P 66 1024 512' "$TEST_IMG.base" | _filter_qemu_io
+
+$QEMU_IMG convert -f raw -O $IMGFMT "$TEST_IMG.base" "$TEST_IMG"
+
+$QEMU_IO -c 'read -P 42 0 512' -c 'read -P 23 512 512' \
+         -c 'read -P 66 1024 512' "json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"$IMGFMT\",
+        \"file\": {
+            \"filename\": \"$TEST_IMG\"
+        }
+    }
+}" | _filter_qemu_io
+
+# This should fail (see test 072)
+$QEMU_IO -c 'read -P 42 0 512' "$TEST_IMG" | _filter_qemu_io
+
+
+# Taken from test 071
+echo
+echo "=== Testing blkdebug ==="
+echo
+
+_make_test_img $IMG_SIZE
+
+$QEMU_IO -c 'write -P 42 0x38000 512' "$TEST_IMG" | _filter_qemu_io
+
+# The "image.filename" part tests whether "a": { "b": "c" } and "a.b": "c" do
+# the same (which they should).
+$QEMU_IO -c 'read -P 42 0x38000 512' "json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"blkdebug\",
+        \"inject-error\": [{
+            \"event\": \"l2_load\"
+        }],
+        \"image.filename\": \"$TEST_IMG\"
+    }
+}" | _filter_qemu_io
+
+
+echo
+echo "=== Testing qemu-img info output ==="
+echo
+
+$QEMU_IMG info "json:{\"driver\":\"qcow2\",\"file.filename\":\"$TEST_IMG\"}" \
+    | _filter_testdir | _filter_imgfmt
+
+
+echo
+echo "=== Testing option merging ==="
+echo
+
+# Both options given directly and those given in the filename should be used
+$QEMU_IO -c "open -o driver=qcow2 json:{\"file.filename\":\"$TEST_IMG\"}" \
+         -c "info" 2>&1 | _filter_testdir | _filter_imgfmt
+
+# Options given directly should be prioritized over those given in the filename
+$QEMU_IO -c "open -o driver=qcow2 json:{\"driver\":\"raw\",\"file.filename\":\"$TEST_IMG\"}" \
+         -c "info" 2>&1 | _filter_testdir | _filter_imgfmt
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/089.out b/tests/qemu-iotests/089.out
new file mode 100644
index 0000000..d6cf783
--- /dev/null
+++ b/tests/qemu-iotests/089.out
@@ -0,0 +1,54 @@
+QA output created by 089
+
+=== Testing nested image formats ===
+
+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 
+wrote 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset 1024
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 1024
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Pattern verification failed at offset 0, 512 bytes
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+=== Testing blkdebug ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 
+wrote 512/512 bytes at offset 229376
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read failed: Input/output error
+
+=== Testing qemu-img info output ===
+
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+disk size: 324K
+cluster_size: 65536
+Format specific information:
+    compat: 1.1
+    lazy refcounts: false
+
+=== Testing option merging ===
+
+format name: IMGFMT
+cluster size: 64 KiB
+vm state offset: 512 MiB
+Format specific information:
+    compat: 1.1
+    lazy refcounts: false
+format name: IMGFMT
+cluster size: 64 KiB
+vm state offset: 512 MiB
+Format specific information:
+    compat: 1.1
+    lazy refcounts: false
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index ae09663..7f4b56b 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -95,4 +95,5 @@
 086 rw auto quick
 087 rw auto
 088 rw auto
+089 rw auto quick
 090 rw auto quick
-- 
1.9.2

Re: [Qemu-devel] [PATCH v3 4/4] iotests: Add test for the JSON protocol

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

On 05/08/2014 12:12 PM, Max Reitz wrote:
> Add a test for the JSON protocol driver.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/089     | 130 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/089.out |  54 +++++++++++++++++++
>  tests/qemu-iotests/group   |   1 +
>  3 files changed, 185 insertions(+)
>  create mode 100755 tests/qemu-iotests/089
>  create mode 100644 tests/qemu-iotests/089.out

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH v3 4/4] iotests: Add test for the JSON protocol

[Kevin Wolf]

Am 08.05.2014 um 20:12 hat Max Reitz geschrieben:
> Add a test for the JSON protocol driver.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>

This will need a follow-up fix. On one test machine, I get failures like
this (not sure why the size differs there):

--- 089.out     2014-05-19 01:28:00.715230617 -0400
+++ 089.out.bad 2014-05-19 02:43:07.417417728 -0400
@@ -31,7 +31,7 @@
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 64M (67108864 bytes)
-disk size: 324K
+disk size: 328K
 cluster_size: 65536
 Format specific information:
     compat: 1.1

Re: [Qemu-devel] [PATCH v3 0/4] block: Allow JSON filenames

[Kevin Wolf]

Am 08.05.2014 um 20:12 hat Max Reitz geschrieben:
> This series acts as some kind of alternative or v6 to the "block/json:
> Add JSON protocol driver" series. It makes bdrv_open() parse filenames
> prefixed by "json:" as JSON objects (discarding the prefix beforehand)
> and then use the resulting QDict as the options for the block device to
> be opened with a NULL filename.
> 
> The purpose of this is that it may sometimes be desirable to specify
> options for a block device where only a filename can be given, e.g., for
> backing files. Using this should obviously be the exception, but it is
> nice to have if actually needed.
> 
> 
> After having written this series, I do indeed agree that it is much
> nicer than the old version of having a dedicated "virtual" block driver
> for this purpose.
> 
> Patches 1 and 2 are taken directly from the block/json v3, patch 4 has
> been enriched by an additional scenario (options given both directly and
> through the filename) and its comments have been adapted to the new
> implementation.

Thanks, applied all to the block branch.

Kevin