[PULL 0/4] NBD patches for 2019-09-24

[PULL 0/4] NBD patches for 2019-09-24

[Eric Blake]

The following changes since commit 2f93a3ecdd3bb060bd04f698ccafe66efd98563a:

  Merge remote-tracking branch 'remotes/davidhildenbrand/tags/s390x-tcg-2019-09-23' into staging (2019-09-23 15:44:52 +0100)

are available in the Git repository at:

  https://repo.or.cz/qemu/ericb.git tags/pull-nbd-2019-09-24

for you to fetch changes up to 506902c6fa80210b002e30ff33794bfc718b15c6:

  tests: Use iothreads during iotest 223 (2019-09-24 07:30:19 -0500)

----------------------------------------------------------------
nbd patches for 2019-09-05

- Improved error message for plaintext client of encrypted server
- Fix various assertions when -object iothread is in use

----------------------------------------------------------------
Eric Blake (3):
      nbd/client: Add hint when TLS is missing
      nbd: Grab aio context lock in more places
      tests: Use iothreads during iotest 223

Sergio Lopez (1):
      nbd/server: attach client channel to the export's AioContext

 include/block/nbd.h        |  1 +
 blockdev-nbd.c             | 14 ++++++++++++--
 nbd/client.c               |  1 +
 nbd/server.c               | 27 +++++++++++++++++++++++----
 tests/qemu-iotests/223     |  6 ++++--
 tests/qemu-iotests/223.out |  1 +
 tests/qemu-iotests/233.out |  2 ++
 7 files changed, 44 insertions(+), 8 deletions(-)

-- 
2.21.0

[PULL 1/4] nbd/client: Add hint when TLS is missing

[Eric Blake]

I received an off-list report of failure to connect to an NBD server
expecting an x509 certificate, when the client was attempting something
similar to this command line:

$ ./x86_64-softmmu/qemu-system-x86_64 -name 'blah' -machine q35 -nodefaults \
  -object tls-creds-x509,id=tls0,endpoint=client,dir=$path_to_certs \
  -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x6 \
  -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=nbd:localhost:9000,werror=stop,rerror=stop,tls-creds=tls0 \
  -device scsi-hd,id=image1,drive=drive_image1,bootindex=0
qemu-system-x86_64: -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=nbd:localhost:9000,werror=stop,rerror=stop,tls-creds=tls0: TLS negotiation required before option 7 (go)
server reported: Option 0x7 not permitted before TLS

The problem?  As specified, -drive is trying to pass tls-creds to the
raw format driver instead of the nbd protocol driver, but before we
get to the point where we can detect that raw doesn't know what to do
with tls-creds, the nbd driver has already failed because the server
complained.  The fix to the broken command line?  Pass
'...,file.tls-creds=tls0' to ensure the tls-creds option is handed to
nbd, not raw.  But since the error message was rather cryptic, I'm
trying to improve the error message.

With this patch, the error message adds a line:

qemu-system-x86_64: -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=nbd:localhost:9000,werror=stop,rerror=stop,tls-creds=tls0: TLS negotiation required before option 7 (go)
Did you forget a valid tls-creds?
server reported: Option 0x7 not permitted before TLS

And with luck, someone grepping for that error message will find this
commit message and figure out their command line mistake.  Sadly, the
only mention of file.tls-creds in our docs relates to an --image-opts
use of PSK encryption with qemu-img as the client, rather than x509
certificate encryption with qemu-kvm as the client.

CC: Tingting Mao <timao@redhat.com>
CC: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190907172055.26870-1-eblake@redhat.com>
[eblake: squash in iotest 233 fix]
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 nbd/client.c               | 1 +
 tests/qemu-iotests/233.out | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/nbd/client.c b/nbd/client.c
index b9dc829175f9..f6733962b49b 100644
--- a/nbd/client.c
+++ b/nbd/client.c
@@ -204,6 +204,7 @@ static int nbd_handle_reply_err(QIOChannel *ioc, NBDOptionReply *reply,
     case NBD_REP_ERR_TLS_REQD:
         error_setg(errp, "TLS negotiation required before option %" PRIu32
                    " (%s)", reply->option, nbd_opt_lookup(reply->option));
+        error_append_hint(errp, "Did you forget a valid tls-creds?\n");
         break;

     case NBD_REP_ERR_UNKNOWN:
diff --git a/tests/qemu-iotests/233.out b/tests/qemu-iotests/233.out
index 24321efa113b..c3c344811b2b 100644
--- a/tests/qemu-iotests/233.out
+++ b/tests/qemu-iotests/233.out
@@ -21,8 +21,10 @@ server reported: TLS not configured

 == check plain client to TLS server fails ==
 qemu-img: Could not open 'nbd://localhost:PORT': TLS negotiation required before option 7 (go)
+Did you forget a valid tls-creds?
 server reported: Option 0x7 not permitted before TLS
 qemu-nbd: TLS negotiation required before option 3 (list)
+Did you forget a valid tls-creds?
 server reported: Option 0x3 not permitted before TLS

 == check TLS works ==
-- 
2.21.0

[PULL 2/4] nbd/server: attach client channel to the export's AioContext

[Eric Blake]

From: Sergio Lopez <slp@redhat.com>

On creation, the export's AioContext is set to the same one as the
BlockBackend, while the AioContext in the client QIOChannel is left
untouched.

As a result, when using data-plane, nbd_client_receive_next_request()
schedules coroutines in the IOThread AioContext, while the client's
QIOChannel is serviced from the main_loop, potentially triggering the
assertion at qio_channel_restart_[read|write].

To fix this, as soon we have the export corresponding to the client,
we call qio_channel_attach_aio_context() to attach the QIOChannel
context to the export's AioContext. This matches with the logic at
blk_aio_attached().

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1748253
Signed-off-by: Sergio Lopez <slp@redhat.com>
Message-Id: <20190912110032.26395-1-slp@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 nbd/server.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/nbd/server.c b/nbd/server.c
index 28c3c8be854c..31d624e146cb 100644
--- a/nbd/server.c
+++ b/nbd/server.c
@@ -1297,6 +1297,11 @@ static coroutine_fn int nbd_negotiate(NBDClient *client, Error **errp)
         return ret;
     }

+    /* Attach the channel to the same AioContext as the export */
+    if (client->exp && client->exp->ctx) {
+        qio_channel_attach_aio_context(client->ioc, client->exp->ctx);
+    }
+
     assert(!client->optlen);
     trace_nbd_negotiate_success();

-- 
2.21.0

[PULL 3/4] nbd: Grab aio context lock in more places

[Eric Blake]

When iothreads are in use, the failure to grab the aio context results
in an assertion failure when trying to unlock things during blk_unref,
when trying to unlock a mutex that was not locked.  In short, all
calls to nbd_export_put need to done while within the correct aio
context.  But since nbd_export_put can recursively reach itself via
nbd_export_close, and recursively grabbing the context would deadlock,
we can't do the context grab directly in those functions, but must do
so in their callers.

Hoist the use of the correct aio_context from nbd_export_new() to its
caller qmp_nbd_server_add().  Then tweak qmp_nbd_server_remove(),
nbd_eject_notifier(), and nbd_esport_close_all() to grab the right
context, so that all callers during qemu now own the context before
nbd_export_put() can call blk_unref().

Remaining uses in qemu-nbd don't matter (since that use case does not
support iothreads).

Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190917023917.32226-1-eblake@redhat.com>
Reviewed-by: Sergio Lopez <slp@redhat.com>
---
 include/block/nbd.h |  1 +
 blockdev-nbd.c      | 14 ++++++++++++--
 nbd/server.c        | 22 ++++++++++++++++++----
 3 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/include/block/nbd.h b/include/block/nbd.h
index 21550747cf35..316fd705a9e4 100644
--- a/include/block/nbd.h
+++ b/include/block/nbd.h
@@ -340,6 +340,7 @@ void nbd_export_put(NBDExport *exp);

 BlockBackend *nbd_export_get_blockdev(NBDExport *exp);

+AioContext *nbd_export_aio_context(NBDExport *exp);
 NBDExport *nbd_export_find(const char *name);
 void nbd_export_close_all(void);

diff --git a/blockdev-nbd.c b/blockdev-nbd.c
index 213f226ac1c4..6a8b206e1d74 100644
--- a/blockdev-nbd.c
+++ b/blockdev-nbd.c
@@ -151,6 +151,7 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name,
     BlockBackend *on_eject_blk;
     NBDExport *exp;
     int64_t len;
+    AioContext *aio_context;

     if (!nbd_server) {
         error_setg(errp, "NBD server not running");
@@ -173,11 +174,13 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name,
         return;
     }

+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
     len = bdrv_getlength(bs);
     if (len < 0) {
         error_setg_errno(errp, -len,
                          "Failed to determine the NBD export's length");
-        return;
+        goto out;
     }

     if (!has_writable) {
@@ -190,13 +193,16 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name,
     exp = nbd_export_new(bs, 0, len, name, NULL, bitmap, !writable, !writable,
                          NULL, false, on_eject_blk, errp);
     if (!exp) {
-        return;
+        goto out;
     }

     /* The list of named exports has a strong reference to this export now and
      * our only way of accessing it is through nbd_export_find(), so we can drop
      * the strong reference that is @exp. */
     nbd_export_put(exp);
+
+ out:
+    aio_context_release(aio_context);
 }

 void qmp_nbd_server_remove(const char *name,
@@ -204,6 +210,7 @@ void qmp_nbd_server_remove(const char *name,
                            Error **errp)
 {
     NBDExport *exp;
+    AioContext *aio_context;

     if (!nbd_server) {
         error_setg(errp, "NBD server not running");
@@ -220,7 +227,10 @@ void qmp_nbd_server_remove(const char *name,
         mode = NBD_SERVER_REMOVE_MODE_SAFE;
     }

+    aio_context = nbd_export_aio_context(exp);
+    aio_context_acquire(aio_context);
     nbd_export_remove(exp, mode, errp);
+    aio_context_release(aio_context);
 }

 void qmp_nbd_server_stop(Error **errp)
diff --git a/nbd/server.c b/nbd/server.c
index 31d624e146cb..d8d1e6245532 100644
--- a/nbd/server.c
+++ b/nbd/server.c
@@ -1461,7 +1461,12 @@ static void blk_aio_detach(void *opaque)
 static void nbd_eject_notifier(Notifier *n, void *data)
 {
     NBDExport *exp = container_of(n, NBDExport, eject_notifier);
+    AioContext *aio_context;
+
+    aio_context = exp->ctx;
+    aio_context_acquire(aio_context);
     nbd_export_close(exp);
+    aio_context_release(aio_context);
 }

 NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,
@@ -1480,12 +1485,11 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,
      * NBD exports are used for non-shared storage migration.  Make sure
      * that BDRV_O_INACTIVE is cleared and the image is ready for write
      * access since the export could be available before migration handover.
+     * ctx was acquired in the caller.
      */
     assert(name);
     ctx = bdrv_get_aio_context(bs);
-    aio_context_acquire(ctx);
     bdrv_invalidate_cache(bs, NULL);
-    aio_context_release(ctx);

     /* Don't allow resize while the NBD server is running, otherwise we don't
      * care what happens with the node. */
@@ -1493,7 +1497,7 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,
     if (!readonly) {
         perm |= BLK_PERM_WRITE;
     }
-    blk = blk_new(bdrv_get_aio_context(bs), perm,
+    blk = blk_new(ctx, perm,
                   BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED |
                   BLK_PERM_WRITE | BLK_PERM_GRAPH_MOD);
     ret = blk_insert_bs(blk, bs, errp);
@@ -1560,7 +1564,7 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,
     }

     exp->close = close;
-    exp->ctx = blk_get_aio_context(blk);
+    exp->ctx = ctx;
     blk_add_aio_context_notifier(blk, blk_aio_attached, blk_aio_detach, exp);

     if (on_eject_blk) {
@@ -1593,6 +1597,12 @@ NBDExport *nbd_export_find(const char *name)
     return NULL;
 }

+AioContext *
+nbd_export_aio_context(NBDExport *exp)
+{
+    return exp->ctx;
+}
+
 void nbd_export_close(NBDExport *exp)
 {
     NBDClient *client, *next;
@@ -1687,9 +1697,13 @@ BlockBackend *nbd_export_get_blockdev(NBDExport *exp)
 void nbd_export_close_all(void)
 {
     NBDExport *exp, *next;
+    AioContext *aio_context;

     QTAILQ_FOREACH_SAFE(exp, &exports, next, next) {
+        aio_context = exp->ctx;
+        aio_context_acquire(aio_context);
         nbd_export_close(exp);
+        aio_context_release(aio_context);
     }
 }

-- 
2.21.0

[PULL 4/4] tests: Use iothreads during iotest 223

[Eric Blake]

Doing so catches the bugs we just fixed with NBD not properly using
correct contexts.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190920220729.31801-1-eblake@redhat.com>
---
 tests/qemu-iotests/223     | 6 ++++--
 tests/qemu-iotests/223.out | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/223 b/tests/qemu-iotests/223
index cc48e78ea7dc..2ba3d8124b4f 100755
--- a/tests/qemu-iotests/223
+++ b/tests/qemu-iotests/223
@@ -2,7 +2,7 @@
 #
 # Test reading dirty bitmap over NBD
 #
-# Copyright (C) 2018 Red Hat, Inc.
+# Copyright (C) 2018-2019 Red Hat, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -109,7 +109,7 @@ echo
 echo "=== End dirty bitmaps, and start serving image over NBD ==="
 echo

-_launch_qemu 2> >(_filter_nbd)
+_launch_qemu -object iothread,id=io0 2> >(_filter_nbd)

 # Intentionally provoke some errors as well, to check error handling
 silent=
@@ -117,6 +117,8 @@ _send_qemu_cmd $QEMU_HANDLE '{"execute":"qmp_capabilities"}' "return"
 _send_qemu_cmd $QEMU_HANDLE '{"execute":"blockdev-add",
   "arguments":{"driver":"qcow2", "node-name":"n",
     "file":{"driver":"file", "filename":"'"$TEST_IMG"'"}}}' "return"
+_send_qemu_cmd $QEMU_HANDLE '{"execute":"x-blockdev-set-iothread",
+  "arguments":{"node-name":"n", "iothread":"io0"}}' "return"
 _send_qemu_cmd $QEMU_HANDLE '{"execute":"block-dirty-bitmap-disable",
   "arguments":{"node":"n", "name":"b"}}' "return"
 _send_qemu_cmd $QEMU_HANDLE '{"execute":"nbd-server-add",
diff --git a/tests/qemu-iotests/223.out b/tests/qemu-iotests/223.out
index 5d00398c11cb..23b34fcd202e 100644
--- a/tests/qemu-iotests/223.out
+++ b/tests/qemu-iotests/223.out
@@ -27,6 +27,7 @@ wrote 2097152/2097152 bytes at offset 2097152
 {"return": {}}
 {"return": {}}
 {"return": {}}
+{"return": {}}
 {"error": {"class": "GenericError", "desc": "NBD server not running"}}
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "NBD server already running"}}
-- 
2.21.0

Re: [PULL 4/4] tests: Use iothreads during iotest 223

[Nir Soffer]

[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]

On Tue, Sep 24, 2019 at 4:18 PM Eric Blake <eblake@redhat.com> wrote:

> Doing so catches the bugs we just fixed with NBD not properly using
> correct contexts.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Message-Id: <20190920220729.31801-1-eblake@redhat.com>
> ---
>  tests/qemu-iotests/223     | 6 ++++--
>  tests/qemu-iotests/223.out | 1 +
>  2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/tests/qemu-iotests/223 b/tests/qemu-iotests/223
> index cc48e78ea7dc..2ba3d8124b4f 100755
> --- a/tests/qemu-iotests/223
> +++ b/tests/qemu-iotests/223
> @@ -2,7 +2,7 @@
>  #
>  # Test reading dirty bitmap over NBD
>  #
> -# Copyright (C) 2018 Red Hat, Inc.
> +# Copyright (C) 2018-2019 Red Hat, Inc.
>  #
>  # This program is free software; you can redistribute it and/or modify
>  # it under the terms of the GNU General Public License as published by
> @@ -109,7 +109,7 @@ echo
>  echo "=== End dirty bitmaps, and start serving image over NBD ==="
>  echo
>
> -_launch_qemu 2> >(_filter_nbd)
> +_launch_qemu -object iothread,id=io0 2> >(_filter_nbd)
>

But now we will not catch bugs in flows that do not use iothreads.

I think it will be better to run this test twice, one with iothreads, one
without.

 # Intentionally provoke some errors as well, to check error handling
>  silent=
> @@ -117,6 +117,8 @@ _send_qemu_cmd $QEMU_HANDLE
> '{"execute":"qmp_capabilities"}' "return"
>  _send_qemu_cmd $QEMU_HANDLE '{"execute":"blockdev-add",
>    "arguments":{"driver":"qcow2", "node-name":"n",
>      "file":{"driver":"file", "filename":"'"$TEST_IMG"'"}}}' "return"
> +_send_qemu_cmd $QEMU_HANDLE '{"execute":"x-blockdev-set-iothread",
> +  "arguments":{"node-name":"n", "iothread":"io0"}}' "return"
>  _send_qemu_cmd $QEMU_HANDLE '{"execute":"block-dirty-bitmap-disable",
>    "arguments":{"node":"n", "name":"b"}}' "return"
>  _send_qemu_cmd $QEMU_HANDLE '{"execute":"nbd-server-add",
> diff --git a/tests/qemu-iotests/223.out b/tests/qemu-iotests/223.out
> index 5d00398c11cb..23b34fcd202e 100644
> --- a/tests/qemu-iotests/223.out
> +++ b/tests/qemu-iotests/223.out
> @@ -27,6 +27,7 @@ wrote 2097152/2097152 bytes at offset 2097152
>  {"return": {}}
>  {"return": {}}
>  {"return": {}}
> +{"return": {}}
>  {"error": {"class": "GenericError", "desc": "NBD server not running"}}
>  {"return": {}}
>  {"error": {"class": "GenericError", "desc": "NBD server already running"}}
> --
> 2.21.0
>
>
>

[-- Attachment #2: Type: text/html, Size: 4160 bytes --]

Re: [PULL 4/4] tests: Use iothreads during iotest 223

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 947 bytes --]

On 9/24/19 8:49 AM, Nir Soffer wrote:
> On Tue, Sep 24, 2019 at 4:18 PM Eric Blake <eblake@redhat.com> wrote:
> 
>> Doing so catches the bugs we just fixed with NBD not properly using
>> correct contexts.
>>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> Message-Id: <20190920220729.31801-1-eblake@redhat.com>
>> ---
>>  tests/qemu-iotests/223     | 6 ++++--
>>  tests/qemu-iotests/223.out | 1 +
>>  2 files changed, 5 insertions(+), 2 deletions(-)

>> -_launch_qemu 2> >(_filter_nbd)
>> +_launch_qemu -object iothread,id=io0 2> >(_filter_nbd)
>>
> 
> But now we will not catch bugs in flows that do not use iothreads.
> 
> I think it will be better to run this test twice, one with iothreads, one
> without.

Too late for this pull request, but I will propose that as a followup patch.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PULL 0/4] NBD patches for 2019-09-24

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 1747 bytes --]

On 9/24/19 7:39 AM, Eric Blake wrote:
> The following changes since commit 2f93a3ecdd3bb060bd04f698ccafe66efd98563a:
> 
>   Merge remote-tracking branch 'remotes/davidhildenbrand/tags/s390x-tcg-2019-09-23' into staging (2019-09-23 15:44:52 +0100)
> 
> are available in the Git repository at:
> 
>   https://repo.or.cz/qemu/ericb.git tags/pull-nbd-2019-09-24
> 
> for you to fetch changes up to 506902c6fa80210b002e30ff33794bfc718b15c6:
> 
>   tests: Use iothreads during iotest 223 (2019-09-24 07:30:19 -0500)
> 
> ----------------------------------------------------------------
> nbd patches for 2019-09-05
> 
> - Improved error message for plaintext client of encrypted server
> - Fix various assertions when -object iothread is in use
> 

As this has not been merged yet, I'll post a v2 series adding one more
patch fixing a Coverity error, since Vladimir pointed out I missed that.

> ----------------------------------------------------------------
> Eric Blake (3):
>       nbd/client: Add hint when TLS is missing
>       nbd: Grab aio context lock in more places
>       tests: Use iothreads during iotest 223
> 
> Sergio Lopez (1):
>       nbd/server: attach client channel to the export's AioContext
> 
>  include/block/nbd.h        |  1 +
>  blockdev-nbd.c             | 14 ++++++++++++--
>  nbd/client.c               |  1 +
>  nbd/server.c               | 27 +++++++++++++++++++++++----
>  tests/qemu-iotests/223     |  6 ++++--
>  tests/qemu-iotests/223.out |  1 +
>  tests/qemu-iotests/233.out |  2 ++
>  7 files changed, 44 insertions(+), 8 deletions(-)
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]