From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
To: qemu-block@nongnu.org
Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, berrange@redhat.com,
qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org
Subject: [PATCH v2 3/5] block/nbd: on shutdown terminate connection attempt
Date: Mon, 27 Jul 2020 21:47:49 +0300 [thread overview]
Message-ID: <20200727184751.15704-4-vsementsov@virtuozzo.com> (raw)
In-Reply-To: <20200727184751.15704-1-vsementsov@virtuozzo.com>
On shutdown nbd driver may be in a connecting state. We should shutdown
it as well, otherwise we may hang in
nbd_teardown_connection, waiting for conneciton_co to finish in
BDRV_POLL_WHILE(bs, s->connection_co) loop if remote server is down.
How to reproduce the dead lock:
1. Create nbd-fault-injector.conf with the following contents:
[inject-error "mega1"]
event=data
io=readwrite
when=before
2. In one terminal run nbd-fault-injector in a loop, like this:
n=1; while true; do
echo $n; ((n++));
./nbd-fault-injector.py 127.0.0.1:10000 nbd-fault-injector.conf;
done
3. In another terminal run qemu-io in a loop, like this:
n=1; while true; do
echo $n; ((n++));
./qemu-io -c 'read 0 512' nbd://127.0.0.1:10000;
done
After some time, qemu-io will hang. Note, that this hang may be
triggered by another bug, so the whole case is fixed only together with
commit "block/nbd: allow drain during reconnect attempt".
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/nbd.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/block/nbd.c b/block/nbd.c
index 6d19f3c660..dfe1408b2d 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -209,11 +209,15 @@ static void nbd_teardown_connection(BlockDriverState *bs)
{
BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
- if (s->state == NBD_CLIENT_CONNECTED) {
+ if (s->ioc) {
/* finish any pending coroutines */
- assert(s->ioc);
qio_channel_shutdown(s->ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
+ } else if (s->sioc) {
+ /* abort negotiation */
+ qio_channel_shutdown(QIO_CHANNEL(s->sioc), QIO_CHANNEL_SHUTDOWN_BOTH,
+ NULL);
}
+
s->state = NBD_CLIENT_QUIT;
if (s->connection_co) {
if (s->connection_co_sleep_ns_state) {
@@ -1459,6 +1463,9 @@ static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *sioc,
int ret;
trace_nbd_client_handshake(s->export);
+
+ s->sioc = sioc;
+
qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL);
qio_channel_attach_aio_context(QIO_CHANNEL(sioc), aio_context);
@@ -1473,6 +1480,7 @@ static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *sioc,
g_free(s->info.name);
if (ret < 0) {
object_unref(OBJECT(sioc));
+ s->sioc = NULL;
return ret;
}
if (s->x_dirty_bitmap && !s->info.base_allocation) {
@@ -1498,8 +1506,6 @@ static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *sioc,
}
}
- s->sioc = sioc;
-
if (!s->ioc) {
s->ioc = QIO_CHANNEL(sioc);
object_ref(OBJECT(s->ioc));
@@ -1520,6 +1526,7 @@ static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *sioc,
nbd_send_request(s->ioc ?: QIO_CHANNEL(sioc), &request);
object_unref(OBJECT(sioc));
+ s->sioc = NULL;
return ret;
}
--
2.21.0
next prev parent reply other threads:[~2020-07-27 18:52 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-27 18:47 [PATCH v2 for-5.1? 0/5] Fix nbd reconnect dead-locks Vladimir Sementsov-Ogievskiy
2020-07-27 18:47 ` [PATCH v2 1/5] block/nbd: split nbd_establish_connection out of nbd_client_connect Vladimir Sementsov-Ogievskiy
2020-07-27 21:11 ` Eric Blake
2020-07-27 18:47 ` [PATCH v2 2/5] block/nbd: allow drain during reconnect attempt Vladimir Sementsov-Ogievskiy
2020-07-27 21:14 ` Eric Blake
2020-07-27 18:47 ` Vladimir Sementsov-Ogievskiy [this message]
2020-07-27 18:47 ` [PATCH v2 4/5] block/nbd: nbd_co_reconnect_loop(): don't sleep if drained Vladimir Sementsov-Ogievskiy
2020-07-27 18:47 ` [PATCH v2 5/5] block/nbd: use non-blocking connect: fix vm hang on connect() Vladimir Sementsov-Ogievskiy
2020-07-27 22:01 ` [PATCH v2 for-5.1? 0/5] Fix nbd reconnect dead-locks Eric Blake
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200727184751.15704-4-vsementsov@virtuozzo.com \
--to=vsementsov@virtuozzo.com \
--cc=berrange@redhat.com \
--cc=den@openvz.org \
--cc=kwolf@redhat.com \
--cc=mreitz@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).