From: Eric Blake <eblake@redhat.com>
To: qemu-devel@nongnu.org
Cc: kwolf@redhat.com, hreitz@redhat.com, berrange@redhat.com,
qemu-block@nongnu.org, den@virtuozzo.com,
andrey.drobyshev@virtuozzo.com, alexander.ivanov@virtuozzo.com,
vsementsov@yandex-team.ru
Subject: [PATCH v3 1/2] nbd: CVE-2024-7409: Close stray client sockets at server shutdown
Date: Mon, 5 Aug 2024 21:21:35 -0500 [thread overview]
Message-ID: <20240806022542.381883-5-eblake@redhat.com> (raw)
In-Reply-To: <20240806022542.381883-4-eblake@redhat.com>
A malicious client can attempt to connect to an NBD server, and then
intentionally delay progress in the handshake, including if it does
not know the TLS secrets. Although this behavior can be bounded by
the max-connections parameter, the QMP nbd-server-start currently
defaults to unlimited incoming client connections. Worse, if the
client waits to close the socket until after the QMP nbd-server-stop
command is executed, qemu will then SEGV when trying to dereference
the NULL nbd_server global whish is no longer present, which amounts
to a denial of service attack. If another NBD server is started
before the malicious client disconnects, I cannot rule out additional
adverse effects when the old client interferes with the connection
count of the new server.
For environments without this patch, the CVE can be mitigated by
ensuring (such as via a firewall) that only trusted clients can
connect to an NBD server. Note that using frameworks like libvirt
that ensure that TLS is used and that nbd-server-stop is not executed
while any trusted clients are still connected will only help if there
is also no possibility for an untrusted client to open a connection
but then stall on the NBD handshake.
This patch fixes the problem by tracking all client sockets opened
while the server is running, and forcefully closing any such sockets
remaining without a completed handshake at the time of
nbd-server-stop, then waiting until the coroutines servicing those
sockets notice the state change. nbd-server-stop may now take
slightly longer to execute, but the extra time is independent of
client response behaviors, and is generally no worse than the time
already taken by the blk_exp_close_all_type() that disconnects all
clients that completed handshakes (since that code also has an
AIO_WAIT_WHILE_UNLOCKED). For a long-running server with lots of
clients rapidly connecting and disconnecting, the memory used to track
all client sockets can result in some memory overhead, but it is not a
leak; the next patch will further optimize that by cleaning up memory
as clients go away. At any rate, this patch in isolation is
sufficient to fix the CVE.
This patch relies heavily on the fact that nbd/server.c guarantees
that it only calls nbd_blockdev_client_closed() from the main loop
(see the assertion in nbd_client_put() and the hoops used in
nbd_client_put_nonzero() to achieve that); if we did not have that
guarantee, we would also need a mutex protecting our accesses of the
list of connections to survive re-entrancy from independent iothreads.
Although I did not actually try to test old builds, it looks like this
problem has existed since at least commit 862172f45c (v2.12.0, 2017) -
even back in that patch to start using a QIONetListener to handle
listening on multiple sockets, nbd_server_free() was already unaware
that the nbd_blockdev_client_closed callback can be reached later by a
client thread that has not completed handshakes (and therefore the
client's socket never got added to the list closed in
nbd_export_close_all), despite that patch intentionally tearing down
the QIONetListener to prevent new clients.
Reported-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
Fixes: CVE-2024-7409
Signed-off-by: Eric Blake <eblake@redhat.com>
---
blockdev-nbd.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/blockdev-nbd.c b/blockdev-nbd.c
index 213012435f4..b8f00f402c6 100644
--- a/blockdev-nbd.c
+++ b/blockdev-nbd.c
@@ -21,12 +21,18 @@
#include "io/channel-socket.h"
#include "io/net-listener.h"
+typedef struct NBDConn {
+ QIOChannelSocket *cioc;
+ QSLIST_ENTRY(NBDConn) next;
+} NBDConn;
+
typedef struct NBDServerData {
QIONetListener *listener;
QCryptoTLSCreds *tlscreds;
char *tlsauthz;
uint32_t max_connections;
uint32_t connections;
+ QSLIST_HEAD(, NBDConn) conns;
} NBDServerData;
static NBDServerData *nbd_server;
@@ -51,6 +57,8 @@ int nbd_server_max_connections(void)
static void nbd_blockdev_client_closed(NBDClient *client, bool ignored)
{
+ assert(qemu_in_main_thread() && nbd_server);
+
nbd_client_put(client);
assert(nbd_server->connections > 0);
nbd_server->connections--;
@@ -60,7 +68,13 @@ static void nbd_blockdev_client_closed(NBDClient *client, bool ignored)
static void nbd_accept(QIONetListener *listener, QIOChannelSocket *cioc,
gpointer opaque)
{
+ NBDConn *conn = g_new0(NBDConn, 1);
+
+ assert(qemu_in_main_thread() && nbd_server);
nbd_server->connections++;
+ object_ref(OBJECT(cioc));
+ conn->cioc = cioc;
+ QSLIST_INSERT_HEAD(&nbd_server->conns, conn, next);
nbd_update_server_watch(nbd_server);
qio_channel_set_name(QIO_CHANNEL(cioc), "nbd-server");
@@ -83,8 +97,24 @@ static void nbd_server_free(NBDServerData *server)
return;
}
+ /*
+ * Forcefully close the listener socket, and any clients that have
+ * not yet disconnected on their own.
+ */
qio_net_listener_disconnect(server->listener);
object_unref(OBJECT(server->listener));
+ while (!QSLIST_EMPTY(&server->conns)) {
+ NBDConn *conn = QSLIST_FIRST(&server->conns);
+
+ qio_channel_shutdown(QIO_CHANNEL(conn->cioc), QIO_CHANNEL_SHUTDOWN_BOTH,
+ NULL);
+ object_unref(OBJECT(conn->cioc));
+ QSLIST_REMOVE_HEAD(&server->conns, next);
+ g_free(conn);
+ }
+
+ AIO_WAIT_WHILE_UNLOCKED(NULL, server->connections > 0);
+
if (server->tlscreds) {
object_unref(OBJECT(server->tlscreds));
}
--
2.45.2
next prev parent reply other threads:[~2024-08-06 2:26 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-06 2:21 [PATCH for-9.1 v3 0/2] NBD CVE-2024-7409 Eric Blake
2024-08-06 2:21 ` Eric Blake [this message]
2024-08-06 9:27 ` [PATCH v3 1/2] nbd: CVE-2024-7409: Close stray client sockets at server shutdown Daniel P. Berrangé
2024-08-06 2:21 ` [PATCH v3 2/2] nbd: Clean up clients more efficiently Eric Blake
2024-08-06 2:36 ` Eric Blake
2024-08-06 9:32 ` Daniel P. Berrangé
2024-08-06 12:58 ` Eric Blake
2024-08-06 13:56 ` Eric Blake
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240806022542.381883-5-eblake@redhat.com \
--to=eblake@redhat.com \
--cc=alexander.ivanov@virtuozzo.com \
--cc=andrey.drobyshev@virtuozzo.com \
--cc=berrange@redhat.com \
--cc=den@virtuozzo.com \
--cc=hreitz@redhat.com \
--cc=kwolf@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=vsementsov@yandex-team.ru \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).