[PATCH v4 0/3] nbd: Increase unix socket buffer size

[PATCH v4 0/3] nbd: Increase unix socket buffer size

[Nir Soffer]

On both macOS and Linux, the default send buffer size is too small causing poor
performance when reading and writing to qemu-nbd. A simple way to experience
this is to compare TCP and unix sockets, showing that TCP socket is much
faster. Programs like nbdcopy partly mitigate this by using multiple NBD
connections.

On macOS the default send buffer size is 8192 bytes. Increasing the send buffer
size to 2 MiB shows up to *12.6 times higher throughput* and lower cpu usage.

On Linux the default and maximum buffer size is 212992 bytes. Increasing the
send buffer size to 2 MiB shows up to *2.7 times higher throughput* and lower
cpu usage. On older machine we see very little improvement, up to 1.03 times
higher throughput.

We likely have the same issue on other platforms. It should be easy to enable
this change for more platform by defining UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE.

Changes since v3:
- Modify only NBD sockets, since not all sockets need a larger buffer size, and
  the fixed memory overhead per socker may be a significant amount of extra
  memory overhead on a host. (Daniel)
- Add qio_channel_socket_set_send_buffer() helper to set buffer size from nbd
  code.
- Remove the check for SOCK_STREAM since NBD does not support SOCK_DGRAM.
- Warn if we fail to set the socket send buffer size.

v3 was here:
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html

Nir Soffer (3):
  io: Add helper for setting socket send buffer size
  nbd: Set unix socket send buffer on macOS
  nbd: Set unix socket send buffer on Linux

 include/io/channel-socket.h | 13 +++++++++++++
 io/channel-socket.c         | 11 +++++++++++
 nbd/client-connection.c     |  3 +++
 nbd/common.c                | 26 ++++++++++++++++++++++++++
 nbd/nbd-internal.h          |  5 +++++
 nbd/server.c                |  2 ++
 6 files changed, 60 insertions(+)

-- 
2.39.5 (Apple Git-154)

[PATCH v4 1/3] io: Add helper for setting socket send buffer size

[Nir Soffer]

Testings reading and writing from qemu-nbd using a unix domain socket
shows that the platform default send buffer size is too low, leading to
poor performance and hight cpu usage.

Add a helper for setting socket send buffer size to be used in NBD code.
It can also be used in other context.

We don't need a helper for receive buffer size since it is not used with
unix domain sockets. This is documented for Linux, and not documented
for macOS.

Failing to set the socket buffer size is not a fatal error, but the
caller may want to warn about the failure.

Signed-off-by: Nir Soffer <nirsof@gmail.com>
---
 include/io/channel-socket.h | 13 +++++++++++++
 io/channel-socket.c         | 11 +++++++++++
 2 files changed, 24 insertions(+)

diff --git a/include/io/channel-socket.h b/include/io/channel-socket.h
index ab15577d38..a88cf8b3a9 100644
--- a/include/io/channel-socket.h
+++ b/include/io/channel-socket.h
@@ -261,5 +261,18 @@ QIOChannelSocket *
 qio_channel_socket_accept(QIOChannelSocket *ioc,
                           Error **errp);
 
+/**
+ * qio_channel_socket_set_send_buffer:
+ * @ioc: the socket channel object
+ * @size: buffer size
+ * @errp: pointer to a NULL-initialized error object
+ *
+ * Set the underlying socket send buffer size.
+ *
+ * Retruns: 0 on success, or -1 on error.
+ */
+int qio_channel_socket_set_send_buffer(QIOChannelSocket *ioc,
+                                       size_t size,
+                                       Error **errp);
 
 #endif /* QIO_CHANNEL_SOCKET_H */
diff --git a/io/channel-socket.c b/io/channel-socket.c
index 608bcf066e..a776cd45f5 100644
--- a/io/channel-socket.c
+++ b/io/channel-socket.c
@@ -78,6 +78,17 @@ qio_channel_socket_new(void)
     return sioc;
 }
 
+int qio_channel_socket_set_send_buffer(QIOChannelSocket *ioc,
+                                       size_t size,
+                                       Error **errp)
+{
+    if (setsockopt(ioc->fd, SOL_SOCKET, SO_SNDBUF, &size, sizeof(size)) < 0) {
+        error_setg_errno(errp, errno, "Unable to set socket send buffer size");
+        return -1;
+    }
+
+    return 0;
+}
 
 static int
 qio_channel_socket_set_fd(QIOChannelSocket *sioc,
-- 
2.39.5 (Apple Git-154)

Re: [PATCH v4 1/3] io: Add helper for setting socket send buffer size

[Daniel P. Berrangé]

On Sat, May 17, 2025 at 11:11:52PM +0300, Nir Soffer wrote:
> Testings reading and writing from qemu-nbd using a unix domain socket
> shows that the platform default send buffer size is too low, leading to
> poor performance and hight cpu usage.
> 
> Add a helper for setting socket send buffer size to be used in NBD code.
> It can also be used in other context.
> 
> We don't need a helper for receive buffer size since it is not used with
> unix domain sockets. This is documented for Linux, and not documented
> for macOS.
> 
> Failing to set the socket buffer size is not a fatal error, but the
> caller may want to warn about the failure.
> 
> Signed-off-by: Nir Soffer <nirsof@gmail.com>
> ---
>  include/io/channel-socket.h | 13 +++++++++++++
>  io/channel-socket.c         | 11 +++++++++++
>  2 files changed, 24 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 1/3] io: Add helper for setting socket send buffer size

[Richard W.M. Jones]

On Sat, May 17, 2025 at 11:11:52PM +0300, Nir Soffer wrote:
> Testings reading and writing from qemu-nbd using a unix domain socket

Nit pick that this should be "Testing", not "Testings".  (Or
"Experiments" if you really wanted a plural noun.)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: [PATCH v4 1/3] io: Add helper for setting socket send buffer size

[Eric Blake]

On Mon, May 19, 2025 at 12:20:48PM +0100, Richard W.M. Jones wrote:
> On Sat, May 17, 2025 at 11:11:52PM +0300, Nir Soffer wrote:
> > Testings reading and writing from qemu-nbd using a unix domain socket
> 
> Nit pick that this should be "Testing", not "Testings".  (Or
> "Experiments" if you really wanted a plural noun.)

I'll tweak that while staging the patch for a pull request; no need
for another revision.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

[PATCH v4 2/3] nbd: Set unix socket send buffer on macOS

[Nir Soffer]

On macOS we need to increase unix socket buffers size on the client and
server to get good performance. We set socket buffers on macOS after
connecting or accepting a client connection.

Testing shows that setting socket receive buffer size (SO_RCVBUF) has no
effect on performance, so we set only the send buffer size (SO_SNDBUF).
It seems to work like Linux but not documented.

Testing shows that optimal buffer size is 512k to 4 MiB, depending on
the test case. The difference is very small, so I chose 2 MiB.

I tested reading from qemu-nbd and writing to qemu-nbd with qemu-img and
computing a blkhash with nbdcopy and blksum.

To focus on NBD communication and get less noisy results, I tested
reading and writing to null-co driver. I added a read-pattern option to
the null-co driver to return data full of 0xff:

    NULL="json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '10g', 'read-pattern': 255}}"

For testing buffer size I added an environment variable for setting the
socket buffer size.

Read from qemu-nbd via qemu-img convert. In this test buffer size of 2m
is optimal (12.6 times faster).

    qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
    qemu-img convert -f raw -O raw -W -n "nbd+unix:///?socket=/tmp/nbd.sock" "$NULL"

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |  13.361 |   2.653 |   5.702 |
|       65536 |   2.283 |   0.204 |   1.318 |
|      131072 |   1.673 |   0.062 |   1.008 |
|      262144 |   1.592 |   0.053 |   0.952 |
|      524288 |   1.496 |   0.049 |   0.887 |
|     1048576 |   1.234 |   0.047 |   0.738 |
|     2097152 |   1.060 |   0.080 |   0.602 |
|     4194304 |   1.061 |   0.076 |   0.604 |

Write to qemu-nbd with qemu-img convert. In this test buffer size of 2m
is optimal (9.2 times faster).

    qemu-nbd -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
    qemu-img convert -f raw -O raw -W -n "$NULL" "nbd+unix:///?socket=/tmp/nbd.sock"

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |   8.063 |   2.522 |   4.184 |
|       65536 |   1.472 |   0.430 |   0.867 |
|      131072 |   1.071 |   0.297 |   0.654 |
|      262144 |   1.012 |   0.239 |   0.587 |
|      524288 |   0.970 |   0.201 |   0.514 |
|     1048576 |   0.895 |   0.184 |   0.454 |
|     2097152 |   0.877 |   0.174 |   0.440 |
|     4194304 |   0.944 |   0.231 |   0.535 |

Compute a blkhash with nbdcopy, using 4 NBD connections and 256k request
size. In this test buffer size of 4m is optimal (5.1 times faster).

    qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
    nbdcopy --blkhash "nbd+unix:///?socket=/tmp/nbd.sock" null:

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |   8.624 |   5.727 |   6.507 |
|       65536 |   2.563 |   4.760 |   2.498 |
|      131072 |   1.903 |   4.559 |   2.093 |
|      262144 |   1.759 |   4.513 |   1.935 |
|      524288 |   1.729 |   4.489 |   1.924 |
|     1048576 |   1.696 |   4.479 |   1.884 |
|     2097152 |   1.710 |   4.480 |   1.763 |
|     4194304 |   1.687 |   4.479 |   1.712 |

Compute a blkhash with blksum, using 1 NBD connection and 256k read
size. In this test buffer size of 512k is optimal (10.3 times faster).

    qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
    blksum "nbd+unix:///?socket=/tmp/nbd.sock"

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |  13.085 |   5.664 |   6.461 |
|       65536 |   3.299 |   5.106 |   2.515 |
|      131072 |   2.396 |   4.989 |   2.069 |
|      262144 |   1.607 |   4.724 |   1.555 |
|      524288 |   1.271 |   4.528 |   1.224 |
|     1048576 |   1.294 |   4.565 |   1.333 |
|     2097152 |   1.299 |   4.569 |   1.344 |
|     4194304 |   1.291 |   4.559 |   1.327 |

Signed-off-by: Nir Soffer <nirsof@gmail.com>
---
 nbd/client-connection.c |  3 +++
 nbd/common.c            | 25 +++++++++++++++++++++++++
 nbd/nbd-internal.h      |  5 +++++
 nbd/server.c            |  2 ++
 4 files changed, 35 insertions(+)

diff --git a/nbd/client-connection.c b/nbd/client-connection.c
index b11e266807..79ea97e4cc 100644
--- a/nbd/client-connection.c
+++ b/nbd/client-connection.c
@@ -31,6 +31,8 @@
 #include "qapi/clone-visitor.h"
 #include "qemu/coroutine.h"
 
+#include "nbd/nbd-internal.h"
+
 struct NBDClientConnection {
     /* Initialization constants, never change */
     SocketAddress *saddr; /* address to connect to */
@@ -140,6 +142,7 @@ static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr,
         return ret;
     }
 
+    nbd_set_socket_send_buffer(sioc);
     qio_channel_set_delay(QIO_CHANNEL(sioc), false);
 
     if (!info) {
diff --git a/nbd/common.c b/nbd/common.c
index 589a748cfe..9436e9d1d1 100644
--- a/nbd/common.c
+++ b/nbd/common.c
@@ -18,6 +18,9 @@
 
 #include "qemu/osdep.h"
 #include "trace.h"
+#include "io/channel-socket.h"
+#include "qapi/error.h"
+#include "qemu/units.h"
 #include "nbd-internal.h"
 
 /* Discard length bytes from channel.  Return -errno on failure and 0 on
@@ -264,3 +267,25 @@ const char *nbd_mode_lookup(NBDMode mode)
         return "<unknown>";
     }
 }
+
+/*
+ * Testing shows that 2m send buffer is optimal. Changing the receive buffer
+ * size has no effect on performance.
+ */
+#if defined(__APPLE__)
+#define UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE (2 * MiB)
+#endif
+
+void nbd_set_socket_send_buffer(QIOChannelSocket *sioc)
+{
+#ifdef UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE
+    if (sioc->localAddr.ss_family == AF_UNIX) {
+        size_t size = UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE;
+        Error *errp = NULL;
+
+        if (qio_channel_socket_set_send_buffer(sioc, size, &errp) < 0) {
+            warn_report_err(errp);
+        }
+    }
+#endif /* UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE */
+}
diff --git a/nbd/nbd-internal.h b/nbd/nbd-internal.h
index 715d92d6ef..6bafeef5dd 100644
--- a/nbd/nbd-internal.h
+++ b/nbd/nbd-internal.h
@@ -74,4 +74,9 @@ static inline int nbd_write(QIOChannel *ioc, const void *buffer, size_t size,
 
 int nbd_drop(QIOChannel *ioc, size_t size, Error **errp);
 
+/* nbd_set_socket_send_buffer
+ * Set the socket send buffer size for optimal performance.
+ */
+void nbd_set_socket_send_buffer(QIOChannelSocket *sioc);
+
 #endif
diff --git a/nbd/server.c b/nbd/server.c
index 2076fb2666..d242be9811 100644
--- a/nbd/server.c
+++ b/nbd/server.c
@@ -3291,6 +3291,8 @@ void nbd_client_new(QIOChannelSocket *sioc,
     client->close_fn = close_fn;
     client->owner = owner;
 
+    nbd_set_socket_send_buffer(sioc);
+
     co = qemu_coroutine_create(nbd_co_client_start, client);
     qemu_coroutine_enter(co);
 }
-- 
2.39.5 (Apple Git-154)

Re: [PATCH v4 2/3] nbd: Set unix socket send buffer on macOS

[Daniel P. Berrangé]

On Sat, May 17, 2025 at 11:11:53PM +0300, Nir Soffer wrote:
> On macOS we need to increase unix socket buffers size on the client and
> server to get good performance. We set socket buffers on macOS after
> connecting or accepting a client connection.
> 
> Testing shows that setting socket receive buffer size (SO_RCVBUF) has no
> effect on performance, so we set only the send buffer size (SO_SNDBUF).
> It seems to work like Linux but not documented.
> 
> Testing shows that optimal buffer size is 512k to 4 MiB, depending on
> the test case. The difference is very small, so I chose 2 MiB.
> 
> I tested reading from qemu-nbd and writing to qemu-nbd with qemu-img and
> computing a blkhash with nbdcopy and blksum.
> 
> To focus on NBD communication and get less noisy results, I tested
> reading and writing to null-co driver. I added a read-pattern option to
> the null-co driver to return data full of 0xff:
> 
>     NULL="json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '10g', 'read-pattern': 255}}"
> 
> For testing buffer size I added an environment variable for setting the
> socket buffer size.
> 
> Read from qemu-nbd via qemu-img convert. In this test buffer size of 2m
> is optimal (12.6 times faster).
> 
>     qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
>     qemu-img convert -f raw -O raw -W -n "nbd+unix:///?socket=/tmp/nbd.sock" "$NULL"
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |  13.361 |   2.653 |   5.702 |
> |       65536 |   2.283 |   0.204 |   1.318 |
> |      131072 |   1.673 |   0.062 |   1.008 |
> |      262144 |   1.592 |   0.053 |   0.952 |
> |      524288 |   1.496 |   0.049 |   0.887 |
> |     1048576 |   1.234 |   0.047 |   0.738 |
> |     2097152 |   1.060 |   0.080 |   0.602 |
> |     4194304 |   1.061 |   0.076 |   0.604 |
> 
> Write to qemu-nbd with qemu-img convert. In this test buffer size of 2m
> is optimal (9.2 times faster).
> 
>     qemu-nbd -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
>     qemu-img convert -f raw -O raw -W -n "$NULL" "nbd+unix:///?socket=/tmp/nbd.sock"
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   8.063 |   2.522 |   4.184 |
> |       65536 |   1.472 |   0.430 |   0.867 |
> |      131072 |   1.071 |   0.297 |   0.654 |
> |      262144 |   1.012 |   0.239 |   0.587 |
> |      524288 |   0.970 |   0.201 |   0.514 |
> |     1048576 |   0.895 |   0.184 |   0.454 |
> |     2097152 |   0.877 |   0.174 |   0.440 |
> |     4194304 |   0.944 |   0.231 |   0.535 |
> 
> Compute a blkhash with nbdcopy, using 4 NBD connections and 256k request
> size. In this test buffer size of 4m is optimal (5.1 times faster).
> 
>     qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
>     nbdcopy --blkhash "nbd+unix:///?socket=/tmp/nbd.sock" null:
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   8.624 |   5.727 |   6.507 |
> |       65536 |   2.563 |   4.760 |   2.498 |
> |      131072 |   1.903 |   4.559 |   2.093 |
> |      262144 |   1.759 |   4.513 |   1.935 |
> |      524288 |   1.729 |   4.489 |   1.924 |
> |     1048576 |   1.696 |   4.479 |   1.884 |
> |     2097152 |   1.710 |   4.480 |   1.763 |
> |     4194304 |   1.687 |   4.479 |   1.712 |
> 
> Compute a blkhash with blksum, using 1 NBD connection and 256k read
> size. In this test buffer size of 512k is optimal (10.3 times faster).
> 
>     qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" &
>     blksum "nbd+unix:///?socket=/tmp/nbd.sock"
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |  13.085 |   5.664 |   6.461 |
> |       65536 |   3.299 |   5.106 |   2.515 |
> |      131072 |   2.396 |   4.989 |   2.069 |
> |      262144 |   1.607 |   4.724 |   1.555 |
> |      524288 |   1.271 |   4.528 |   1.224 |
> |     1048576 |   1.294 |   4.565 |   1.333 |
> |     2097152 |   1.299 |   4.569 |   1.344 |
> |     4194304 |   1.291 |   4.559 |   1.327 |
> 
> Signed-off-by: Nir Soffer <nirsof@gmail.com>
> ---
>  nbd/client-connection.c |  3 +++
>  nbd/common.c            | 25 +++++++++++++++++++++++++
>  nbd/nbd-internal.h      |  5 +++++
>  nbd/server.c            |  2 ++
>  4 files changed, 35 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v4 3/3] nbd: Set unix socket send buffer on Linux

[Nir Soffer]

Like macOS we have similar issue on Linux. For TCP socket the send
buffer size is 2626560 bytes (~2.5 MiB) and we get good performance.
However for unix socket the default and maximum buffer size is 212992
bytes (208 KiB) and we see poor performance when using one NBD
connection, up to 4 times slower than macOS on the same machine.

Tracing shows that for every 2 MiB payload (qemu uses 2 MiB io size), we
do 1 recvmsg call with TCP socket, and 10 recvmsg calls with unix
socket.

Fixing this issue requires changing the maximum send buffer size (the
receive buffer size is ignored). This can be done using:

    $ cat /etc/sysctl.d/net-mem-max.conf
    net.core.wmem_max = 2097152

    $ sudo sysctl -p /etc/sysctl.d/net-mem-max.conf

With this we can set the socket buffer size to 2 MiB. With the defaults
the value requested by qemu is clipped to the maximum size and has no
effect.

I tested on 2 machines:
- Fedora 42 VM on MacBook Pro M2 Max
- Dell PowerEdge R640 (Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz)

On the older Dell machine we see very little improvement, up to 1.03
higher throughput. On the M2 machine we see up to 2.67 times higher
throughput. The following results are from the M2 machine.

Reading from qemu-nbd with qemu-img convert. In this test buffer size of
4m is optimal (2.28 times faster).

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |   4.292 |   0.243 |   1.604 |
|      524288 |   2.167 |   0.058 |   1.288 |
|     1048576 |   2.041 |   0.060 |   1.238 |
|     2097152 |   1.884 |   0.060 |   1.191 |
|     4194304 |   1.881 |   0.054 |   1.196 |

Writing to qemu-nbd with qemu-img convert. In this test buffer size of
1m is optimal (2.67 times faster).

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |   3.113 |   0.334 |   1.094 |
|      524288 |   1.173 |   0.179 |   0.654 |
|     1048576 |   1.164 |   0.164 |   0.670 |
|     2097152 |   1.227 |   0.197 |   0.663 |
|     4194304 |   1.227 |   0.198 |   0.666 |

Computing a blkhash with nbdcopy. In this test buffer size of 512k is
optimal (1.19 times faster).

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |   2.140 |   4.483 |   2.681 |
|      524288 |   1.794 |   4.467 |   2.572 |
|     1048576 |   1.807 |   4.447 |   2.644 |
|     2097152 |   1.822 |   4.461 |   2.698 |
|     4194304 |   1.827 |   4.465 |   2.700 |

Computing a blkhash with blksum. In this test buffer size of 4m is
optimal (2.65 times faster).

| buffer size | time    | user    | system  |
|-------------|---------|---------|---------|
|     default |   3.582 |   4.595 |   2.392 |
|      524288 |   1.499 |   4.384 |   1.482 |
|     1048576 |   1.377 |   4.381 |   1.345 |
|     2097152 |   1.388 |   4.389 |   1.354 |
|     4194304 |   1.352 |   4.395 |   1.302 |

Signed-off-by: Nir Soffer <nirsof@gmail.com>
---
 nbd/common.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/nbd/common.c b/nbd/common.c
index 9436e9d1d1..2a133a66c3 100644
--- a/nbd/common.c
+++ b/nbd/common.c
@@ -271,8 +271,9 @@ const char *nbd_mode_lookup(NBDMode mode)
 /*
  * Testing shows that 2m send buffer is optimal. Changing the receive buffer
  * size has no effect on performance.
+ * On Linux we need to increase net.core.wmem_max to make this effective.
  */
-#if defined(__APPLE__)
+#if defined(__APPLE__) || defined(__linux__)
 #define UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE (2 * MiB)
 #endif
 
-- 
2.39.5 (Apple Git-154)

Re: [PATCH v4 3/3] nbd: Set unix socket send buffer on Linux

[Daniel P. Berrangé]

On Sat, May 17, 2025 at 11:11:54PM +0300, Nir Soffer wrote:
> Like macOS we have similar issue on Linux. For TCP socket the send
> buffer size is 2626560 bytes (~2.5 MiB) and we get good performance.
> However for unix socket the default and maximum buffer size is 212992
> bytes (208 KiB) and we see poor performance when using one NBD
> connection, up to 4 times slower than macOS on the same machine.
> 
> Tracing shows that for every 2 MiB payload (qemu uses 2 MiB io size), we
> do 1 recvmsg call with TCP socket, and 10 recvmsg calls with unix
> socket.
> 
> Fixing this issue requires changing the maximum send buffer size (the
> receive buffer size is ignored). This can be done using:
> 
>     $ cat /etc/sysctl.d/net-mem-max.conf
>     net.core.wmem_max = 2097152
> 
>     $ sudo sysctl -p /etc/sysctl.d/net-mem-max.conf
> 
> With this we can set the socket buffer size to 2 MiB. With the defaults
> the value requested by qemu is clipped to the maximum size and has no
> effect.
> 
> I tested on 2 machines:
> - Fedora 42 VM on MacBook Pro M2 Max
> - Dell PowerEdge R640 (Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz)
> 
> On the older Dell machine we see very little improvement, up to 1.03
> higher throughput. On the M2 machine we see up to 2.67 times higher
> throughput. The following results are from the M2 machine.
> 
> Reading from qemu-nbd with qemu-img convert. In this test buffer size of
> 4m is optimal (2.28 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   4.292 |   0.243 |   1.604 |
> |      524288 |   2.167 |   0.058 |   1.288 |
> |     1048576 |   2.041 |   0.060 |   1.238 |
> |     2097152 |   1.884 |   0.060 |   1.191 |
> |     4194304 |   1.881 |   0.054 |   1.196 |
> 
> Writing to qemu-nbd with qemu-img convert. In this test buffer size of
> 1m is optimal (2.67 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   3.113 |   0.334 |   1.094 |
> |      524288 |   1.173 |   0.179 |   0.654 |
> |     1048576 |   1.164 |   0.164 |   0.670 |
> |     2097152 |   1.227 |   0.197 |   0.663 |
> |     4194304 |   1.227 |   0.198 |   0.666 |
> 
> Computing a blkhash with nbdcopy. In this test buffer size of 512k is
> optimal (1.19 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   2.140 |   4.483 |   2.681 |
> |      524288 |   1.794 |   4.467 |   2.572 |
> |     1048576 |   1.807 |   4.447 |   2.644 |
> |     2097152 |   1.822 |   4.461 |   2.698 |
> |     4194304 |   1.827 |   4.465 |   2.700 |
> 
> Computing a blkhash with blksum. In this test buffer size of 4m is
> optimal (2.65 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   3.582 |   4.595 |   2.392 |
> |      524288 |   1.499 |   4.384 |   1.482 |
> |     1048576 |   1.377 |   4.381 |   1.345 |
> |     2097152 |   1.388 |   4.389 |   1.354 |
> |     4194304 |   1.352 |   4.395 |   1.302 |
> 
> Signed-off-by: Nir Soffer <nirsof@gmail.com>
> ---
>  nbd/common.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 0/3] nbd: Increase unix socket buffer size

[Eric Blake]

On Sat, May 17, 2025 at 11:11:51PM +0300, Nir Soffer wrote:
> On both macOS and Linux, the default send buffer size is too small causing poor
> performance when reading and writing to qemu-nbd. A simple way to experience
> this is to compare TCP and unix sockets, showing that TCP socket is much
> faster. Programs like nbdcopy partly mitigate this by using multiple NBD
> connections.
> 
> On macOS the default send buffer size is 8192 bytes. Increasing the send buffer
> size to 2 MiB shows up to *12.6 times higher throughput* and lower cpu usage.
> 
> On Linux the default and maximum buffer size is 212992 bytes. Increasing the
> send buffer size to 2 MiB shows up to *2.7 times higher throughput* and lower
> cpu usage. On older machine we see very little improvement, up to 1.03 times
> higher throughput.
> 
> We likely have the same issue on other platforms. It should be easy to enable
> this change for more platform by defining UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE.

Thanks; I'm staging this through my NBD tree.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org