Re: [PATCH v4 3/3] nbd: Set unix socket send buffer on Linux

["Daniel P. Berrangé" <berrange@redhat.com>]

On Sat, May 17, 2025 at 11:11:54PM +0300, Nir Soffer wrote:
> Like macOS we have similar issue on Linux. For TCP socket the send
> buffer size is 2626560 bytes (~2.5 MiB) and we get good performance.
> However for unix socket the default and maximum buffer size is 212992
> bytes (208 KiB) and we see poor performance when using one NBD
> connection, up to 4 times slower than macOS on the same machine.
> 
> Tracing shows that for every 2 MiB payload (qemu uses 2 MiB io size), we
> do 1 recvmsg call with TCP socket, and 10 recvmsg calls with unix
> socket.
> 
> Fixing this issue requires changing the maximum send buffer size (the
> receive buffer size is ignored). This can be done using:
> 
>     $ cat /etc/sysctl.d/net-mem-max.conf
>     net.core.wmem_max = 2097152
> 
>     $ sudo sysctl -p /etc/sysctl.d/net-mem-max.conf
> 
> With this we can set the socket buffer size to 2 MiB. With the defaults
> the value requested by qemu is clipped to the maximum size and has no
> effect.
> 
> I tested on 2 machines:
> - Fedora 42 VM on MacBook Pro M2 Max
> - Dell PowerEdge R640 (Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz)
> 
> On the older Dell machine we see very little improvement, up to 1.03
> higher throughput. On the M2 machine we see up to 2.67 times higher
> throughput. The following results are from the M2 machine.
> 
> Reading from qemu-nbd with qemu-img convert. In this test buffer size of
> 4m is optimal (2.28 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   4.292 |   0.243 |   1.604 |
> |      524288 |   2.167 |   0.058 |   1.288 |
> |     1048576 |   2.041 |   0.060 |   1.238 |
> |     2097152 |   1.884 |   0.060 |   1.191 |
> |     4194304 |   1.881 |   0.054 |   1.196 |
> 
> Writing to qemu-nbd with qemu-img convert. In this test buffer size of
> 1m is optimal (2.67 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   3.113 |   0.334 |   1.094 |
> |      524288 |   1.173 |   0.179 |   0.654 |
> |     1048576 |   1.164 |   0.164 |   0.670 |
> |     2097152 |   1.227 |   0.197 |   0.663 |
> |     4194304 |   1.227 |   0.198 |   0.666 |
> 
> Computing a blkhash with nbdcopy. In this test buffer size of 512k is
> optimal (1.19 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   2.140 |   4.483 |   2.681 |
> |      524288 |   1.794 |   4.467 |   2.572 |
> |     1048576 |   1.807 |   4.447 |   2.644 |
> |     2097152 |   1.822 |   4.461 |   2.698 |
> |     4194304 |   1.827 |   4.465 |   2.700 |
> 
> Computing a blkhash with blksum. In this test buffer size of 4m is
> optimal (2.65 times faster).
> 
> | buffer size | time    | user    | system  |
> |-------------|---------|---------|---------|
> |     default |   3.582 |   4.595 |   2.392 |
> |      524288 |   1.499 |   4.384 |   1.482 |
> |     1048576 |   1.377 |   4.381 |   1.345 |
> |     2097152 |   1.388 |   4.389 |   1.354 |
> |     4194304 |   1.352 |   4.395 |   1.302 |
> 
> Signed-off-by: Nir Soffer <nirsof@gmail.com>
> ---
>  nbd/common.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|