From: Bart Van Assche <bvanassche@acm.org>
To: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Cc: linux-block@vger.kernel.org, Bart Van Assche <bvanassche@acm.org>
Subject: [PATCH blktests v2 2/2] nbd/006: Measure NBD IOPS with a null_blk backing device
Date: Fri, 20 Nov 2026 13:34:34 -0800 [thread overview]
Message-ID: <20261120213434.2945873-3-bvanassche@acm.org> (raw)
In-Reply-To: <20261120213434.2945873-1-bvanassche@acm.org>
Add a test that measures the random read and random write IOPS that the
NBD driver can achieve, both for a single outstanding request and for
the maximum achievable parallelism. Such a test is useful to detect
latency and throughput regressions in the NBD driver and to evaluate NBD
performance optimizations.
Minimize the overhead outside the NBD driver as follows:
- Back the NBD export with a 1 GiB null_blk device that is tuned for
maximum IOPS: no memory backing, immediate completion, a 4 KiB logical
block size, one hardware queue per CPU and a hardware queue depth of
1024. Keep the device small because nbd-server opens the exported
device without O_DIRECT and hence all NBD I/O passes through the page
cache of the null_blk device.
- Select the "none" I/O scheduler for the null_blk device and disable
readahead such that each 4 KiB NBD read results in a single 4 KiB read
of the null_blk device.
- Connect nbd-client and nbd-server over a Unix domain socket if
nbd-client supports the -unix option and nbd-server supports the
"unixsock" configuration option. Fall back to a TCP connection over
the loopback interface otherwise or if the Unix domain socket does not
appear.
- Use one NBD connection per CPU, with a maximum of 16 connections.
- Measure the IOPS with fio using the io_uring I/O engine, the "none" I/O
scheduler for the NBD device, direct I/O and 4 KiB requests.
Run fio four times, each run taking a quarter of the test time:
- Random reads and random writes with a single job and a queue depth of
one. These results reflect the NBD round-trip latency.
- Random reads and random writes with one job per NBD connection and a
queue depth of 64. These results reflect the maximum NBD throughput.
Report the results via the "read iops" and "write iops" performance
fields, prefixed with "numjobs=<n> iodepth=<d> " to tell the two
configurations apart.
Note: since nbd-server performs buffered writes, the write IOPS mainly
reflect how fast nbd-server can dirty pages in the page cache.
Fail the test if configuring the null_blk request queue fails or if
nbd-server does not start listening on the TCP port in time. Fall back
to TCP instead of failing if the Unix domain socket does not appear in
time.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
tests/nbd/006 | 202 ++++++++++++++++++++++++++++++++++++++++++++++
tests/nbd/006.out | 2 +
2 files changed, 204 insertions(+)
create mode 100755 tests/nbd/006
create mode 100644 tests/nbd/006.out
diff --git a/tests/nbd/006 b/tests/nbd/006
new file mode 100755
index 000000000000..03bb0fc66bb4
--- /dev/null
+++ b/tests/nbd/006
@@ -0,0 +1,202 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2026 Google LLC
+#
+# Measure the random read and random write IOPS of an NBD device that is backed
+# by a null_blk device. Use a Unix domain socket for the NBD connections if
+# supported by nbd-client and nbd-server and TCP otherwise.
+
+. tests/nbd/rc
+. common/null_blk
+
+DESCRIPTION="measure NBD read and write IOPS with null_blk as backing device"
+TIMED=1
+
+requires() {
+ _have_nbd
+ _have_null_blk
+ _have_fio
+}
+
+# Whether both nbd-client and nbd-server support Unix domain sockets:
+# nbd-client must support the -unix option and nbd-server must recognize the
+# "unixsock" configuration option. Since nbd-server does not report which
+# configuration options it supports, look for the option name in the
+# nbd-server executable.
+_nbd_have_unix_socket() {
+ local server
+
+ nbd-client --help 2>&1 |
+ grep -qE -- '(^|[^[:alnum:]-])-unix([^[:alnum:]-]|$)' || return 1
+ server=$(type -P nbd-server) || return 1
+ grep -aqw unixsock "$server"
+}
+
+# Undo those setup steps of test() that have been completed. Uses the local
+# variables of test() via dynamic scoping.
+cleanup() {
+ local -i j
+ local pid
+
+ if ((nbd_connected)); then
+ nbd-client -d "$nbd_dev" >> "$FULL" 2>&1
+ _wait_for_nbd_disconnect
+ fi
+ if [[ -s "$pid_file" ]]; then
+ pid=$(<"$pid_file")
+ kill -SIGTERM "$pid"
+ # Wait until nbd-server has exited such that it no longer keeps
+ # the null_blk device open.
+ for ((j = 0; j < 100; j++)); do
+ kill -0 "$pid" 2>/dev/null || break
+ sleep .1
+ done
+ fi
+ rm -f "$pid_file" "$sock"
+ udevadm settle
+ if ((null_blk_loaded)); then
+ _exit_null_blk
+ fi
+}
+
+test() {
+ local nullb=nullb1 nullb_dev=/dev/nullb1 nbd_dev=/dev/nbd0
+ local sock="${TMPDIR}/nbd.sock" pid_file="${TMPDIR}/nbd.pid"
+ local conf="${TMPDIR}/nbd.conf"
+ local -i nr_cpus nr_conns i use_unix=0
+ local -a connect_args
+ # Setup progress, used by cleanup().
+ local -i null_blk_loaded=0 nbd_connected=0
+
+ echo "Running ${TEST_NAME}"
+
+ nr_cpus=$(nproc)
+ nr_conns=$((nr_cpus < 16 ? nr_cpus : 16))
+
+ # Load null_blk without creating any device and next create a single
+ # null_blk device via configfs that is tuned for maximum IOPS: no
+ # memory backing, immediate completion and one hardware queue per CPU.
+ if ! _init_null_blk nr_devices=0; then
+ echo "Loading null_blk failed"
+ return 1
+ fi
+ null_blk_loaded=1
+ local nullb_params=(
+ blocksize=4096
+ completion_nsec=0
+ hw_queue_depth=1024
+ irqmode=0
+ memory_backed=0
+ queue_mode=2
+ size=1024 # MiB
+ submit_queues="$nr_cpus"
+ power=1
+ )
+ if ! _configure_null_blk "$nullb" "${nullb_params[@]}"; then
+ echo "Configuring null_blk failed"
+ cleanup
+ return 1
+ fi
+ # nbd-server opens the exported device without O_DIRECT. Disable
+ # readahead such that each NBD read causes a single 4 KiB read of the
+ # null_blk device.
+ if ! _write_queue_sched none "/sys/block/${nullb}/queue/scheduler" ||
+ ! grep -q '\[none\]' "/sys/block/${nullb}/queue/scheduler" ||
+ ! echo 0 > "/sys/block/${nullb}/queue/read_ahead_kb"; then
+ echo "Configuring the ${nullb} request queue failed"
+ cleanup
+ return 1
+ fi
+
+ _nbd_have_unix_socket && use_unix=1
+
+ {
+ echo "[generic]"
+ echo "allowlist=true"
+ if ((use_unix)); then
+ echo "unixsock=${sock}"
+ fi
+ echo "[export]"
+ echo "exportname=${nullb_dev}"
+ } > "$conf"
+ cat "$conf" >> "$FULL"
+
+ if ! nbd-server -p "$pid_file" -C "$conf" >> "$FULL" 2>&1; then
+ echo "Starting nbd-server failed"
+ cleanup
+ return 1
+ fi
+
+ # Wait until nbd-server listens on the Unix domain socket. Fall back to
+ # TCP if the Unix domain socket does not show up.
+ if ((use_unix)); then
+ for ((i = 0; i < 100; i++)); do
+ [[ -S "$sock" ]] && break
+ sleep .1
+ done
+ if ((i >= 100)); then
+ echo "${sock} did not appear; falling back to TCP" >> "$FULL"
+ use_unix=0
+ fi
+ fi
+ if ((use_unix)); then
+ echo "Using Unix domain socket ${sock}" >> "$FULL"
+ connect_args=(-unix "$sock")
+ else
+ echo "Using TCP socket" >> "$FULL"
+ connect_args=(localhost)
+ for ((i = 0; i < 100; i++)); do
+ nbd-client -l localhost >> "$FULL" 2>&1 && break
+ sleep .1
+ done
+ if ((i >= 100)); then
+ echo "Timeout while waiting for nbd-server"
+ cleanup
+ return 1
+ fi
+ fi
+
+ # Use -connections instead of -C because some nbd-client versions
+ # do not declare that -C takes an argument and pass a NULL pointer
+ # to strtol() if -C is used.
+ nbd_connected=1
+ if ! nbd-client -L -N export -connections "$nr_conns" -b 4096 \
+ "${connect_args[@]}" "$nbd_dev" >> "$FULL" 2>&1 ||
+ ! _wait_for_nbd_connect; then
+ echo "Connecting to nbd-server failed"
+ cleanup
+ return 1
+ fi
+ udevadm settle
+
+ # Measure the read and write IOPS twice: once with a single job and
+ # queue depth one and once with the parameters that maximize the IOPS.
+ # _run_fio passes --group_reporting=1 and --runtime="$TIMEOUT" to fio.
+ : "${TIMEOUT:=60}"
+ _divide_timeout 4
+
+ local fio_params=(
+ --bs=4k
+ --direct=1
+ --filename="$nbd_dev"
+ --ioengine=io_uring
+ --ioscheduler=none
+ --norandommap
+ --time_based=1
+ )
+ local rw numjobs iodepth numjobs_iodepth
+ for numjobs_iodepth in "1 1" "${nr_conns} 64"; do
+ read -r numjobs iodepth <<< "$numjobs_iodepth"
+ FIO_PERF_PREFIX="numjobs=${numjobs} iodepth=${iodepth} "
+ for rw in read write; do
+ FIO_PERF_FIELDS=("${rw} iops")
+ _fio_perf --name="nbd-${rw}-iops" --rw="rand${rw}" \
+ --numjobs="$numjobs" --iodepth="$iodepth" \
+ "${fio_params[@]}"
+ done
+ done
+
+ cleanup
+
+ echo "Test complete"
+}
diff --git a/tests/nbd/006.out b/tests/nbd/006.out
new file mode 100644
index 000000000000..acc8af5bfb2e
--- /dev/null
+++ b/tests/nbd/006.out
@@ -0,0 +1,2 @@
+Running nbd/006
+Test complete
prev parent reply other threads:[~2026-11-20 21:34 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 21:34 [PATCH blktests v2 0/2] Add an NBD performance test Bart Van Assche
2026-11-20 21:34 ` [PATCH blktests v2 1/2] nbd/rc: Declare the loop variables of the wait helpers local Bart Van Assche
2026-11-20 21:34 ` Bart Van Assche [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120213434.2945873-3-bvanassche@acm.org \
--to=bvanassche@acm.org \
--cc=linux-block@vger.kernel.org \
--cc=shinichiro.kawasaki@wdc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox