Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>]

[-- Attachment #1: Type: text/plain, Size: 2250 bytes --]

On Feb 23, 2026 / 07:44, Johannes Thumshirn wrote:
> On 2/15/26 10:18 PM, Haris Iqbal wrote:
> >>  From my view, blktests keep on finding kernel bugs. I think it demonstrates the
> >> value of this community effort, and I'm happy about it. Said that, I find what
> >> blktests can improve more, of course. Here I share the list of improvement
> >> opportunities from my view point (I already mentioned the first three items).
> > A possible feature for blktest could be integration with something
> > like virtme-ng.
> > Running on VM can be versatile and fast. The run can be made parallel
> > too, by spawning multiple VMs simultaneously.
> 
> This is actually rather trivial to solve I have some pre-made things for 
> fstests and that can be adopted for blktests as well:
> 
> vng \
>      --user=root -v --name vng-tcmu-runner \
>      -a loglevel=3 \
>      --run $KDIR \
>      --cpus=8 --memory=8G \
>      --exec "~johannes/src/ci/run-fstests.sh" \
>      --qemu-opts="-device virtio-scsi,id=scsi0 -drive 
> file=/dev/sda,format=raw,if=none,id=zbc0 -device 
> scsi-block,bus=scsi0.0,drive=zbc0" \
>      --qemu-opts="-device virtio-scsi,id=scsi1 -drive 
> file=/dev/sdb,format=raw,if=none,id=zbc1 -device 
> scsi-block,bus=scsi1.0,drive=zbc1"
> 
> and run-fstests.sh is:
> 
> #!/bin/sh
> # SPDX-License-Identifier: GPL-2.0
> 
> DIR="/tmp/"
> MKFS="mkfs.btrfs -f"
> FSTESTS_DIR="/home/johannes/src/fstests"
> HOSTCONF="$FSTESTS_DIR/configs/$(hostname -s)"
> TESTDEV="$(grep TEST_DEV $HOSTCONF | cut -d '=' -f 2)"
> 
> mkdir -p $DIR/{test,scratch,results}
> $MKFS $TESTDEV
> 
> cd $FSTESTS_DIR
> ./check -x raid
> 
> I'm not sure it'll make sense to include this into blktests other than 
> maybe providing an example in the README.

I guess the example can be added in contrib/. This idea interested me, so I did
some quick scripting and created the patch attached. I did some blktests runs
with virtme-ng, and found that:

- virtme-ng allows to skip kernel installation step. Fast and useful as Haris
  pointed out.
- systemd does not look working well with virtme-ng, even when I specify the
  --systemd option. This will be a condition difference from normal
  blktests runs.

I hope that the patch will help discussion at the conference.

[-- Attachment #2: 0001-contrib-add-scripts-to-run-blktests-with-virtme-ng.patch --]
[-- Type: text/plain, Size: 5517 bytes --]

From 4a310c98f674e65218fb7e33e3b176a85d897333 Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Date: Tue, 21 Apr 2026 14:26:01 +0900
Subject: [PATCH blktests RFC] contrib: add scripts to run blktests with
 virtme-ng

It takes rather long time to run blktests repeatedly for debug target
kernels because of the time to install the test target kernel to the
test target system. To reduce the turn around time, virtme-ng [1] is
useful. Add helper scripts to run blktests using virtme-ng per
suggestion by Johaness [2].

After building a kernel in ~/linux, the command lines below will run
blktests on the built kernel. With this, kernel installation step is no
longer required.

 $ KDIR=~/linux contrib/run-vng loop/010
 virtme: waiting for virtiofsd to start
 virtme: use 'microvm' QEMU architecture
 early console in setup code
 Probing EDD (edd=off to disable)... ok
 [    0.000000][    T0] Linux version 7.0.0 (shin@shinmob) (gcc (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7), GNU ld version 2.45.1-4.fc43) #22 SMP PREEMPT_DYNAMIC Mon Apr 13 15:44:07 JST 2026
 [    0.000000][    T0] Command line: virtme_hostname=vng-blktests-runner nr_open=2147483584 virtme_link_mods=/home/shin/linux/.virtme_mods/lib/modules/0.0.0 virtme_initmount0=tmp virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var virtme_rw_overlay7=/tmp virtme_console=ttyS0 console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1 virtme.exec=`L2hvbWUvc2hpbi9CbGt0ZXN0cy9ibGt0ZXN0cy9jb250cmliL3J1bi1ibGt0ZXN0cy1xdWljayBsb29wLzAwMQ==` virtme.ssh virtme_ssh_channel=vsock virtme_ssh_cache=/home/shin/.cache/virtme-ng/.ssh virtme_chdir=home/shin/Blktests/blktests/contrib debug loglevel=3 init=/usr/lib/python3.14/site-packages/virtme/guest/bin/virtme-ng-init
 [    0.000000][    T0] x86/split lock detection: #DB: warning on user-space bus_locks
 [    0.000000][    T0] BIOS-provided physical RAM map:
 [    0.000000][    T0] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff]  System RAM
 [    0.000000][    T0] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [gap 0x00000000000a0000-0x00000000000effff]
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [mem 0x0000000000100000-0x00000000bfffefff]  System RAM
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000bffff000-0x00000000bfffffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [gap 0x00000000c0000000-0x00000000feffbfff]
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [gap 0x00000000ff000000-0x00000000fffbffff]
 [    0.000000][    T0] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff]  device reserved
 [    0.000000][    T0] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff]  System RAM
 [    0.000000][    T0] printk: legacy bootconsole [earlyser0] enabled
 Poking KASLR using RDRAND RDTSC...
 Clean up /tmp/results ...
 loop/001 (scan loop device partitions)
 loop/001 (scan loop device partitions)                       [passed]
     runtime    ...  14.610s
 Run logs:
 [   90.927921][    T1] reboot: Power down

When I ran the script on my Fedora 43 testnode for some test groups, I
observed hangs at nvme/010 and nvme/012. Also I observed a failure at
loop/010. Some fixes will be required in blktests and kernel to run the
new script in stable manner.

Of note is that the script does not support test target block device
pass-through yet.

[1] https://github.com/arighi/virtme-ng
[2] https://lore.kernel.org/linux-block/ae47ef06-3f66-4aab-b4ab-f3ae2b634f87@wdc.com/

Suggested-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 contrib/run-blktests-quick | 27 +++++++++++++++++++++++++++
 contrib/run-vng            | 18 ++++++++++++++++++
 2 files changed, 45 insertions(+)
 create mode 100755 contrib/run-blktests-quick
 create mode 100755 contrib/run-vng

diff --git a/contrib/run-blktests-quick b/contrib/run-blktests-quick
new file mode 100755
index 0000000..9324c38
--- /dev/null
+++ b/contrib/run-blktests-quick
@@ -0,0 +1,27 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2026 Western Digital Corporation or its affiliates.
+
+declare DIR LOGDIR BLKTESTS_DIR
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+BLKTESTS_DIR="${DIR}/../"
+
+LOGDIR="/tmp/results"
+
+if [[ -e $LOGDIR ]]; then
+	echo "Clean up ${LOGDIR} ..."
+	rm -rf $LOGDIR
+fi
+mkdir -p "$LOGDIR"
+
+cd "$BLKTESTS_DIR" || exit
+
+cat > config << EOF
+QUICK_RUN=1
+TIMEOUT=10
+EOF
+
+./check -o "$LOGDIR" "$@"
+echo "Run logs: ${LOGIDR}"
+
diff --git a/contrib/run-vng b/contrib/run-vng
new file mode 100755
index 0000000..9aa4b91
--- /dev/null
+++ b/contrib/run-vng
@@ -0,0 +1,18 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2026 Western Digital Corporation or its affiliates.
+
+declare DIR
+declare -a BLKTESTS_OPTS
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+
+if (($#)); then
+	BLKTESTS_OPTS=("$@")
+else
+	BLKTESTS_OPTS=(block nvme loop scsi)
+fi
+
+vng \
+ --user=root --verbose --name vng-blktests-runner --cpus=4 --memory=4G \
+ --append loglevel=3 --run "$KDIR" --ssh \
+ --rwdir=/tmp --exec "$DIR/run-blktests-quick ${BLKTESTS_OPTS[*]}"
-- 
2.53.0