Hi, Since addition of the nvme_log_error(), we are getting reports of the error messages are printed from internal passthru commands. This also uncovered target execute identify code path where we don't handle the case for NVME_CSI_NVM and in the host where we issue non-mdts limit for the discovery contoller. First patch fixes target execute identify case, second patch prevents configuring non-mdts-limits for the dicovery controller and thrid patch masks the internal passthru error warnings tested with nvme error injection. Regarding the third patch Alan has reported it does mask nvme-admin-passthru commands errors when used with error injection framework, I wasn't able to reporduce that, maybe we can wait for Alan to confirm. Following is the test execution log with :- 1. Tests with CONFIG_BLK_DEV_ZONED enabled. 1.1 nvme-admin-passthru error injection logging error mesaages. 1.2 blktests results. 1.3 Test fio verify job with NVMeOF ZBD backend. 2. Tests with CONFIG_BLK_DEV_ZONED disabled :- 2.1 nvme-admin-passthru error injection logging error mesaages. 2.2 blktests results. 2.3 Try and create NVMeOF controller with ZBD backend & fail grececully. -ck Chaitanya Kulkarni (3): nvmet: handle admin default command set identifier nvme-core: don't check non-mdts for disc ctrl nvme-core: mark internal passthru req REQ_QUIET drivers/nvme/host/core.c | 7 ++++++- drivers/nvme/target/admin-cmd.c | 22 ++++++++++++++-------- drivers/nvme/target/nvmet.h | 2 +- drivers/nvme/target/zns.c | 2 +- 4 files changed, 22 insertions(+), 11 deletions(-) 1. Tests with CONFIG_BLK_DEV_ZONED enabled :- ----------------------------------------------------------------------- nvme (nvme-5.18) # git log -3 commit 0d77072e26fa074da0c36e423a5802fd18ef976d (HEAD -> nvme-5.18) Author: Chaitanya Kulkarni Date: Fri Apr 8 16:45:18 2022 -0700 nvme-core: mark internal passthru req REQ_QUIET Mark internal passthru requests quiet in the submission path with RQF_QUIET flag added in the __nvme_submit_sync_cmd(). In the completion path, if nvme request is resulted in the error and request is marked RQF_QUIET then don't log the error with nvme_error_log(). Signed-off-by: Chaitanya Kulkarni commit d4df054957f3cbd37601d65d8e7460aaaee8733d Author: Chaitanya Kulkarni Date: Fri Apr 8 15:50:10 2022 -0700 nvme-core: don't check non-mdts for disc ctrl Don't check the non mdts limits for the discovery controller as we don't support I/O command set, this also masks the following warning reported by the nvme_log_error() when running blktest nvme/002: - [ 2005.155946] run blktests nvme/002 at 2022-04-09 16:57:47 [ 2005.192223] loop: module loaded [ 2005.196429] nvmet: adding nsid 1 to subsystem blktests-subsystem-0 [ 2005.200334] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 <------------------------------SNIP----------------------------------> [ 2008.958108] nvmet: adding nsid 1 to subsystem blktests-subsystem-997 [ 2008.962082] nvmet: adding nsid 1 to subsystem blktests-subsystem-998 [ 2008.966102] nvmet: adding nsid 1 to subsystem blktests-subsystem-999 [ 2008.973132] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN testhostnqn. *[ 2008.973196] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) MORE DNR* [ 2008.974595] nvme nvme1: new ctrl: "nqn.2014-08.org.nvmexpress.discovery" [ 2009.103248] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" Signed-off-by: Chaitanya Kulkarni commit 18aac654ff824a4b169e51d72b674245a7a1b79c Author: Chaitanya Kulkarni Date: Thu Apr 7 12:11:37 2022 -0700 nvmet: handle admin default command set identifier The NVMeOF in kernel host when configuring non-mdts limits uses idnetify controller cns value to NVME_ID_CNS_CS_CTRL and csi value to NVME_CSI_NVM. In target code we only handle case for NVME_ID_CNS_CS_CTRL and NVME_CSI_ZNS when CONFIG_BLK_DEV_ZONED is set. Handle missing case for CONFIG_BLK_DEV_ZONED and !CONFIG_BLK_DEV_ZONED so it can handle NVME_ID_CNS_CS_CTRL with NVME_CSI_NVM. Use this opportunity to add default for the NVME_ID_CNS_CTRL case which makes code uniform for all the cases in the nvmet_execute_identify_cns_cs_ctrl_nvm(). Also, rename nvmet_execute_identify_ctrl() to nvmet_execute_identify_cns_cs_ctrl_nvm() and nvmet_execute_identify_cns_cs_ctrl() to nvmet_execute_identify_cns_cs_ctrl_zns(). This also masks the following warning reported by the nvme_log_error() when running blktest nvme/012:- [ 2131.702140] run blktests nvme/012 at 2022-04-09 16:59:53 [ 2131.722144] loop0: detected capacity change from 0 to 2097152 [ 2131.730586] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 2131.738826] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN testhostnqn. *[ 2131.738911] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) MORE DNR* [ 2131.740540] nvme nvme1: creating 48 I/O queues. [ 2131.743925] nvme nvme1: new ctrl: "blktests-subsystem-1" [ 2132.790058] XFS (nvme1n1): Mounting V5 Filesystem [ 2132.793667] XFS (nvme1n1): Ending clean mount [ 2132.794030] xfs filesystem being mounted at /mnt/blktests supports timestamps until 2038 (0x7fffffff) [ 2142.471812] XFS (nvme1n1): Unmounting Filesystem [ 2142.492566] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1" Signed-off-by: Chaitanya Kulkarni nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # grep -i zoned .config CONFIG_BLK_DEV_ZONED=y CONFIG_BLK_DEBUG_FS_ZONED=y # CONFIG_DM_ZONED is not set nvme (nvme-5.18) # ./compile_nvme.sh + umount /mnt/nvme0n1 + clear_dmesg ./compile_nvme.sh: line 3: clear_dmesg: command not found umount: /mnt/nvme0n1: no mount point specified. + ./delete.sh + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 0 controller(s) real 0m0.010s user 0m0.001s sys 0m0.004s + rm -fr '/sys/kernel/config/nvmet/ports/1/subsystems/*' + rmdir /sys/kernel/config/nvmet/ports/1 rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory + for subsys in /sys/kernel/config/nvmet/subsystems/* + for ns in ${subsys}/namespaces/* + echo 0 ./delete.sh: line 14: /sys/kernel/config/nvmet/subsystems/*/namespaces/*/enable: No such file or directory + rmdir '/sys/kernel/config/nvmet/subsystems/*/namespaces/*' rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*/namespaces/*': No such file or directory + rmdir '/sys/kernel/config/nvmet/subsystems/*' rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*': No such file or directory + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files + modprobe -r nvme-fabrics + modprobe -r nvme_loop + modprobe -r nvmet + modprobe -r nvme + sleep 1 + modprobe -r nvme-core + lsmod + grep nvme + sleep 1 + git diff + sleep 1 ++ nproc + make -j 48 M=drivers/nvme/target/ clean ++ nproc + make -j 48 M=drivers/nvme/ modules CC [M] drivers/nvme/target/core.o CC [M] drivers/nvme/target/configfs.o CC [M] drivers/nvme/target/admin-cmd.o CC [M] drivers/nvme/target/fabrics-cmd.o CC [M] drivers/nvme/target/discovery.o CC [M] drivers/nvme/target/io-cmd-file.o CC [M] drivers/nvme/target/io-cmd-bdev.o CC [M] drivers/nvme/target/passthru.o CC [M] drivers/nvme/target/zns.o CC [M] drivers/nvme/target/trace.o CC [M] drivers/nvme/target/loop.o CC [M] drivers/nvme/target/rdma.o CC [M] drivers/nvme/target/fc.o CC [M] drivers/nvme/target/fcloop.o CC [M] drivers/nvme/target/tcp.o LD [M] drivers/nvme/target/nvme-loop.o LD [M] drivers/nvme/target/nvme-fcloop.o LD [M] drivers/nvme/target/nvmet.o LD [M] drivers/nvme/target/nvmet-fc.o LD [M] drivers/nvme/target/nvmet-tcp.o LD [M] drivers/nvme/target/nvmet-rdma.o MODPOST drivers/nvme/Module.symvers CC [M] drivers/nvme/target/nvme-fcloop.mod.o CC [M] drivers/nvme/target/nvme-loop.mod.o CC [M] drivers/nvme/target/nvmet-fc.mod.o CC [M] drivers/nvme/target/nvmet-rdma.mod.o CC [M] drivers/nvme/target/nvmet-tcp.mod.o CC [M] drivers/nvme/target/nvmet.mod.o LD [M] drivers/nvme/target/nvmet-tcp.ko LD [M] drivers/nvme/target/nvme-loop.ko LD [M] drivers/nvme/target/nvmet-rdma.ko LD [M] drivers/nvme/target/nvmet-fc.ko LD [M] drivers/nvme/target/nvme-fcloop.ko LD [M] drivers/nvme/target/nvmet.ko + HOST=drivers/nvme/host + TARGET=drivers/nvme/target ++ uname -r + HOST_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ ++ uname -r + TARGET_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target/ + cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host// + cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target// + ls -lrth /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target// /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/: total 6.3M -rw-r--r--. 1 root root 2.7M Apr 9 15:46 nvme-core.ko -rw-r--r--. 1 root root 426K Apr 9 15:46 nvme-fabrics.ko -rw-r--r--. 1 root root 925K Apr 9 15:46 nvme-fc.ko -rw-r--r--. 1 root root 714K Apr 9 15:46 nvme.ko -rw-r--r--. 1 root root 856K Apr 9 15:46 nvme-rdma.ko -rw-r--r--. 1 root root 799K Apr 9 15:46 nvme-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//: total 6.3M -rw-r--r--. 1 root root 475K Apr 9 15:46 nvme-fcloop.ko -rw-r--r--. 1 root root 419K Apr 9 15:46 nvme-loop.ko -rw-r--r--. 1 root root 734K Apr 9 15:46 nvmet-fc.ko -rw-r--r--. 1 root root 3.2M Apr 9 15:46 nvmet.ko -rw-r--r--. 1 root root 822K Apr 9 15:46 nvmet-rdma.ko -rw-r--r--. 1 root root 671K Apr 9 15:46 nvmet-tcp.ko + sync + sync + sync + modprobe nvme + echo 'Press enter to continue ...' Press enter to continue ... + read next 1.1 nvme-admin-passthru error injection logging error mesaages :- ------------------------------------------------------------------------ nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # sh error_inject.sh ++ NN=1 ++ NQN=testnqn ++ let NR_DEVICES=NN+1 ++ modprobe -r null_blk ++ modprobe -r nvme ++ modprobe null_blk nr_devices=0 ++ modprobe nvme ++ modprobe nvme-fabrics ++ modprobe nvmet ++ modprobe nvme-loop ++ dmesg -c ++ sleep 2 ++ tree /sys/kernel/config /sys/kernel/config ├── nullb │   └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file ++ sleep 1 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn +++ shuf -i 1-1 -n 1 ++ for i in `shuf -i 1-$NN -n $NN` ++ mkdir config/nullb/nullb1 ++ echo 1 ++ echo 4096 ++ echo 2048 ++ echo 1 +++ cat config/nullb/nullb1/index ++ IDX=0 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ echo ' ####### /dev/nullb0' ####### /dev/nullb0 ++ echo -n /dev/nullb0 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb0 ++ echo 1 ++ dmesg -c [ 735.981707] nvmet: adding nsid 1 to subsystem testnqn ++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop ++ echo -n 1 ++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ echo transport=loop,nqn=testnqn ++ sleep 1 ++ mount ++ column -t ++ grep nvme ++ dmesg -c [ 735.990847] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:8a582f37-906c-4e38-b4f1-63da52eb618c. [ 735.990963] nvme nvme1: creating 48 I/O queues. [ 735.994732] nvme nvme1: new ctrl: "testnqn" Node SN Model Namespace Usage Format FW Rev nvme1n1 7eb450b4757e46995c7e Linux 1 2.15 GB / 2.15 GB 4 KiB + 0 B 5.17.0-r nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0 NVMe status: Access Denied: Access to the namespace and/or LBA range is denied due to lack of access rights(0x4286) Node SN Model Namespace Usage Format FW Rev nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0 [ 737.006671] FAULT_INJECTION: forcing a failure. name fault_inject, interval 1, probability 100, space 0, times 1000 [ 737.006679] CPU: 21 PID: 38553 Comm: kworker/21:0 Tainted: G OE 5.17.0-rc2nvme+ #70 [ 737.006683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 737.006686] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 737.006694] Call Trace: [ 737.006713] [ 737.006716] dump_stack_lvl+0x48/0x5e [ 737.006722] should_fail.cold+0x32/0x37 [ 737.006727] nvme_should_fail+0x38/0x90 [nvme_core] [ 737.006739] nvme_loop_queue_response+0xc9/0x143 [nvme_loop] [ 737.006743] nvmet_req_complete+0x11/0x50 [nvmet] [ 737.006752] process_one_work+0x1af/0x380 [ 737.006756] ? rescuer_thread+0x370/0x370 [ 737.006758] worker_thread+0x50/0x3a0 [ 737.006771] ? rescuer_thread+0x370/0x370 [ 737.006773] kthread+0xe7/0x110 [ 737.006776] ? kthread_complete_and_exit+0x20/0x20 [ 737.006780] ret_from_fork+0x22/0x30 [ 737.006786] [ 737.006792] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR [ 737.011040] FAULT_INJECTION: forcing a failure. name fault_inject, interval 1, probability 100, space 0, times 999 [ 737.011047] CPU: 22 PID: 10449 Comm: kworker/22:9 Tainted: G OE 5.17.0-rc2nvme+ #70 [ 737.011051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 737.011053] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 737.011062] Call Trace: [ 737.011065] [ 737.011067] dump_stack_lvl+0x48/0x5e [ 737.011074] should_fail.cold+0x32/0x37 [ 737.011079] nvme_should_fail+0x38/0x90 [nvme_core] [ 737.011091] nvme_loop_queue_response+0xc9/0x143 [nvme_loop] [ 737.011095] nvmet_req_complete+0x11/0x50 [nvmet] [ 737.011103] process_one_work+0x1af/0x380 [ 737.011108] worker_thread+0x50/0x3a0 [ 737.011111] ? rescuer_thread+0x370/0x370 [ 737.011113] kthread+0xe7/0x110 [ 737.011117] ? kthread_complete_and_exit+0x20/0x20 [ 737.011120] ret_from_fork+0x22/0x30 [ 737.011126] [ 737.011133] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.339s user 0m0.001s sys 0m0.004s + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 + for subsys in /sys/kernel/config/nvmet/subsystems/* + for ns in ${subsys}/namespaces/* + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + rmdir config/nullb/nullb1 + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files 1.2 blktests results :- ----------------------------------------------------------------------- nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # cdblktests blktests (master) # ./check nvme nvme/002 (create many subsystems and test discovery) [passed] runtime 17.137s ... 17.312s nvme/003 (test if we're sending keep-alives to a discovery controller) [passed] runtime 10.083s ... 10.089s nvme/004 (test nvme and nvmet UUID NS descriptors) [passed] runtime 1.443s ... 1.433s nvme/005 (reset local loopback target) [passed] runtime 6.770s ... 6.795s nvme/006 (create an NVMeOF target with a block device-backed ns) [passed] runtime 0.070s ... 0.063s nvme/007 (create an NVMeOF target with a file-backed ns) [passed] runtime 0.035s ... 0.042s nvme/008 (create an NVMeOF host with a block device-backed ns) [passed] runtime 1.455s ... 1.457s nvme/009 (create an NVMeOF host with a file-backed ns) [passed] runtime 1.418s ... 1.420s nvme/010 (run data verification fio job on NVMeOF block device-backed ns) [passed] runtime 7.805s ... 7.906s nvme/011 (run data verification fio job on NVMeOF file-backed ns) [passed] runtime 76.991s ... 76.979s nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed] runtime 11.506s ... 12.301s nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [passed] runtime 79.186s ... 80.436s nvme/014 (flush a NVMeOF block device-backed ns) [passed] runtime 8.907s ... 8.850s nvme/015 (unit test for NVMe flush for file backed ns) [passed] runtime 8.685s ... 8.669s nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [passed] runtime 11.425s ... 10.842s nvme/017 (create/delete many file-ns and test discovery) [passed] runtime 11.678s ... 11.080s nvme/018 (unit test NVMe-oF out of range access on a file backend) [passed] runtime 1.416s ... 1.402s nvme/019 (test NVMe DSM Discard command on NVMeOF block-device ns) [passed] runtime 1.441s ... 1.428s nvme/020 (test NVMe DSM Discard command on NVMeOF file-backed ns) [passed] runtime 1.426s ... 1.412s nvme/021 (test NVMe list command on NVMeOF file-backed ns) [passed] runtime 1.417s ... 1.401s nvme/022 (test NVMe reset command on NVMeOF file-backed ns) [passed] runtime 6.770s ... 6.736s nvme/023 (test NVMe smart-log command on NVMeOF block-device ns) [passed] runtime 1.427s ... 1.455s nvme/024 (test NVMe smart-log command on NVMeOF file-backed ns) [passed] runtime 1.406s ... 1.406s nvme/025 (test NVMe effects-log command on NVMeOF file-backed ns) [passed] runtime 1.430s ... 1.410s nvme/026 (test NVMe ns-descs command on NVMeOF file-backed ns) [passed] runtime 1.410s ... 1.406s nvme/027 (test NVMe ns-rescan command on NVMeOF file-backed ns) [passed] runtime 1.419s ... 1.422s nvme/028 (test NVMe list-subsys command on NVMeOF file-backed ns) [passed] runtime 1.412s ... 1.433s nvme/029 (test userspace IO via nvme-cli read/write interface) [passed] runtime 1.564s ... 1.536s nvme/030 (ensure the discovery generation counter is updated appropriately) [passed] runtime 0.186s ... 0.192s nvme/031 (test deletion of NVMeOF controllers immediately after setup) [passed] runtime 53.898s ... 53.842s nvme/038 (test deletion of NVMeOF subsystem without enabling) [passed] runtime 0.018s ... 0.019s 1.3 Test fio verify job with NVMeOF ZBD backend :- ----------------------------------------------------------------------- nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # ./zbdev.sh 1 ++ SUBSYS=/sys/kernel/config/nvmet/subsystems ++ PORT=/sys/kernel/config/nvmet/ports ++ NQN=testnqn ++ NN=1 ++ main ++ unload ++ modprobe -r null_blk ++ modprobe null_blk zoned=1 nr_devices=0 ++ modprobe nvme ++ modprobe nvme-fabrics ++ modprobe nvmet ++ modprobe nvme-loop ++ dmesg -c ++ sleep 2 ++ make_subsys ++ tree /sys/kernel/config /sys/kernel/config ├── nullb │   └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn ++ echo -n 1 ++ make_port ++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop ++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ sleep 1 ++ connect_host ++ echo transport=loop,nqn=testnqn ++ make_ns +++ shuf -i 1-1 -n 1 ++ for i in `shuf -i 1-$NN -n $NN` ++ mkdir config/nullb/nullb1 ++ echo 1 ++ echo 4096 ++ echo 1024 ++ echo 128 ++ echo 1 ++ echo 1 +++ cat config/nullb/nullb1/index ++ IDX=0 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ echo ' ####### /dev/nullb0' ####### /dev/nullb0 ++ echo -n /dev/nullb0 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb0 ++ echo /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/enable /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/enable ++ echo 1 ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 ++ cnt=1 ++ echo 1 1 ++ '[' 1 -gt 1 ']' ++ sleep 1 ++ dmesg -c [ 36.186752] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:7f0546f8-2eab-4195-b673-0a434fa67768. [ 36.186849] nvme nvme1: creating 48 I/O queues. [ 36.190938] nvme nvme1: new ctrl: "testnqn" [ 36.198983] nvmet: adding nsid 1 to subsystem testnqn [ 36.200458] nvme nvme1: rescanning namespaces. ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 /dev/nvme1n1 ++ cnt=2 ++ echo 2 2 ++ '[' 2 -gt 1 ']' ++ break ++ dmesg -c ++ ls -lrth /dev/nvme0 /dev/nvme0n1 /dev/nvme1 /dev/nvme1n1 /dev/nvme-fabrics crw-------. 1 root root 244, 0 Apr 9 16:24 /dev/nvme0 brw-rw----. 1 root disk 259, 0 Apr 9 16:24 /dev/nvme0n1 crw-------. 1 root root 10, 125 Apr 9 16:24 /dev/nvme-fabrics crw-------. 1 root root 244, 1 Apr 9 16:24 /dev/nvme1 brw-rw----. 1 root disk 259, 2 Apr 9 16:24 /dev/nvme1n1 nvme (nvme-5.18) # blkzone report /dev/nvme1n1 start: 0x000000000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] start: 0x000040000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] start: 0x000080000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] start: 0x0000c0000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] start: 0x000100000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] start: 0x000140000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] start: 0x000180000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] start: 0x0001c0000, len 0x040000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)] nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # fio fio/zverify.fio --filename=/dev/nvme1n1 --showcmd fio --name=write-and-verify --rw=randwrite --zonemode=zbd --bs=4k --direct=1 --ioengine=libaio --iodepth=4 --norandommap --randrepeat=1 --verify=crc32c --size=800m --group_reporting nvme (nvme-5.18) # fio fio/zverify.fio --filename=/dev/nvme1n1 write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27 Starting 1 process /dev/nvme1n1: rounded down io_size from 838860800 to 805306368 Jobs: 1 (f=1): [V(1)][100.0%][r=418MiB/s][r=107k IOPS][eta 00m:00s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=3219: Sat Apr 9 16:28:57 2022 read: IOPS=107k, BW=417MiB/s (438MB/s)(768MiB/1840msec) slat (nsec): min=2414, max=66507, avg=5908.64, stdev=3564.30 clat (nsec): min=11421, max=82798, avg=30300.22, stdev=7387.49 lat (nsec): min=17532, max=93117, avg=36300.44, stdev=6939.55 clat percentiles (nsec): | 1.00th=[17024], 5.00th=[18560], 10.00th=[22400], 20.00th=[25216], | 30.00th=[26240], 40.00th=[27264], 50.00th=[28288], 60.00th=[31104], | 70.00th=[33536], 80.00th=[36096], 90.00th=[39680], 95.00th=[44288], | 99.00th=[51456], 99.50th=[55040], 99.90th=[62720], 99.95th=[65280], | 99.99th=[71168] write: IOPS=76.2k, BW=298MiB/s (312MB/s)(768MiB/2579msec); 6 zone resets slat (nsec): min=3557, max=75212, avg=10193.65, stdev=4898.93 clat (usec): min=13, max=2587, avg=39.71, stdev=15.45 lat (usec): min=18, max=2593, avg=50.02, stdev=15.59 clat percentiles (usec): | 1.00th=[ 22], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 39], 60.00th=[ 42], | 70.00th=[ 44], 80.00th=[ 47], 90.00th=[ 53], 95.00th=[ 59], | 99.00th=[ 73], 99.50th=[ 79], 99.90th=[ 95], 99.95th=[ 103], | 99.99th=[ 123] bw ( KiB/s): min=49520, max=320424, per=85.97%, avg=262144.00, stdev=108036.16, samples=6 iops : min=12380, max=80106, avg=65536.00, stdev=27009.04, samples=6 lat (usec) : 20=3.61%, 50=89.00%, 100=7.37%, 250=0.03% lat (msec) : 4=0.01% cpu : usr=21.05%, sys=31.87%, ctx=426449, majf=0, minf=4626 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=196608,196608,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=4 Run status group 0 (all jobs): READ: bw=417MiB/s (438MB/s), 417MiB/s-417MiB/s (438MB/s-438MB/s), io=768MiB (805MB), run=1840-1840msec WRITE: bw=298MiB/s (312MB/s), 298MiB/s-298MiB/s (312MB/s-312MB/s), io=768MiB (805MB), run=2579-2579msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% nvme (nvme-5.18) # nvme (nvme-5.18) # ./delete.sh + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.340s user 0m0.000s sys 0m0.006s + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 + for subsys in /sys/kernel/config/nvmet/subsystems/* + for ns in ${subsys}/namespaces/* + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + rmdir config/nullb/nullb1 + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files nvme (nvme-5.18) # 2. Tests with CONFIG_BLK_DEV_ZONED disabled :- ----------------------------------------------------------------------- nvme (nvme-5.18) # grep -i zoned .config # CONFIG_BLK_DEV_ZONED is not set nvme (nvme-5.18) # git log -3 nvme (nvme-5.18) # git log -3 commit 0d77072e26fa074da0c36e423a5802fd18ef976d (HEAD -> nvme-5.18) Author: Chaitanya Kulkarni Date: Fri Apr 8 16:45:18 2022 -0700 nvme-core: mark internal passthru req REQ_QUIET Mark internal passthru requests quiet in the submission path with RQF_QUIET flag added in the __nvme_submit_sync_cmd(). In the completion path, if nvme request is resulted in the error and request is marked RQF_QUIET then don't log the error with nvme_error_log(). Signed-off-by: Chaitanya Kulkarni commit d4df054957f3cbd37601d65d8e7460aaaee8733d Author: Chaitanya Kulkarni Date: Fri Apr 8 15:50:10 2022 -0700 nvme-core: don't check non-mdts for disc ctrl Don't check the non mdts limits for the discovery controller as we don't support I/O command set, this also masks the following warning reported by the nvme_log_error() when running blktest nvme/002: - [ 2005.155946] run blktests nvme/002 at 2022-04-09 16:57:47 [ 2005.192223] loop: module loaded [ 2005.196429] nvmet: adding nsid 1 to subsystem blktests-subsystem-0 [ 2005.200334] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 <------------------------------SNIP----------------------------------> [ 2008.958108] nvmet: adding nsid 1 to subsystem blktests-subsystem-997 [ 2008.962082] nvmet: adding nsid 1 to subsystem blktests-subsystem-998 [ 2008.966102] nvmet: adding nsid 1 to subsystem blktests-subsystem-999 [ 2008.973132] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN testhostnqn. *[ 2008.973196] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) MORE DNR* [ 2008.974595] nvme nvme1: new ctrl: "nqn.2014-08.org.nvmexpress.discovery" [ 2009.103248] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" Signed-off-by: Chaitanya Kulkarni commit 18aac654ff824a4b169e51d72b674245a7a1b79c Author: Chaitanya Kulkarni Date: Thu Apr 7 12:11:37 2022 -0700 nvmet: handle admin default command set identifier The NVMeOF in kernel host when configuring non-mdts limits uses idnetify controller cns value to NVME_ID_CNS_CS_CTRL and csi value to NVME_CSI_NVM. In target code we only handle case for NVME_ID_CNS_CS_CTRL and NVME_CSI_ZNS when CONFIG_BLK_DEV_ZONED is set. Handle missing case for CONFIG_BLK_DEV_ZONED and !CONFIG_BLK_DEV_ZONED so it can handle NVME_ID_CNS_CS_CTRL with NVME_CSI_NVM. Use this opportunity to add default for the NVME_ID_CNS_CTRL case which makes code uniform for all the cases in the nvmet_execute_identify_cns_cs_ctrl_nvm(). Also, rename nvmet_execute_identify_ctrl() to nvmet_execute_identify_cns_cs_ctrl_nvm() and nvmet_execute_identify_cns_cs_ctrl() to nvmet_execute_identify_cns_cs_ctrl_zns(). This also masks the following warning reported by the nvme_log_error() when running blktest nvme/012:- [ 2131.702140] run blktests nvme/012 at 2022-04-09 16:59:53 [ 2131.722144] loop0: detected capacity change from 0 to 2097152 [ 2131.730586] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 2131.738826] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN testhostnqn. *[ 2131.738911] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) MORE DNR* [ 2131.740540] nvme nvme1: creating 48 I/O queues. [ 2131.743925] nvme nvme1: new ctrl: "blktests-subsystem-1" [ 2132.790058] XFS (nvme1n1): Mounting V5 Filesystem [ 2132.793667] XFS (nvme1n1): Ending clean mount [ 2132.794030] xfs filesystem being mounted at /mnt/blktests supports timestamps until 2038 (0x7fffffff) [ 2142.471812] XFS (nvme1n1): Unmounting Filesystem [ 2142.492566] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1" Signed-off-by: Chaitanya Kulkarni nvme (nvme-5.18) # ./compile_nvme.sh + umount /mnt/nvme0n1 + clear_dmesg ./compile_nvme.sh: line 3: clear_dmesg: command not found umount: /mnt/nvme0n1: no mount point specified. + ./delete.sh + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 0 controller(s) real 0m0.003s user 0m0.000s sys 0m0.002s + rm -fr '/sys/kernel/config/nvmet/ports/1/subsystems/*' + rmdir /sys/kernel/config/nvmet/ports/1 rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory + for subsys in /sys/kernel/config/nvmet/subsystems/* + for ns in ${subsys}/namespaces/* + echo 0 ./delete.sh: line 14: /sys/kernel/config/nvmet/subsystems/*/namespaces/*/enable: No such file or directory + rmdir '/sys/kernel/config/nvmet/subsystems/*/namespaces/*' rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*/namespaces/*': No such file or directory + rmdir '/sys/kernel/config/nvmet/subsystems/*' rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*': No such file or directory + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files + modprobe -r nvme-fabrics + modprobe -r nvme_loop + modprobe -r nvmet + modprobe -r nvme + sleep 1 cdblk+ modprobe -r nvme-core + lsmod + grep nvme + sleep 1 t+ git diff + sleep 1 ++ nproc + make -j 48 M=drivers/nvme/target/ clean ++ nproc + make -j 48 M=drivers/nvme/ modules CC [M] drivers/nvme/target/core.o CC [M] drivers/nvme/target/configfs.o CC [M] drivers/nvme/target/admin-cmd.o CC [M] drivers/nvme/target/fabrics-cmd.o CC [M] drivers/nvme/target/discovery.o CC [M] drivers/nvme/target/io-cmd-file.o CC [M] drivers/nvme/target/io-cmd-bdev.o CC [M] drivers/nvme/target/passthru.o CC [M] drivers/nvme/target/trace.o CC [M] drivers/nvme/target/loop.o CC [M] drivers/nvme/target/rdma.o CC [M] drivers/nvme/target/fc.o CC [M] drivers/nvme/target/fcloop.o CC [M] drivers/nvme/target/tcp.o LD [M] drivers/nvme/target/nvme-loop.o LD [M] drivers/nvme/target/nvme-fcloop.o LD [M] drivers/nvme/target/nvmet.o LD [M] drivers/nvme/target/nvmet-fc.o LD [M] drivers/nvme/target/nvmet-tcp.o LD [M] drivers/nvme/target/nvmet-rdma.o MODPOST drivers/nvme/Module.symvers CC [M] drivers/nvme/target/nvme-fcloop.mod.o CC [M] drivers/nvme/target/nvme-loop.mod.o CC [M] drivers/nvme/target/nvmet-fc.mod.o CC [M] drivers/nvme/target/nvmet-rdma.mod.o CC [M] drivers/nvme/target/nvmet-tcp.mod.o CC [M] drivers/nvme/target/nvmet.mod.o LD [M] drivers/nvme/target/nvmet-fc.ko LD [M] drivers/nvme/target/nvme-fcloop.ko LD [M] drivers/nvme/target/nvme-loop.ko LD [M] drivers/nvme/target/nvmet-tcp.ko LD [M] drivers/nvme/target/nvmet.ko LD [M] drivers/nvme/target/nvmet-rdma.ko + HOST=drivers/nvme/host + TARGET=drivers/nvme/target ++ uname -r + HOST_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ ++ uname -r + TARGET_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target/ + cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host// + cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target// + ls -lrth /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target// /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/: total 6.1M -rw-r--r--. 1 root root 2.4M Apr 9 16:15 nvme-core.ko -rw-r--r--. 1 root root 426K Apr 9 16:15 nvme-fabrics.ko -rw-r--r--. 1 root root 924K Apr 9 16:15 nvme-fc.ko -rw-r--r--. 1 root root 714K Apr 9 16:15 nvme.ko -rw-r--r--. 1 root root 856K Apr 9 16:15 nvme-rdma.ko -rw-r--r--. 1 root root 799K Apr 9 16:15 nvme-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//: total 6.0M -rw-r--r--. 1 root root 475K Apr 9 16:15 nvme-fcloop.ko -rw-r--r--. 1 root root 418K Apr 9 16:15 nvme-loop.ko -rw-r--r--. 1 root root 733K Apr 9 16:15 nvmet-fc.ko -rw-r--r--. 1 root root 3.0M Apr 9 16:15 nvmet.ko -rw-r--r--. 1 root root 822K Apr 9 16:15 nvmet-rdma.ko -rw-r--r--. 1 root root 670K Apr 9 16:15 nvmet-tcp.ko + sync + sync + sync + modprobe nvme + echo 'Press enter to continue ...' Press enter to continue ... + read next 2.1 nvme-admin-passthru error injection logging error mesaages. nvme (nvme-5.18) # vim error_inject.sh nvme (nvme-5.18) # sh error_inject.sh ++ NN=1 ++ NQN=testnqn ++ let NR_DEVICES=NN+1 ++ modprobe -r null_blk ++ modprobe -r nvme ++ modprobe null_blk nr_devices=0 ++ modprobe nvme ++ modprobe nvme-fabrics ++ modprobe nvmet ++ modprobe nvme-loop ++ dmesg -c ++ sleep 2 ++ tree /sys/kernel/config /sys/kernel/config ├── nullb │   └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file ++ sleep 1 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn +++ shuf -i 1-1 -n 1 ++ for i in `shuf -i 1-$NN -n $NN` ++ mkdir config/nullb/nullb1 ++ echo 1 ++ echo 4096 ++ echo 2048 ++ echo 1 +++ cat config/nullb/nullb1/index ++ IDX=0 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ echo ' ####### /dev/nullb0' ####### /dev/nullb0 ++ echo -n /dev/nullb0 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb0 ++ echo 1 ++ dmesg -c [ 413.375009] nvme nvme0: 48/0/0 default/read/poll queues [ 416.369926] nvmet: adding nsid 1 to subsystem testnqn ++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop ++ echo -n 1 ++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ echo transport=loop,nqn=testnqn ++ sleep 1 ++ mount ++ grep nvme ++ column -t ++ dmesg -c [ 416.376943] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:6b4a42a3-6ddc-44fc-bcfa-0cab47623a1f. [ 416.377040] nvme nvme1: creating 48 I/O queues. [ 416.380887] nvme nvme1: new ctrl: "testnqn" Node SN Model Namespace Usage Format FW Rev nvme1n1 7c695f8c7cfff9e62d0f Linux 1 2.15 GB / 2.15 GB 4 KiB + 0 B 5.17.0-r nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0 NVMe status: Access Denied: Access to the namespace and/or LBA range is denied due to lack of access rights(0x4286) Node SN Model Namespace Usage Format FW Rev nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0 [ 417.393147] FAULT_INJECTION: forcing a failure. name fault_inject, interval 1, probability 100, space 0, times 1000 [ 417.393155] CPU: 25 PID: 10499 Comm: kworker/25:9 Tainted: G OE 5.17.0-rc2nvme+ #71 [ 417.393159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 417.393161] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 417.393170] Call Trace: [ 417.393188] [ 417.393190] dump_stack_lvl+0x48/0x5e [ 417.393196] should_fail.cold+0x32/0x37 [ 417.393201] nvme_should_fail+0x38/0x90 [nvme_core] [ 417.393213] nvme_loop_queue_response+0xc9/0x143 [nvme_loop] [ 417.393217] nvmet_req_complete+0x11/0x50 [nvmet] [ 417.393225] process_one_work+0x1af/0x380 [ 417.393229] worker_thread+0x50/0x3a0 [ 417.393232] ? rescuer_thread+0x370/0x370 [ 417.393234] kthread+0xe7/0x110 [ 417.393238] ? kthread_complete_and_exit+0x20/0x20 [ 417.393241] ret_from_fork+0x22/0x30 [ 417.393247] [ 417.393254] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR [ 417.397215] FAULT_INJECTION: forcing a failure. name fault_inject, interval 1, probability 100, space 0, times 999 [ 417.397221] CPU: 26 PID: 10947 Comm: kworker/26:7 Tainted: G OE 5.17.0-rc2nvme+ #71 [ 417.397225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 417.397227] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 417.397235] Call Trace: [ 417.397238] [ 417.397241] dump_stack_lvl+0x48/0x5e [ 417.397247] should_fail.cold+0x32/0x37 [ 417.397252] nvme_should_fail+0x38/0x90 [nvme_core] [ 417.397264] nvme_loop_queue_response+0xc9/0x143 [nvme_loop] [ 417.397268] nvmet_req_complete+0x11/0x50 [nvmet] [ 417.397276] process_one_work+0x1af/0x380 [ 417.397281] worker_thread+0x50/0x3a0 [ 417.397283] ? rescuer_thread+0x370/0x370 [ 417.397296] kthread+0xe7/0x110 [ 417.397300] ? kthread_complete_and_exit+0x20/0x20 [ 417.397303] ret_from_fork+0x22/0x30 [ 417.397310] [ 417.397316] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.357s user 0m0.000s sys 0m0.006s + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 + for subsys in /sys/kernel/config/nvmet/subsystems/* + for ns in ${subsys}/namespaces/* + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + rmdir config/nullb/nullb1 + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files nvme (nvme-5.18) # nvme (nvme-5.18) # 2.2 blktests results. ----------------------------------------------------------------------- nvme (nvme-5.18) # cdblktests blktests (master) # ./check nvme nvme/002 (create many subsystems and test discovery) [passed] runtime 17.312s ... 16.744s nvme/003 (test if we're sending keep-alives to a discovery controller) [passed] runtime 10.089s ... 10.091s nvme/0nvme/004 (test nvme and nvmet UUID NS descriptors) [passed] runtime 1.433s ... 1.445s nvme/005 (reset local loopback target) [passed] runtime 6.795s ... 6.802s nvme/006 (create an NVMeOF target with a block device-backed ns) [passed] runtime 0.063s ... 0.089s nvme/007 (create an NVMeOF target with a file-backed ns) [passed] runtime 0.042s ... 0.040s nvme/008 (create an NVMeOF host with a block device-backed ns) [passed] runtime 1.457s ... 1.451s nvme/009 (create an NVMeOF host with a file-backed ns) [passed] runtime 1.420s ... 1.423s nvme/010 (run data verification fio job on NVMeOF block device-backed ns) [passed] runtime 7.906s ... 7.961s nvme/011 (run data verification fio job on NVMeOF file-backed ns) [passed] runtime 76.979s ... 75.022s nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed] runtime 12.301s ... 11.106s nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [passed] runtime 80.436s ... 77.184s nvme/014 (flush a NVMeOF block device-backed ns) [passed] runtime 8.850s ... 8.883s nvme/015 (unit test for NVMe flush for file backed ns) [passed] runtime 8.669s ... 8.710s nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [passed] runtime 10.842s ... 11.078s nvme/017 (create/delete many file-ns and test discovery) [passed] runtime 11.080s ... 11.047s nvme/018 (unit test NVMe-oF out of range access on a file backend) [passed] runtime 1.402s ... 1.440s nvme/019 (test NVMe DSM Discard command on NVMeOF block-device ns) [passed] runtime 1.428s ... 1.467s nvme/020 (test NVMe DSM Discard command on NVMeOF file-backed ns) [passed] runtime 1.412s ... 1.419s nvme/021 (test NVMe list command on NVMeOF file-backed ns) [passed] runtime 1.401s ... 1.405s nvme/022 (test NVMe reset command on NVMeOF file-backed ns) [passed] runtime 6.736s ... 6.733s nvme/023 (test NVMe smart-log command on NVMeOF block-device ns) [passed] runtime 1.455s ... 1.437s nvme/024 (test NVMe smart-log command on NVMeOF file-backed ns) [passed] runtime 1.406s ... 1.407s nvme/025 (test NVMe effects-log command on NVMeOF file-backed ns) [passed] runtime 1.410s ... 1.430s nvme/026 (test NVMe ns-descs command on NVMeOF file-backed ns) [passed] runtime 1.406s ... 1.410s nvme/027 (test NVMe ns-rescan command on NVMeOF file-backed ns) [passed] runtime 1.422s ... 1.436s nvme/028 (test NVMe list-subsys command on NVMeOF file-backed ns) [passed] runtime 1.433s ... 1.405s nvme/029 (test userspace IO via nvme-cli read/write interface) [passed] runtime 1.536s ... 1.544s nvme/030 (ensure the discovery generation counter is updated appropriately) [passed] runtime 0.192s ... 0.183s nvme/031 (test deletion of NVMeOF controllers immediately after setup) [passed] runtime 53.842s ... 53.901s nvme/038 (test deletion of NVMeOF subsystem without enabling) [passed] runtime 0.019s ... 0.018s blktests (master) # blktests (master) # nvme (nvme-5.18) # nvme (nvme-5.18) # nvme (nvme-5.18) # 2.3 Try and create NVMeOF controller with ZBD backend fail grececully. ----------------------------------------------------------------------- nvme (nvme-5.18) # nvme (nvme-5.18) # ./zbdev.sh 1 ++ SUBSYS=/sys/kernel/config/nvmet/subsystems ++ PORT=/sys/kernel/config/nvmet/ports ++ NQN=testnqn ++ NN=1 ++ main ++ unload ++ modprobe -r null_blk ++ modprobe null_blk zoned=1 nr_devices=0 ++ modprobe nvme ++ modprobe nvme-fabrics ++ modprobe nvmet ++ modprobe nvme-loop ++ dmesg -c ++ sleep 2 ++ make_subsys ++ tree /sys/kernel/config /sys/kernel/config ├── nullb │   └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn ++ echo -n 1 ++ make_port ++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop ++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ sleep 1 ++ connect_host ++ echo transport=loop,nqn=testnqn ++ make_ns +++ shuf -i 1-1 -n 1 ++ for i in `shuf -i 1-$NN -n $NN` ++ mkdir config/nullb/nullb1 ++ echo 1 ++ echo 4096 ++ echo 1024 ++ echo 128 ++ echo 1 ++ echo 1 ./zbdev.sh: line 39: echo: write error: Invalid argument +++ cat config/nullb/nullb1/index ++ IDX=0 ++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ echo ' ####### /dev/nullb0' ####### /dev/nullb0 ++ echo -n /dev/nullb0 ++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb0 ++ echo /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/enable /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/enable ++ echo 1 ./zbdev.sh: line 49: echo: write error: No such file or directory ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 ++ cnt=1 ++ echo 1 1 ++ '[' 1 -gt 1 ']' ++ sleep 1 ++ dmesg -c [ 424.749844] nvme nvme0: 48/0/0 default/read/poll queues [ 427.740698] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:477f34ee-8fbe-47c2-858e-0312fd3b85b5. [ 427.740788] nvme nvme1: creating 48 I/O queues. [ 427.745480] nvme nvme1: new ctrl: "testnqn" [ 427.748984] null_blk: CONFIG_BLK_DEV_ZONED not enabled [ 427.752564] nvmet: adding nsid 1 to subsystem testnqn [ 427.754146] nvmet: failed to open block device /dev/nullb0: (-2) ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 ++ cnt=1 ++ echo 1 1 ++ '[' 1 -gt 1 ']' ++ sleep 1 ++ dmesg -c ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 ++ cnt=1 ++ echo 1 1 ++ '[' 1 -gt 1 ']' ++ sleep 1 ++ dmesg -c ++ '[' 1 ']' +++ wc -l +++ ls -l /dev/nvme1 ++ cnt=1 ++ echo 1 1 ++ '[' 1 -gt 1 ']' ++ sleep 1 ^Cnvme (nvme-5.18) # ./delete.sh + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m1.041s user 0m0.000s sys 0m0.004s + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 + for subsys in /sys/kernel/config/nvmet/subsystems/* + for ns in ${subsys}/namespaces/* + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + rmdir config/nullb/nullb1 + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files nvme (nvme-5.18) # -- 2.29.0