Re: Saw I/O errors while delete/create/attach a namespace on nvme device.

[Wen Xiong <wenxiong@linux.ibm.com>]

On 2023-11-07 09:18, Keith Busch wrote:

Hi Keith,

> "rescanning namespaces" message right after the 'detach-ns' command, 
> and
> before subsequent 'attach-ns' command. It looks here that the rescan
> didn't run until after the 'attach-ns' occured. Instead of tearing down
> the original, the driver just sees the namespace it previously knew
> about has changed unexpectedly; the processing for the namespace 
> removal
> didn't happen prior to the attach-ns command.

Re-did:
# nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81
detach-ns: Success, nsid:1
# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces
# nvme delete-ns /dev/nvme0 --namespace-id=1
delete-ns: Success, deleted nsid:1
# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces.
# nvme create-ns /dev/nvme0 --nsze=562805846 --ncap=562805846 --flbas=0 
-dps=0 -nmic=1
create-ns: Success, created nsid:1
[root@ltcrain119-lp4 ~]# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces.


> If you drop all open references to /dev/nvme0n1, then the handle should
> get deleted, and a manual rescan after that should get your new
> namespace visible.

# nvme attach-ns /dev/nvme0 -n 1  --controller=0x81
attach-ns: Success, nsid:1
# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces.
[ 5219.493625] nvme nvme0: rescanning namespaces.
[ 5219.502136] nguid mismatch
[ 5219.502146] nvme nvme0: identifiers changed for nsid 1
[ 5219.506668] block nvme0n1: no usable path - requeuing I/O
[ 5219.662788] block nvme0n1: no available path - failing I/O
[ 5219.662824] block nvme0n1: no available path - failing I/O
[ 5219.662841] Buffer I/O error on dev nvme0n1, logical block 281402912, 
async page read
[ 5219.662859] block nvme0n1: no available path - failing I/O
[ 5219.662875] Buffer I/O error on dev nvme0n1, logical block 281402913, 
async page read
[ 5219.662887] block nvme0n1: no available path - failing I/O
[ 5219.662894] Buffer I/O error on dev nvme0n1, logical block 281402914, 
async page read
[ 5219.662913] block nvme0n1: no available path - failing I/O
[ 5219.662926] Buffer I/O error on dev nvme0n1, logical block 281402915, 
async page read
[ 5219.662956] block nvme0n1: no available path - failing I/O
[ 5219.662970] Buffer I/O error on dev nvme0n1, logical block 281402916, 
async page read
[ 5219.662985] bio_check_eod: 7 callbacks suppressed
[ 5219.662988] systemd-udevd: attempt to access beyond end of device
                nvme0n1: rw=0, sector=4502446672, nr_sectors = 16 limit=0
[ 5219.663022] Buffer I/O error on dev nvme0n1, logical block 281402917, 
async page read
[ 5219.663035] systemd-udevd: attempt to access beyond end of device
                nvme0n1: rw=0, sector=4502446688, nr_sectors = 16 limit=0
[ 5219.663052] Buffer I/O error on dev nvme0n1, logical block 281402918, 
async page read
[ 5219.663065] systemd-udevd: attempt to access beyond end of device
                nvme0n1: rw=0, sector=4502446704, nr_sectors = 16 limit=0
[ 5219.663099] Buffer I/O error on dev nvme0n1, logical block 281402919, 
async page read
# nvme ns-rescan /dev/nvme0n1
/dev/nvme0n1: No such file or directory
Usage: nvme ns-rescan <device> [OPTIONS]

Rescans the NVMe namespaces

# nvme ns-rescan /dev/nvme0n1
/dev/nvme0n1: No such file or directory
Usage: nvme ns-rescan <device> [OPTIONS]

Rescans the NVMe namespaces

# ls -l /dev/nvme*
crw-------. 1 root root 240, 0 Nov  7 08:26 /dev/nvme0
crw-------. 1 root root 240, 1 Nov  7 08:13 /dev/nvme1
brw-rw----. 1 root disk 259, 1 Nov  7 08:13 /dev/nvme1n1

[root@ltcrain119-lp4 ~]# nvme attach-ns /dev/nvme0 -n 1  
--controller=0x81
NVMe status: Namespace Already Attached: The controller is already 
attached to the namespace specified(0x2118)

[root@ltcrain119-lp4 ~]# ls -l /dev/nvme*
crw-------. 1 root root 240, 0 Nov  7 08:26 /dev/nvme0
brw-rw----. 1 root disk 259, 3 Nov  7 09:48 /dev/nvme0n1
crw-------. 1 root root 240, 1 Nov  7 08:13 /dev/nvme1
brw-rw----. 1 root disk 259, 1 Nov  7 08:13 /dev/nvme1n1

After attach-ns command, /dev/nvme0n1 is not showed up in /dev/*, 
somehow I have to do the 2nd attach-ns command, nvme ns-rescan works 
after the 2nd attach-ns.

Is a firmware issue on nvme device?

Thanks,
Wendy