Re: nvme native multipath question with two physical paths

[Keith Busch <kbusch@kernel.org>]

On Mon, Feb 19, 2024 at 01:03:48PM -0600, Wen Xiong wrote:
> Hi All,
> 
> In our nvme native multipath support drawer, each nvme has two physical
> paths/two controllers.
> We created a shared namespace on it.
> 
> # nvme list-ns /dev/nvme2
> [   0]:0x1
> # nvme list-ns /dev/nvme6
> [   0]:0x1
> 
> # ls -l /dev/nvme2*
> crw-------. 1 root root 240, 2 Feb 18 18:05 /dev/nvme2
> brw-rw----. 1 root disk 259, 4 Feb 18 18:14 /dev/nvme2n1
> # ls -l /dev/nvme6*
> crw-------. 1 root root 240, 6 Feb 18 18:05 /dev/nvme6
> 
> nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:
> \
> +- nvme2 pcie 052f:80:00.0 live
> +- nvme6 pcie 058f:80:00.0 live
> 
> #nvme list
> /dev/nvme2n1          /dev/ng2n1
> 
> 
> Run stress io test over nvme2n1, we hot unplug nvme2(052f:80:00), /dev/nvme2
> controller is gone.
> /dev/nvme2n1 still existed and io still running over nvme2n1 without error.
> Looks io test still
> running over another path(nvme6)bbut we didn´t see /dev/nvme6n1 in /dev dir.

Nor should you see a /dev/nvme6n1 in this scenario. The mulitpath device
doesn't change names. It assumes the identifier from the subsystem. In
your example, it uses subsystem '2' and will continue to use it no
matter which controller instances are attach to it. This feature is why
applications that have opened the /dev/nvme2n1 handle  don't need to do
anything if the kernel experiences a path failover condition.

You should see the different paths in the hidden block hierarchy, which
you can find in the sysfs directory. There should be an entry in your
example at /sys/block/nvme2c6n1/. I am pretty sure  programs like
'iostat' can show how IO is distributed among all the paths of an nvme
mulitpath device.