nvme native multipath question with two physical paths

nvme native multipath question with two physical paths

[Wen Xiong]

Hi All,

In our nvme native multipath support drawer, each nvme has two physical 
paths/two controllers.
We created a shared namespace on it.

# nvme list-ns /dev/nvme2
[   0]:0x1
# nvme list-ns /dev/nvme6
[   0]:0x1

# ls -l /dev/nvme2*
crw-------. 1 root root 240, 2 Feb 18 18:05 /dev/nvme2
brw-rw----. 1 root disk 259, 4 Feb 18 18:14 /dev/nvme2n1
# ls -l /dev/nvme6*
crw-------. 1 root root 240, 6 Feb 18 18:05 /dev/nvme6

nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:
\
+- nvme2 pcie 052f:80:00.0 live
+- nvme6 pcie 058f:80:00.0 live

#nvme list
/dev/nvme2n1          /dev/ng2n1


Run stress io test over nvme2n1, we hot unplug nvme2(052f:80:00), 
/dev/nvme2 controller is gone.
/dev/nvme2n1 still existed and io still running over nvme2n1 without 
error. Looks io test still
running over another path(nvme6)bbut we didn’t see /dev/nvme6n1 in /dev 
dir.

# ls -l /dev/nvme2*
brw-rw----. 1 root disk 259, 4 Feb 18 18:22 /dev/nvme2n1
# ls -l /dev/nvme6*
crw-------. 1 root root 240, 6 Feb 18 18:05 /dev/nvme6

With nvme native mpath enable,
- What multipath disk name we should use for ANA nvme multipath?
- Why IO still running over nvme2n1 without error even nvme2 controller 
already unplug?
- Shared namespace is still existed since shared namespace  has another 
controller on it?
- How can let nvme6n1 showing up after nvme2 controller gone.

Thanks for your help!
Wendy

Re: nvme native multipath question with two physical paths

[Keith Busch]

On Mon, Feb 19, 2024 at 01:03:48PM -0600, Wen Xiong wrote:
> Hi All,
> 
> In our nvme native multipath support drawer, each nvme has two physical
> paths/two controllers.
> We created a shared namespace on it.
> 
> # nvme list-ns /dev/nvme2
> [   0]:0x1
> # nvme list-ns /dev/nvme6
> [   0]:0x1
> 
> # ls -l /dev/nvme2*
> crw-------. 1 root root 240, 2 Feb 18 18:05 /dev/nvme2
> brw-rw----. 1 root disk 259, 4 Feb 18 18:14 /dev/nvme2n1
> # ls -l /dev/nvme6*
> crw-------. 1 root root 240, 6 Feb 18 18:05 /dev/nvme6
> 
> nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:
> \
> +- nvme2 pcie 052f:80:00.0 live
> +- nvme6 pcie 058f:80:00.0 live
> 
> #nvme list
> /dev/nvme2n1          /dev/ng2n1
> 
> 
> Run stress io test over nvme2n1, we hot unplug nvme2(052f:80:00), /dev/nvme2
> controller is gone.
> /dev/nvme2n1 still existed and io still running over nvme2n1 without error.
> Looks io test still
> running over another path(nvme6)bbut we didn´t see /dev/nvme6n1 in /dev dir.

Nor should you see a /dev/nvme6n1 in this scenario. The mulitpath device
doesn't change names. It assumes the identifier from the subsystem. In
your example, it uses subsystem '2' and will continue to use it no
matter which controller instances are attach to it. This feature is why
applications that have opened the /dev/nvme2n1 handle  don't need to do
anything if the kernel experiences a path failover condition.

You should see the different paths in the hidden block hierarchy, which
you can find in the sysfs directory. There should be an entry in your
example at /sys/block/nvme2c6n1/. I am pretty sure  programs like
'iostat' can show how IO is distributed among all the paths of an nvme
mulitpath device.

Re: nvme native multipath question with two physical paths

[Wen Xiong]

On 2024-02-19 13:28, Keith Busch wrote:

> Nor should you see a /dev/nvme6n1 in this scenario. The mulitpath 
> device
> doesn't change names. It assumes the identifier from the subsystem. In
> your example, it uses subsystem '2' and will continue to use it no
> matter which controller instances are attach to it. This feature is why
> applications that have opened the /dev/nvme2n1 handle  don't need to do
> anything if the kernel experiences a path failover condition.
> 
> You should see the different paths in the hidden block hierarchy, which
> you can find in the sysfs directory. There should be an entry in your
> example at /sys/block/nvme2c6n1/. I am pretty sure  programs like
> 'iostat' can show how IO is distributed among all the paths of an nvme
> mulitpath device.

Hi Ketith,

Thanks for your detail explanation!
Wendy