[PATCH V2] nvme: Add module reference counting for multipath devices

[PATCH V2] nvme: Add module reference counting for multipath devices

[wenxiong]

From: Wen Xiong <wenxiong@linux.ibm.com>

Add proper module reference counting to prevent premature unloading of
NVMe transport modules while multipath namespaces are still active.

When a namespace is added to a multipath device via nvme_mpath_add_disk(),
the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain
loaded as long as the multipath device references that namespace. Without
proper reference counting, the transport module could be unloaded while
the multipath device is still using resources from that module, leading
to the potential system crashes.

This ensures the transport module remains loaded for the entire lifetime
of the multipath namespace association.

Signed-off-by: Wen Xiong <wenxiong@linux.ibm.com>
---
 drivers/nvme/host/multipath.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 9b9a657fa330..707b8f95727d 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -1348,6 +1348,8 @@ void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns)
 	sysfs_remove_link_from_group(kobj, nvme_ns_mpath_attr_group.name,
 			dev_name(target));
 	clear_bit(NVME_NS_SYSFS_ATTR_LINK, &ns->flags);
+
+	module_put(ns->ctrl->ops->module);
 }
 
 void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
@@ -1379,6 +1381,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
 	if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
 		ns->head->disk->nr_zones = ns->disk->nr_zones;
 #endif
+	if (!try_module_get(ns->ctrl->ops->module))
+		dev_err(disk_to_dev(ns->disk),
+				"Failed to get module reference\n");
 }
 
 void nvme_mpath_remove_disk(struct nvme_ns_head *head)
-- 
2.52.0

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Nares Bannoth]

On 13/08/26 4:02 am, wenxiong@linux.ibm.com wrote:
> From: Wen Xiong <wenxiong@linux.ibm.com>
>
> Add proper module reference counting to prevent premature unloading of
> NVMe transport modules while multipath namespaces are still active.
>
> When a namespace is added to a multipath device via nvme_mpath_add_disk(),
> the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain
> loaded as long as the multipath device references that namespace. Without
> proper reference counting, the transport module could be unloaded while
> the multipath device is still using resources from that module, leading
> to the potential system crashes.
>
> This ensures the transport module remains loaded for the entire lifetime
> of the multipath namespace association.
>
> Signed-off-by: Wen Xiong <wenxiong@linux.ibm.com>
> ---

Hi Wen,

I tested this patch and it is working as expected.

Patch Validation Results
--------------------------------
# lsmod | grep -i nvme
nvme_tcp              262144  0
nvme                  262144  9
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  8 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core
# rmmod nvme
rmmod: ERROR: Module nvme is in use.
#
# rmmod nvme_tcp
#
# lsmod | grep -i nvme
nvme                  262144  9
nvme_fabrics          262144  0
nvme_core             458752  7 nvme,nvme_fabrics
nvme_keyring          262144  2 nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core
# rmmod nvme_core
rmmod: ERROR: Module nvme_core is in use by: nvme nvme_fabrics
#
# rmmod nvme_fabrics
#
# rmmod nvme_core
rmmod: ERROR: Module nvme_core is in use by: nvme
#
# lsmod | grep -i nvme
nvme                  262144  9
nvme_core             458752  6 nvme
nvme_keyring          262144  1 nvme_core
nvme_auth             262144  1 nvme_core
#

Please add below tag :

Tested-by: Naresh Bannoth <nbannoth@linux.ibm.com>


Thanks and Regards,

Naresh.

>   drivers/nvme/host/multipath.c | 5 +++++
>   1 file changed, 5 insertions(+)
>
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 9b9a657fa330..707b8f95727d 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -1348,6 +1348,8 @@ void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns)
>   	sysfs_remove_link_from_group(kobj, nvme_ns_mpath_attr_group.name,
>   			dev_name(target));
>   	clear_bit(NVME_NS_SYSFS_ATTR_LINK, &ns->flags);
> +
> +	module_put(ns->ctrl->ops->module);
>   }
>   
>   void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
> @@ -1379,6 +1381,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
>   	if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
>   		ns->head->disk->nr_zones = ns->disk->nr_zones;
>   #endif
> +	if (!try_module_get(ns->ctrl->ops->module))
> +		dev_err(disk_to_dev(ns->disk),
> +				"Failed to get module reference\n");
>   }
>   
>   void nvme_mpath_remove_disk(struct nvme_ns_head *head)

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Sagi Grimberg]

On 13/08/2026 1:32, wenxiong@linux.ibm.com wrote:
> From: Wen Xiong <wenxiong@linux.ibm.com>
>
> Add proper module reference counting to prevent premature unloading of
> NVMe transport modules while multipath namespaces are still active.
>
> When a namespace is added to a multipath device via nvme_mpath_add_disk(),
> the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain
> loaded as long as the multipath device references that namespace. Without
> proper reference counting, the transport module could be unloaded while
> the multipath device is still using resources from that module, leading
> to the potential system crashes.
>
> This ensures the transport module remains loaded for the entire lifetime
> of the multipath namespace association.

Can you provide details about the issue that this patch is supposed to 
address?
(e.g. potential system crashes). The driver unload should be able to 
disconnect all of the connected controllers when tearing down...

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Sagi Grimberg]

On 23/08/2026 3:20, Sagi Grimberg wrote:
>
>
> On 13/08/2026 1:32, wenxiong@linux.ibm.com wrote:
>> From: Wen Xiong <wenxiong@linux.ibm.com>
>>
>> Add proper module reference counting to prevent premature unloading of
>> NVMe transport modules while multipath namespaces are still active.
>>
>> When a namespace is added to a multipath device via 
>> nvme_mpath_add_disk(),
>> the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain
>> loaded as long as the multipath device references that namespace. 
>> Without
>> proper reference counting, the transport module could be unloaded while
>> the multipath device is still using resources from that module, leading
>> to the potential system crashes.
>>
>> This ensures the transport module remains loaded for the entire lifetime
>> of the multipath namespace association.
>
> Can you provide details about the issue that this patch is supposed to 
> address?
> (e.g. potential system crashes). The driver unload should be able to 
> disconnect all of the connected controllers when tearing down...

OK, I now read the original report regarding nvme root device. please 
disregard.

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Wen Xiong]

On 2026-08-22 19:46, Sagi Grimberg wrote:

> OK, I now read the original report regarding nvme root device. please 
> disregard.
Hi Sagi,

Thanks for taking the time to review this!

If multipath NVMe is the root device and there is no single NVMe device 
in the system, the tester can still rmmod nvme because the NVMe module's 
reference count is 0 (PCIe).

For example,
System has 2 multipath nvme devices(nvme0 has 1 NS and nvme1 has 2 NS).
# nvme list
Node                  Generic               SN                   Model   
                                  Namespace  Usage                      
Format           FW Rev
--------------------- --------------------- -------------------- 
---------------------------------------- ---------- 
-------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            S6RUNE0R900042       1.6TB 
NVMe Gen4 U.2 SSD III              0x1        400.08  GB / 400.08  GB    
   4 KiB +  0 B   REV.SN66
/dev/nvme1n1          /dev/ng1n1            S6KZNE0RA00056       800GB 
NVMe Gen4 U.2 SSD                  0x1        400.08  GB / 400.08  GB    
   4 KiB +  0 B   REV.SN4B
/dev/nvme1n2          /dev/ng1n2            S6KZNE0RA00056       800GB 
NVMe Gen4 U.2 SSD

Without patch, reference count of nvme module is 0, "rmmod nvme" causes 
system crashing.
#lsmod|grep nvme
  nvme_tcp              262144  0
nvme                  262144  0
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

static int nvme_ns_open(struct nvme_ns *ns)
{

         /* should never be called due to GENHD_FL_HIDDEN */
         if (WARN_ON_ONCE(nvme_ns_head_multipath(ns->head))) ---------> 
skip multipath devs
                 goto fail;
         if (!nvme_get_ns(ns))
                 goto fail;
         if (!try_module_get(ns->ctrl->ops->module))
                 goto fail_put_ns;

}

#rmmod nvme
[  434.859171] block nvme2n1: no available path - failing I/O
[  434.859204] XFS (nvme2n1p2): log I/O error -5
[  434.859211] XFS (nvme2n1p2): Filesystem has been shut down due to log 
error (0x2).
[  434.859215] XFS (nvme2n1p2): Please unmount the filesystem and 
rectify the problem(s).
[root@ltcrain119-lp4 nvme_upstream]# [  439.349490] XFS (dm-0): metadata 
I/O error in "xfs_imap_to_bp+0x74/0x108 [xfs]" at daddr 0x73d4ca0 len 32 
error 5
[  439.349651] XFS (dm-0): metadata I/O error in 
"xfs_imap_to_bp+0x74/0x108 [xfs]" at daddr 0x73d4ca0 len 32 error 5
[  439.353244] coredump: 26799(fwupd): 
|/usr/lib/systemd/systemd-coredump pipe failed
[  457.438961] XFS (dm-0): log I/O error -5
[  457.438998] XFS (dm-0): Filesystem has been shut down due to log 
error (0x2).
[  457.439005] XFS (dm-0): Please unmount the filesystem and rectify the 
problem(s).

With patch, reference count of nvme is 6 after system boots up.
2 paths for nvme0 and 4 paths for nvme1:
Can't do "rmmod nvme" command now.

#lsmod|grep nvme
# lsmod|grep nvme
nvme_tcp              262144  0
nvme                  262144  6
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

# rmmod nvme
rmmod: ERROR: Module nvme is in use

If I remove 1 NS nvme device, reference count decreases 6 – 2 = 4 paths 
in total.
# lsmod|grep nvme
nvme_tcp              262144  0
nvme                  262144  4
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

If you need any additional information or have any other considerations, 
please let me know.
Could you please help review the patch and share any feedback?

Thanks,
Wen

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Wen Xiong]

On 2026-08-22 19:46, Sagi Grimberg wrote:

> OK, I now read the original report regarding nvme root device. please 
> disregard.

Hi Keith and Sagi,

For single port nvme device, I found out:
If NS does not have any filesystems configured and the root filesystem 
is located on another SCSI disk, the NVMe device's reference count is 0.
When a filesystem is created on NS, the NVMe device's reference count 
increases to 1.

With my V2 patch for multipath devices,
Nvme's reference count always increases if NS doesn't have any 
filesystem configured.
If nvme1 has 1 NS without any filesystem, reference count is 2 after 
system boots up.

Should Nvme's reference count increases if no filesystem configured on a 
NS?

Thanks,
Wendy

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Nilay Shroff]

On 8/13/26 4:02 AM, wenxiong@linux.ibm.com wrote:
> From: Wen Xiong <wenxiong@linux.ibm.com>
> 
> Add proper module reference counting to prevent premature unloading of
> NVMe transport modules while multipath namespaces are still active.
> 
> When a namespace is added to a multipath device via nvme_mpath_add_disk(),
> the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain
> loaded as long as the multipath device references that namespace. Without
> proper reference counting, the transport module could be unloaded while
> the multipath device is still using resources from that module, leading
> to the potential system crashes.
> 
> This ensures the transport module remains loaded for the entire lifetime
> of the multipath namespace association.
> 
> Signed-off-by: Wen Xiong <wenxiong@linux.ibm.com>
> ---
>   drivers/nvme/host/multipath.c | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 9b9a657fa330..707b8f95727d 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -1348,6 +1348,8 @@ void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns)
>   	sysfs_remove_link_from_group(kobj, nvme_ns_mpath_attr_group.name,
>   			dev_name(target));
>   	clear_bit(NVME_NS_SYSFS_ATTR_LINK, &ns->flags);
> +
> +	module_put(ns->ctrl->ops->module);
>   }
>   
>   void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
> @@ -1379,6 +1381,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
>   	if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
>   		ns->head->disk->nr_zones = ns->disk->nr_zones;
>   #endif
> +	if (!try_module_get(ns->ctrl->ops->module))
> +		dev_err(disk_to_dev(ns->disk),
> +				"Failed to get module reference\n");
>   }
>   
>   void nvme_mpath_remove_disk(struct nvme_ns_head *head)

I know I am late to the party here, but I don't think adding the
module refcounting in nvme_mpath_{add|remove}_sysfs_link() is the
right approach. That would increment the transport module reference
for every namespace/path, which means the transport module could
not be unloaded until all namespaces using it are deleted, even
when none of those namespaces are actually in use. So for instance,
with this change now, I can't rmmod nvme.ko until I delete
all namespaces/paths created under PCIe nvme subsystem, even though
no one is using the namespace/path under that subsystem.

Instead, I think the module reference should track the lifetime of an
active user of the multipath namespace. When the namespace is in use,
the corresponding transport module(s) need to remain loaded and once
the namespace is no longer in use, those references should be released.

Looking at the current code, nvme_ns_head_{open|release}() seems to be
a more appropriate place for this. When a multipath namespace is opened,
nvme_ns_head_open() could take the required transport module references,
and nvme_ns_head_release() could drop them when the last user releases
the namespace.

Also, since a multipath head can have paths through different transports
, we should not take a reference only to the transport of one namespace/path
found through nvme_find_path(), as was done in v1. Instead, when opening the
head, take a module reference iterating through each controller reachable
from the corresponding NVMe subsystem. This ensures that every transport
that can service I/O for the active multipath namespace remains loaded for
the duration of its use.

Thanks,
--Nilay

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Keith Busch]

On Wed, Aug 26, 2026 at 06:10:09PM +0530, Nilay Shroff wrote:
> when none of those namespaces are actually in use. So for instance,
> with this change now, I can't rmmod nvme.ko until I delete
> all namespaces/paths created under PCIe nvme subsystem, even though
> no one is using the namespace/path under that subsystem.

I'd call that a feature. :)

> Instead, I think the module reference should track the lifetime of an
> active user of the multipath namespace. When the namespace is in use,
> the corresponding transport module(s) need to remain loaded and once
> the namespace is no longer in use, those references should be released.

That's what v1 was trying to do, but it doesn't work. The transports
used to reach a namespace may not be the same across the open and close.

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Wen Xiong]

On 2026-08-26 07:40, Nilay Shroff wrote:

> Also, since a multipath head can have paths through different 
> transports
> , we should not take a reference only to the transport of one 
> namespace/path
> found through nvme_find_path(), as was done in v1. Instead, when 
> opening the
> head, take a module reference iterating through each controller 
> reachable
> from the corresponding NVMe subsystem. This ensures that every 
> transport
> that can service I/O for the active multipath namespace remains loaded 
> for
> the duration of its use.
> 
I will look into iterating though each controller/each namespace from 
nvme subsystem.

Thanks,
Wendy

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Keith Busch]

On Wed, Aug 26, 2026 at 11:18:16AM -0500, Wen Xiong wrote:
> On 2026-08-26 07:40, Nilay Shroff wrote:
> 
> > Also, since a multipath head can have paths through different transports
> > , we should not take a reference only to the transport of one
> > namespace/path
> > found through nvme_find_path(), as was done in v1. Instead, when opening
> > the
> > head, take a module reference iterating through each controller
> > reachable
> > from the corresponding NVMe subsystem. This ensures that every transport
> > that can service I/O for the active multipath namespace remains loaded
> > for
> > the duration of its use.
> > 
> I will look into iterating though each controller/each namespace from nvme
> subsystem.

This is not viable. You can add and remove paths to a namespace at any
time such that the transports counted on open are not the namespace's
transports on close.

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Nilay Shroff]

On 8/26/26 10:06 PM, Keith Busch wrote:
> On Wed, Aug 26, 2026 at 11:18:16AM -0500, Wen Xiong wrote:
>> On 2026-08-26 07:40, Nilay Shroff wrote:
>>
>>> Also, since a multipath head can have paths through different transports
>>> , we should not take a reference only to the transport of one
>>> namespace/path
>>> found through nvme_find_path(), as was done in v1. Instead, when opening
>>> the
>>> head, take a module reference iterating through each controller
>>> reachable
>>> from the corresponding NVMe subsystem. This ensures that every transport
>>> that can service I/O for the active multipath namespace remains loaded
>>> for
>>> the duration of its use.
>>>
>> I will look into iterating though each controller/each namespace from nvme
>> subsystem.
> 
> This is not viable. You can add and remove paths to a namespace at any
> time such that the transports counted on open are not the namespace's
> transports on close.

Yes correct, and I think we need some additional change in the code to
handle this gracefully. I though about it have some initial idea to
address this:

1. Add nr_openers to struct nvme_ns_head.

2. When the ns head is opened, iterate through each namespace associated
    with the head and increment the reference count of its underlying
    transport module. Then increment nr_openers.

3. If a new ns/path is added while the head is open, check nr_openers and,
    if it is non-zero, increment the reference count of the corresponding
    transport module nr_openers times.

4. If an existing ns/path is removed while the head is open, check nr_openers
    and decrement the reference count of the corresponding transport module
    nr_openers times.

5. When the ns head is closed, iterate through the namespaces associated
    with the head and decrement the reference count of each underlying
    transport module. Then decrement nr_openers.

The above operations are serialized by subsys->lock, so nr_openers serves
as the number of users that have actually opened the head node.

For example, suppose we have a shared namespace reachable through
TCP and RDMA paths:

1. User opens the head node:
    head->nr_openers = 1; tcp_module_ref_count = 1; rdma_module_ref_count = 1
2. The RDMA path is removed:
    head->nr_openers = 1; tcp_module_ref_count = 1; rdma_module_ref_count = 0
3. User closes the head node:
    head->nr_openers = 0; tcp_module_ref_count = 0; rdma_module_ref_count = 0

Another example with multiple openers:

1. User A opens the head node:
    head->nr_openers = 1; tcp_module_ref_count = 1; rdma_module_ref_count = 1
2. A new TCP path is added and linked to the head:
    head->nr_openers = 1; tcp_module_ref_count = 2; rdma_module_ref_count = 1
3. User B opens the head node:
    head->nr_openers = 2; tcp_module_ref_count = 4; rdma_module_ref_count = 2
4. User A closes the head node:
    head->nr_openers = 1; tcp_module_ref_count = 2; rdma_module_ref_count = 1
5. The RDMA path is removed:
    head->nr_openers = 1; tcp_module_ref_count = 2; rdma_module_ref_count = 0
6. User B closes the head node:
    head->nr_openers = 0; tcp_module_ref_count = 0; rdma_module_ref_count = 0

This way, the transport module references track the actual number of openers
and the set of paths associated with the head, even when paths are dynamically
added or removed while the head remains open.

Thanks,
--Nilay

Re: [PATCH V2] nvme: Add module reference counting for multipath devices

[Sagi Grimberg]

On 25/08/2026 22:11, Wen Xiong wrote:
> On 2026-08-22 19:46, Sagi Grimberg wrote:
>
>> OK, I now read the original report regarding nvme root device. please 
>> disregard.
>
> Hi Keith and Sagi,
>
> For single port nvme device, I found out:
> If NS does not have any filesystems configured and the root filesystem 
> is located on another SCSI disk, the NVMe device's reference count is 0.
> When a filesystem is created on NS, the NVMe device's reference count 
> increases to 1.
>
> With my V2 patch for multipath devices,
> Nvme's reference count always increases if NS doesn't have any 
> filesystem configured.
> If nvme1 has 1 NS without any filesystem, reference count is 2 after 
> system boots up.
>
> Should Nvme's reference count increases if no filesystem configured on 
> a NS?

No I dont think it should