[PATCH v2 0/2] dmaengine: idxd: Fix refcount and cleanup issues on module unload

[PATCH v2 0/2] dmaengine: idxd: Fix refcount and cleanup issues on module unload

[Yi Sun]

This patch series addresses two issues related to the device reference
counting and cleanup path in the idxd driver.

Recent changes introduced improper put_device() calls and duplicated
cleanup logic, leading to refcount underflow and potential use-after-free
during module unload.

Patch 1 removes an unnecessary call to idxd_free(), which could result in a
use-after-free when paired with asynchronous put_device().

Patch 2 refactors the cleanup path to avoid redundant put_device() calls
introduced in commit a409e919ca3. The existing idxd_unregister_devices()
already handles proper device reference release.

Both patches have been verified on hardware platform.

Both patches have been run through `checkpatch.pl`. Patch 2 gets 1 error
and 1 warning. But these appear to be limitations in the checkpatch script
itself, not reflect issues with the patches.

---
Changes in v2:
- Reworded commit messages supplementing the call traces (Vinicius)
- Explain why the put_device are unnecessary. (Vinicius)

Yi Sun (2):
  dmaengine: idxd: Remove improper idxd_free
  dmaengine: idxd: Fix refcount underflow on module unload

 drivers/dma/idxd/init.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

-- 
2.43.0

[PATCH v2 1/2] dmaengine: idxd: Remove improper idxd_free

[Yi Sun]

The call to idxd_free() introduces a duplicate put_device() leading to a
reference count underflow:
refcount_t: underflow; use-after-free.
WARNING: CPU: 15 PID: 4428 at lib/refcount.c:28 refcount_warn_saturate+0xbe/0x110
...
Call Trace:
 <TASK>
  idxd_remove+0xe4/0x120 [idxd]
  pci_device_remove+0x3f/0xb0
  device_release_driver_internal+0x197/0x200
  driver_detach+0x48/0x90
  bus_remove_driver+0x74/0xf0
  pci_unregister_driver+0x2e/0xb0
  idxd_exit_module+0x34/0x7a0 [idxd]
  __do_sys_delete_module.constprop.0+0x183/0x280
  do_syscall_64+0x54/0xd70
  entry_SYSCALL_64_after_hwframe+0x76/0x7e

The idxd_unregister_devices() which is invoked at the very beginning of
idxd_remove(), already takes care of the necessary put_device() through the
following call path:
idxd_unregister_devices() -> device_unregister() -> put_device()

In addition, when CONFIG_DEBUG_KOBJECT_RELEASE is enabled, put_device() may
trigger asynchronous cleanup via schedule_delayed_work(). If idxd_free() is
called immediately after, it can result in a use-after-free.

Remove the improper idxd_free() to avoid both the refcount underflow and
potential memory corruption during module unload.

Fixes: d5449ff1b04d ("dmaengine: idxd: Add missing idxd cleanup to fix memory leak in remove call")
Signed-off-by: Yi Sun <yi.sun@intel.com>

diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
index 760b7d81fcd8..504aca0fd597 100644
--- a/drivers/dma/idxd/init.c
+++ b/drivers/dma/idxd/init.c
@@ -1324,7 +1324,6 @@ static void idxd_remove(struct pci_dev *pdev)
 	idxd_cleanup(idxd);
 	pci_iounmap(pdev, idxd->reg_base);
 	put_device(idxd_confdev(idxd));
-	idxd_free(idxd);
 	pci_disable_device(pdev);
 }
 
-- 
2.43.0

[PATCH v2 2/2] dmaengine: idxd: Fix refcount underflow on module unload

[Yi Sun]

A recent refactor introduced a misplaced put_device() call, leading to a
reference count underflow during module unload.

There is no need to add additional put_device() calls for idxd groups,
engines, or workqueues. Although commit a409e919ca3 claims:"Note, this
also fixes the missing put_device() for idxd groups, engines, and wqs."
It appears no such omission existed. The required cleanup is already
handled by the call chain:
idxd_unregister_devices() -> device_unregister() -> put_device()

Extend idxd_cleanup() to perform the necessary cleanup, and remove
idxd_cleanup_internals() which was not originally part of the driver
unload path and introduced unintended reference count underflow.

Fixes: a409e919ca32 ("dmaengine: idxd: Refactor remove call with idxd_cleanup() helper")
Signed-off-by: Yi Sun <yi.sun@intel.com>

diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
index 504aca0fd597..a5eabeb6a8bd 100644
--- a/drivers/dma/idxd/init.c
+++ b/drivers/dma/idxd/init.c
@@ -1321,7 +1321,12 @@ static void idxd_remove(struct pci_dev *pdev)
 	device_unregister(idxd_confdev(idxd));
 	idxd_shutdown(pdev);
 	idxd_device_remove_debugfs(idxd);
-	idxd_cleanup(idxd);
+	perfmon_pmu_remove(idxd);
+	idxd_cleanup_interrupts(idxd);
+	if (device_pasid_enabled(idxd))
+		idxd_disable_system_pasid(idxd);
+	if (device_user_pasid_enabled(idxd))
+		idxd_disable_sva(idxd->pdev);
 	pci_iounmap(pdev, idxd->reg_base);
 	put_device(idxd_confdev(idxd));
 	pci_disable_device(pdev);
-- 
2.43.0

Re: [PATCH v2 2/2] dmaengine: idxd: Fix refcount underflow on module unload

[Vinicius Costa Gomes]

Yi Sun <yi.sun@intel.com> writes:

> A recent refactor introduced a misplaced put_device() call, leading to a
> reference count underflow during module unload.
>
> There is no need to add additional put_device() calls for idxd groups,
> engines, or workqueues. Although commit a409e919ca3 claims:"Note, this
> also fixes the missing put_device() for idxd groups, engines, and wqs."
> It appears no such omission existed. The required cleanup is already
> handled by the call chain:
> idxd_unregister_devices() -> device_unregister() -> put_device()
>
> Extend idxd_cleanup() to perform the necessary cleanup, and remove
> idxd_cleanup_internals() which was not originally part of the driver
> unload path and introduced unintended reference count underflow.
>
> Fixes: a409e919ca32 ("dmaengine: idxd: Refactor remove call with idxd_cleanup() helper")
> Signed-off-by: Yi Sun <yi.sun@intel.com>
>
> diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
> index 504aca0fd597..a5eabeb6a8bd 100644
> --- a/drivers/dma/idxd/init.c
> +++ b/drivers/dma/idxd/init.c
> @@ -1321,7 +1321,12 @@ static void idxd_remove(struct pci_dev *pdev)
>  	device_unregister(idxd_confdev(idxd));
>  	idxd_shutdown(pdev);
>  	idxd_device_remove_debugfs(idxd);
> -	idxd_cleanup(idxd);
> +	perfmon_pmu_remove(idxd);
> +	idxd_cleanup_interrupts(idxd);
> +	if (device_pasid_enabled(idxd))
> +		idxd_disable_system_pasid(idxd);
> +	if (device_user_pasid_enabled(idxd))
> +		idxd_disable_sva(idxd->pdev);

idxd_disable_sva() got removed in commit 853b01b5efd7 ("dmaengine: idxd:
Remove unnecessary IOMMU_DEV_FEAT_IOPF")

>  	pci_iounmap(pdev, idxd->reg_base);
>  	put_device(idxd_confdev(idxd));
>  	pci_disable_device(pdev);
> -- 
> 2.43.0
>


Cheers,
-- 
Vinicius

Re: [PATCH v2 2/2] dmaengine: idxd: Fix refcount underflow on module unload

[Yi Sun]

On 13.06.2025 16:20, Vinicius Costa Gomes wrote:
>Yi Sun <yi.sun@intel.com> writes:
>
>> A recent refactor introduced a misplaced put_device() call, leading to a
>> reference count underflow during module unload.
>>
>> There is no need to add additional put_device() calls for idxd groups,
>> engines, or workqueues. Although commit a409e919ca3 claims:"Note, this
>> also fixes the missing put_device() for idxd groups, engines, and wqs."
>> It appears no such omission existed. The required cleanup is already
>> handled by the call chain:
>> idxd_unregister_devices() -> device_unregister() -> put_device()
>>
>> Extend idxd_cleanup() to perform the necessary cleanup, and remove
>> idxd_cleanup_internals() which was not originally part of the driver
>> unload path and introduced unintended reference count underflow.
>>
>> Fixes: a409e919ca32 ("dmaengine: idxd: Refactor remove call with idxd_cleanup() helper")
>> Signed-off-by: Yi Sun <yi.sun@intel.com>
>>
>> diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
>> index 504aca0fd597..a5eabeb6a8bd 100644
>> --- a/drivers/dma/idxd/init.c
>> +++ b/drivers/dma/idxd/init.c
>> @@ -1321,7 +1321,12 @@ static void idxd_remove(struct pci_dev *pdev)
>>  	device_unregister(idxd_confdev(idxd));
>>  	idxd_shutdown(pdev);
>>  	idxd_device_remove_debugfs(idxd);
>> -	idxd_cleanup(idxd);
>> +	perfmon_pmu_remove(idxd);
>> +	idxd_cleanup_interrupts(idxd);
>> +	if (device_pasid_enabled(idxd))
>> +		idxd_disable_system_pasid(idxd);
>> +	if (device_user_pasid_enabled(idxd))
>> +		idxd_disable_sva(idxd->pdev);
>
>idxd_disable_sva() got removed in commit 853b01b5efd7 ("dmaengine: idxd:
>Remove unnecessary IOMMU_DEV_FEAT_IOPF")
I get it, thanks Vinicius for the reminder. Will post another version.

Thanks
    --Sun, Yi