[PATCH v5 0/4] dmaengine: fix kref underflow and UAF in dma_chan_put()

[PATCH v5 0/4] dmaengine: fix kref underflow and UAF in dma_chan_put()

[Shivank Garg]

Fix bugs related to dma_chan_put(), found while testing with SDXI[1].

[1]: https://lore.kernel.org/dmaengine/20260605-sdxi-base-v3-0-4d38ca2bdffe@amd.com

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
Changes in v5:
- Use int return type and add __must_check for dma_device_get() (Frank)
- Drop the comment above synchronize_rcu() and note in the commit message
  that this will delay for grace period dma_list_mutex held (Logan)
- Link to v4: https://lore.kernel.org/r/20260818-dmaengine-kref-fix-v4-0-c6ef991462a0@amd.com

Changes in v4:
- Add dma_device_get() helper (Frank)
- Drop dma_chan_put() move change (Frank)
- Link to v3: https://lore.kernel.org/r/20260816-dmaengine-kref-fix-v3-0-7e76187145df@amd.com

Changes in v3:
- Add patch 3: add synchronize_rcu() to wait for RCU readers to prevent
  use-after-free. (Sashiko)
- Link to v2: https://lore.kernel.org/r/20260526-dmaengine-kref-fix-v2-0-3df60afac01d@amd.com

Changes in v2:
- Add patch 2 fixing the dma_chan_put()/dma_release_channel() use-after-free (sashiko)
- Link to v1: https://lore.kernel.org/r/20260518-dmaengine-kref-fix-v1-1-4d6125048fb7@amd.com

---
Shivank Garg (4):
      dmaengine: add dma_device_get() helper
      dmaengine: Fix device kref underflow in dma_chan_put()
      dmaengine: fix use-after-free in dma_chan_put() and dma_release_channel()
      dmaengine: wait for RCU readers before releasing dma_device

 drivers/dma/dmaengine.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
---
base-commit: a4ff2be345d0abc943da8dd8da98151843b750dc
change-id: 20260518-dmaengine-kref-fix-7b21acb09455

Best regards,
-- 
Shivank Garg <shivankg@amd.com>

[PATCH v5 1/4] dmaengine: add dma_device_get() helper

[Shivank Garg]

Add dma_device_get() helper to match dma_device_put() to make code
symmetric. It wraps open-coded kref_get_unless_zero() and asserts that
dma_list_mutex is held, matching its put counterpart.

No functional change intended.

Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 drivers/dma/dmaengine.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index 9049171df857..cb3b7bccce1f 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -433,6 +433,12 @@ static void dma_device_release(struct kref *ref)
 		device->device_release(device);
 }
 
+static int __must_check dma_device_get(struct dma_device *device)
+{
+	lockdep_assert_held(&dma_list_mutex);
+	return kref_get_unless_zero(&device->ref);
+}
+
 static void dma_device_put(struct dma_device *device)
 {
 	lockdep_assert_held(&dma_list_mutex);
@@ -460,8 +466,7 @@ static int dma_chan_get(struct dma_chan *chan)
 	if (!try_module_get(owner))
 		return -ENODEV;
 
-	ret = kref_get_unless_zero(&chan->device->ref);
-	if (!ret) {
+	if (!dma_device_get(chan->device)) {
 		ret = -ENODEV;
 		goto module_put_out;
 	}

-- 
2.43.0

[PATCH v5 2/4] dmaengine: Fix device kref underflow in dma_chan_put()

[Shivank Garg]

dma_chan_get() takes chan->device->ref only on the slow path:

	/* no kref on fast path */
	if (chan->client_count) {
		__module_get(owner);
		chan->client_count++;
		return 0;
	}
	if (!try_module_get(owner))
		return -ENODEV;
	if (!dma_device_get(chan->device)) { // calls kref_get_unless_zero()

dma_chan_put() drops the ref unconditionally, so every fast-path
get/put pair drops one extra device reference.

The bug fires when two conditions hold together: a non-private
provider has a persistent client holding chan->client_count > 0
and another client cycles dmaengine_get()/dmaengine_put().
When the kref hits zero, the subsequent dma_find_channel() returns
NULL even though the provider module is still loaded.

Fix this by dropping device->ref only on the last put, matching the
single slow-path get.

Fixes: 8ad342a86359 ("dmaengine: Add reference counting to dma_device struct")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 drivers/dma/dmaengine.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index cb3b7bccce1f..f2760b039d93 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -520,7 +520,9 @@ static void dma_chan_put(struct dma_chan *chan)
 		chan->route_data = NULL;
 	}
 
-	dma_device_put(chan->device);
+	/* This channel is not in use anymore, drop the device ref */
+	if (!chan->client_count)
+		dma_device_put(chan->device);
 	module_put(dma_chan_to_owner(chan));
 }
 

-- 
2.43.0

[PATCH v5 3/4] dmaengine: fix use-after-free in dma_chan_put() and dma_release_channel()

[Shivank Garg]

When dma_device_put() drops the last reference on chan->device->ref,
dma_device_release() runs and may free the dma_device along with its
channels.

dma_chan_put() then still reads chan->device->owner via
dma_chan_to_owner() for the trailing module_put(). KASAN catches it:

	slab-use-after-free in dma_chan_put+0x3e6/0x4c0
	Read of size 8 by task insmod/6319
	Freed by task 6319:
	  kfree+0x225/0x470
	  dma_chan_put+0x395/0x4c0
	  dmaengine_put+0xf8/0x160

Cache the module owner in dma_chan_put() before the put so the trailing
module_put() does not need chan->device.

Fixes: 8ad342a86359 ("dmaengine: Add reference counting to dma_device struct")
Suggested-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260518-dmaengine-kref-fix-v1-1-4d6125048fb7@amd.com
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 drivers/dma/dmaengine.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index f2760b039d93..9214b9272a55 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -500,10 +500,13 @@ static int dma_chan_get(struct dma_chan *chan)
  */
 static void dma_chan_put(struct dma_chan *chan)
 {
+	struct module *owner;
+
 	/* This channel is not in use, bail out */
 	if (!chan->client_count)
 		return;
 
+	owner = dma_chan_to_owner(chan);
 	chan->client_count--;
 
 	/* This channel is not in use anymore, free it */
@@ -523,7 +526,7 @@ static void dma_chan_put(struct dma_chan *chan)
 	/* This channel is not in use anymore, drop the device ref */
 	if (!chan->client_count)
 		dma_device_put(chan->device);
-	module_put(dma_chan_to_owner(chan));
+	module_put(owner);
 }
 
 enum dma_status dma_sync_wait(struct dma_chan *chan, dma_cookie_t cookie)

-- 
2.43.0

[PATCH v5 4/4] dmaengine: wait for RCU readers before releasing dma_device

[Shivank Garg]

dma_issue_pending_all() walks the dma_device_list with
list_for_each_entry_rcu() under rcu_read_lock(). dma_device_release()
unlinks the device with list_del_rcu() and then calls
device->device_release() (which in many drivers, such as plx_dma.c,
directly calls kfree()).

Because there is no grace period between unlinking the device and
freeing it, concurrent RCU readers in dma_issue_pending_all() can
access the device after it has been freed.

The lockless walk originally relied on clients holding a dmaengine
reference to pin the provider module, and therefore the device, for as
long as they might traverse the list. Commit 8ad342a86359 ("dmaengine:
Add reference counting to dma_device struct") decoupled the dma_device
lifetime from the module reference, so the device can now be released
while a reader is still walking the list.

Add synchronize_rcu() before the device is freed, so RCU readers are
guaranteed to have finished. Keep it unconditional: providers that do
not implement device_release() free the device themselves once
dma_async_device_unregister() returns. This call will delay for a grace
period with dma_list_mutex held, which is safe and only teardown path is
delayed.

Fixes: 2ba05622b8b1 ("dmaengine: provide a common 'issue_pending_all' implementation")
Suggested-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260526-dmaengine-kref-fix-v2-0-3df60afac01d@amd.com
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 drivers/dma/dmaengine.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index 9214b9272a55..f9bc3e69fe58 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -428,6 +428,7 @@ static void dma_device_release(struct kref *ref)
 
 	list_del_rcu(&device->global_node);
 	dma_channel_rebalance();
+	synchronize_rcu();
 
 	if (device->device_release)
 		device->device_release(device);

-- 
2.43.0

Re: [PATCH v5 2/4] dmaengine: Fix device kref underflow in dma_chan_put()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but dropping the last device reference in `dma_chan_put()` triggers device deallocation, leading to a Use-After-Free access in `dma_release_channel()`. While a subsequent patch in the series ('commit "dmaengine: fix use-after-free in dma_chan_put() and dma_release_channel()"') fixes a UAF inside `dma_chan_put()`, it fails to address the subsequent UAF in `dma_release_channel()` itself.
--

--- Patch [2]: [PATCH v5 2/4] dmaengine: Fix device kref underflow in dma_chan_put() ---
commit 3c7cc7c5c7f5082cf18d425005421e76b4f2aada
Author: Shivank Garg <shivankg@amd.com>

dmaengine: Fix device kref underflow in dma_chan_put()
    
dma_chan_get() takes chan->device->ref only on the slow path:
    
        /* no kref on fast path */
        if (chan->client_count) {
                __module_get(owner);
                chan->client_count++;
                return 0;
        }
        if (!try_module_get(owner))
                return -ENODEV;
        if (!dma_device_get(chan->device)) { // calls kref_get_unless_zero()

[ ... ]
    
Fixes: 8ad342a86359 ("dmaengine: Add reference counting to dma_device struct")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>

> diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
> index cb3b7bccce1f4..f2760b039d932 100644
> --- a/drivers/dma/dmaengine.c
> +++ b/drivers/dma/dmaengine.c
> @@ -520,7 +520,9 @@ static void dma_chan_put(struct dma_chan *chan)
>  		chan->route_data = NULL;
>  	}
>  
> -	dma_device_put(chan->device);
> +	/* This channel is not in use anymore, drop the device ref */
> +	if (!chan->client_count)
> +		dma_device_put(chan->device);
[Severity: High]
This is a pre-existing issue, but dropping the last device reference in
dma_chan_put() can trigger device deallocation, leading to a use-after-free
access in dma_release_channel().

When a DMA provider device is unregistered while a client still holds a channel,
the provider's dma_async_device_unregister() drops its initial reference.
Later, when the client releases the channel by calling dma_release_channel(),
it calls dma_chan_put() which drops the final reference to the dma_device.
This causes the device and its embedded dma_chan structures to be freed.

Immediately after dma_chan_put() returns, dma_release_channel() dereferences
the freed memory:

drivers/dma/dmaengine.c:dma_release_channel() {
	...
	dma_chan_put(chan);

	if (chan->slave) {
		sysfs_remove_link(&chan->dev->device.kobj, DMA_SLAVE_NAME);
	...
}

While a subsequent patch in this series fixes a use-after-free inside
dma_chan_put(), it appears to leave this subsequent use-after-free in
dma_release_channel() unaddressed.

Could the channel sysfs cleanup and debugfs frees happen before we drop the
final device reference?

>  	module_put(dma_chan_to_owner(chan));
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260822-dmaengine-kref-fix-v5-0-d4a4ee47d927@amd.com?part=2

Re: [PATCH v5 4/4] dmaengine: wait for RCU readers before releasing dma_device

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [High] Double `list_del_rcu` on a device's global_node leading to list corruption (`LIST_POISON2` splats) or kernel panic when a provider module is unloaded.
- [High] The exported functions `dma_async_device_channel_register` and `dma_async_device_channel_unregister` call `dma_channel_rebalance()` without holding the required `dma_list_mutex`.
- [High] Lockless TOCTOU race in `dma_issue_pending_all` between `chan->client_count` check and `device_issue_pending` execution.
--

--- Patch [4]: [PATCH v5 4/4] dmaengine: wait for RCU readers before releasing dma_device ---
commit 2e3a690f3e98bd55a6714eab025b6c1af7e3d406
Author: Shivank Garg <shivankg@amd.com>

dmaengine: wait for RCU readers before releasing dma_device
    
This patch introduces a synchronize_rcu() call before releasing the
dma_device in dma_device_release(). It ensures concurrent RCU readers in
dma_issue_pending_all() have finished traversing the list before the device
is freed.

Link: https://sashiko.dev/#/patchset/20260526-dmaengine-kref-fix-v2-0-3df60afac01d@amd.com

> dma_issue_pending_all() walks the dma_device_list with
> list_for_each_entry_rcu() under rcu_read_lock().

[Severity: High]
This is a pre-existing issue, but can this lead to a lockless TOCTOU race
between the client_count check and the execution of device_issue_pending()?

In dma_issue_pending_all(), the code locklessly checks if chan->client_count
is non-zero before calling device->device_issue_pending(chan). Concurrently,
dma_release_channel() or dmaengine_put() could acquire dma_list_mutex,
decrement client_count to zero, and tear down the descriptor rings. 

If dma_issue_pending_all() stalls between the check and execution, could it
invoke device_issue_pending() on a channel that has just been released and
disabled?

> diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
> index 9214b9272a552..f9bc3e69fe58b 100644
> --- a/drivers/dma/dmaengine.c
> +++ b/drivers/dma/dmaengine.c
> @@ -428,6 +428,7 @@ static void dma_device_release(struct kref *ref)
>  
>  	list_del_rcu(&device->global_node);

[Severity: High]
This isn't a bug introduced by this patch, but could this unconditional
removal result in a double list_del_rcu() and list corruption during module
unloading?

If try_module_get() fails in dma_chan_get() during a concurrent module
unload, dmaengine_get() or find_candidate() will manually execute
list_del_rcu(&device->global_node). 

Later, when the unloading module calls dma_async_device_unregister() and
drops the final reference, dma_device_release() is invoked. Will this cause
a second list_del_rcu() on the same node and a kernel panic from
dereferencing LIST_POISON2?

>  	dma_channel_rebalance();

[Severity: High]
This is a pre-existing issue, but is it safe for exported functions to call
dma_channel_rebalance() without holding dma_list_mutex?

When drivers dynamically register or unregister channels using
dma_async_device_channel_register() or dma_async_device_channel_unregister(),
they call dma_channel_rebalance() completely unlocked. 

Since dma_channel_rebalance() mutates the global channel_table and iterates
over dma_device_list without RCU protection, could concurrent structural
modifications to the list lead to data races or list corruption?

> +	synchronize_rcu();
>  
>  	if (device->device_release)
>  		device->device_release(device);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260822-dmaengine-kref-fix-v5-0-d4a4ee47d927@amd.com?part=4

Re: [PATCH v5 3/4] dmaengine: fix use-after-free in dma_chan_put() and dma_release_channel()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [High] Unpatched Use-After-Free in dma_release_channel() due to dereferencing the channel after dma_chan_put().
- [High] Leaked channel, device, and module references in dmaengine_put() for unregistered public devices.
- [High] Double list_del_rcu() on device->global_node causes list corruption and kernel panic.
--

--- Patch [3]: [PATCH v5 3/4] dmaengine: fix use-after-free in dma_chan_put() and dma_release_channel() ---
commit f921c2034a3380b885abc856562df23d25421487
Author: Shivank Garg <shivankg@amd.com>

dmaengine: fix use-after-free in dma_chan_put() and dma_release_channel()

When dma_device_put() drops the last reference on chan->device->ref,
dma_device_release() runs and may free the dma_device along with its
channels. This patch caches the module owner in dma_chan_put() before the
put so the trailing module_put() does not need chan->device, preventing
a use-after-free caught by kasan.

Link: https://sashiko.dev/#/patchset/20260518-dmaengine-kref-fix-v1-1-4d6125048fb7@amd.com

> diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
> index f2760b039d932..9214b9272a552 100644
> --- a/drivers/dma/dmaengine.c
> +++ b/drivers/dma/dmaengine.c
[ ... ]
> @@ -523,7 +526,7 @@ static void dma_chan_put(struct dma_chan *chan)
>  	/* This channel is not in use anymore, drop the device ref */
>  	if (!chan->client_count)
>  		dma_device_put(chan->device);
> -	module_put(dma_chan_to_owner(chan));
> +	module_put(owner);
>  }

[Severity: High]
This isn't a regression introduced by this patch, but since this commit
addresses a use-after-free involving dma_chan_put(), does
dma_release_channel() have a similar issue?

If dma_async_device_unregister() was called earlier, dropping the base device
reference, calling dma_release_channel() will invoke dma_chan_put(). This
decrements the client count to zero, calls dma_device_put(), and drops the
final reference, freeing the device and its embedded channel structure.

However, immediately after dma_chan_put(), dma_release_channel() dereferences
the freed channel:

drivers/dma/dmaengine.c:dma_release_channel() {
	...
	dma_chan_put(chan);

	if (chan->slave) {
		sysfs_remove_link(&chan->dev->device.kobj, DMA_SLAVE_NAME);
	...
}

Does this cause a use-after-free when accessing chan->slave, chan->dev,
and chan->name?

[Severity: High]
This is a pre-existing issue, but I noticed a potential resource leak in
how unregistered public devices are handled.

When dmaengine_get() allocates public channels, it increments
chan->client_count and takes device and module references. If the device is
subsequently unregistered, dma_async_device_unregister() sets the
DMA_PRIVATE flag to hide it:

drivers/dma/dmaengine.c:dma_async_device_unregister() {
	...
	dma_cap_set(DMA_PRIVATE, device->cap_mask);
	...
}

When dmaengine_put() runs later, it explicitly skips any device with
DMA_PRIVATE set:

drivers/dma/dmaengine.c:dmaengine_put() {
	...
		if (dma_has_cap(DMA_PRIVATE, device->cap_mask))
			continue;
		list_for_each_entry(chan, &device->channels, device_node)
			dma_chan_put(chan);
	...
}

Since dma_chan_put() is never called for the channels of the unregistered
device, does this permanently leak chan->client_count, the module reference,
and the device kref?

[Severity: High]
This is a pre-existing issue, but is there a risk of list corruption due to a
double list deletion?

In find_candidate() and dmaengine_get(), if dma_chan_get() fails with -ENODEV
(such as when the backing module is being unloaded), they proactively remove
the device from the list:

drivers/dma/dmaengine.c:find_candidate() {
	...
			if (err == -ENODEV) {
				...
				list_del_rcu(&device->global_node);
	...
}

When the driver's exit routine subsequently executes
dma_async_device_unregister(), it drops the device reference and eventually
invokes dma_device_release(). This unconditionally deletes the node again:

drivers/dma/dmaengine.c:dma_device_release() {
	...
	list_del_rcu(&device->global_node);
	...
}

Can this double list_del_rcu() on device->global_node corrupt the
doubly-linked list?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260822-dmaengine-kref-fix-v5-0-d4a4ee47d927@amd.com?part=3