[PATCH 0/7] dmaengine: dw-edma: Fixes and interrupt-path groundwork

[PATCH 0/7] dmaengine: dw-edma: Fixes and interrupt-path groundwork

[Koichiro Den]

Hi,

This series collects fixes and interrupt-path groundwork for dw-edma,
mostly split out of v1 of the "Support dynamic LL appends" series:

  [PATCH 00/17] dmaengine: dw-edma: Support dynamic LL appends
  https://lore.kernel.org/dmaengine/20260615154111.2174161-1-den@valinux.co.jp/

That series had grown too large to review as one unit. Patches 1-4 are
what remains of v1's patches 01-05, slightly reordered but unchanged
in substance: Frank picked up v1's patch 01 (the residue fix) into
edma_ll v5,

  https://lore.kernel.org/dmaengine/20260709-edma_ll-v5-10-e199053d4300@nxp.com/

so it is no longer carried here. Patches 5-7 are new. None of them
depend on the dynamic-append work, and they are worth having on their
own.

Patches 1-4 are fixes.

- Patch 1 fixes a swapped argument order in the HDMA channel status
  register access (Cc: stable). v1 deferred this to a similar fix
  expected from Devendra, which has not appeared so far. Patch 1
  applies directly to mainline and can be picked independently.
- Patch 2 stops invoking completion callbacks for STOP-terminated
  descriptors.
- Patch 3 fixes vchan descriptor bookkeeping on termination.
- Patch 4 serializes channel state checks in pause/resume/terminate
  (carries Frank's Reviewed-by from the v1 thread).

Patch 5 drops a redundant remove-time pci_free_irq_vectors(): the device
is managed by pcim_enable_device(), so the vectors are released on
device release.

Patch 6 reads the shared DONE/ABORT interrupt status register once per
v0 handler pass. On remote setups every register read is a non-posted
round trip across the PCIe link, so the redundant reads cost real
latency in the hot path.

Patch 7 moves per-channel interrupt handling out of hard IRQ context
into per-channel work items on a dedicated WQ_UNBOUND | WQ_HIGHPRI
workqueue. On SoCs like R-Car S4 the endpoint-side eDMA raises a single
fixed SPI hardwired to CPU0 for all read and write channels; handling
every channel event in that hard IRQ serializes completion processing on
one CPU and turns vc.lock contention into system-wide interrupt latency.

Based on v7.2-rc1 (dmaengine/master).

To maintainers/reviewers:
- Sashiko's pre-existing-issue report on Frank's edma_ll v5 thread
  (lockless pause/resume/terminate_all racing the virt-dma lists,
  https://lore.kernel.org/dmaengine/20260709154606.734B31F00A3A@smtp.kernel.org/)
  is addressed by patch 4 of this series.
- This series textually conflicts with Frank's edma_ll v5 in the v0
  interrupt handler and the termination paths, but the conflicts are
  small. It has been tested with edma_ll v5 applied on top, on an R-Car
  S4 endpoint/host pair, and I am happy to rebase either way if
  preferred.
- The "Support dynamic LL appends" v2 and a follow-up v0 engine
  recovery series will follow shortly. They build on top of this
  series plus whichever revision of Frank's edma_ll lands.

Best regards,
Koichiro


Koichiro Den (7):
  dmaengine: dw-edma: Fix HDMA channel status register access
  dmaengine: dw-edma: Terminate STOP requests without callbacks
  dmaengine: dw-edma: Clean up vchan descriptors on termination
  dmaengine: dw-edma: Serialize channel state checks
  dmaengine: dw-edma-pcie: Drop redundant pci_free_irq_vectors()
  dmaengine: dw-edma: Snapshot the v0 interrupt status once per handler
    pass
  dmaengine: dw-edma: Defer channel IRQ handling to workqueue

 drivers/dma/dw-edma/dw-edma-core.c    | 170 +++++++++++++++++++++++---
 drivers/dma/dw-edma/dw-edma-core.h    |  12 ++
 drivers/dma/dw-edma/dw-edma-pcie.c    |   3 -
 drivers/dma/dw-edma/dw-edma-v0-core.c |  28 ++---
 drivers/dma/dw-edma/dw-hdma-v0-core.c |   2 +-
 5 files changed, 177 insertions(+), 38 deletions(-)

-- 
2.51.0

[PATCH 1/7] dmaengine: dw-edma: Fix HDMA channel status register access

[Koichiro Den]

GET_CH_32() takes the direction before the channel ID, but
dw_hdma_v0_core_ch_status() passed them in the opposite order. This can
make the status callback read another HDMA channel status register.

Use the same argument order as the other HDMA register accesses.

Fixes: e74c39573d35 ("dmaengine: dw-edma: Add support for native HDMA")
Cc: stable@vger.kernel.org
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v2:
  - Split out into this preparation series (was patch 02/17 of the
    dynamic LL appends v1); no changes to the patch itself.
  - Collect Frank's Reviewed-by.

 drivers/dma/dw-edma/dw-hdma-v0-core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index 632abb8b481c..2beec876b184 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -79,7 +79,7 @@ static enum dma_status dw_hdma_v0_core_ch_status(struct dw_edma_chan *chan)
 	u32 tmp;
 
 	tmp = FIELD_GET(HDMA_V0_CH_STATUS_MASK,
-			GET_CH_32(dw, chan->id, chan->dir, ch_stat));
+			GET_CH_32(dw, chan->dir, chan->id, ch_stat));
 
 	if (tmp == 1)
 		return DMA_IN_PROGRESS;
-- 
2.51.0

[PATCH 2/7] dmaengine: dw-edma: Terminate STOP requests without callbacks

[Koichiro Den]

The STOP request path handles device_terminate_all(). The DMA Engine
client documentation says in the "Terminate APIs" section of
Documentation/driver-api/dmaengine/client.rst:

"No callback functions will be called for any incomplete transfers."

dw-edma used vchan_cookie_complete() for a stopped descriptor. This
queues the descriptor on the completed list and schedules its callback.
A late callback after dmaengine_terminate_sync() can dereference
callback state, such as a request object, that the client has already
freed.

Move the stopped descriptor to the terminated list. Complete the cookie
before doing so, so cookie polling observes that the transfer is no
longer in flight, but do not schedule the completion callback. Add a
synchronize callback so virt-dma can release terminated descriptors.

Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v2:
  - Split out into this preparation series (was patch 03/17 of
    the dynamic LL appends v1).
  - No changes.

 drivers/dma/dw-edma/dw-edma-core.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 89a4c498a17b..4e0dc52397e2 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -201,6 +201,13 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 	return 1;
 }
 
+static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
+{
+	list_del(&vd->node);
+	dma_cookie_complete(&vd->tx);
+	vchan_terminate_vdesc(vd);
+}
+
 static void dw_edma_device_caps(struct dma_chan *dchan,
 				struct dma_slave_caps *caps)
 {
@@ -673,8 +680,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 			break;
 
 		case EDMA_REQ_STOP:
-			list_del(&vd->node);
-			vchan_cookie_complete(vd);
+			dw_edma_terminate_vdesc(vd);
 			chan->request = EDMA_REQ_NONE;
 			chan->status = EDMA_ST_IDLE;
 			break;
@@ -856,6 +862,13 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
 	return 0;
 }
 
+static void dw_edma_device_synchronize(struct dma_chan *dchan)
+{
+	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+
+	vchan_synchronize(&chan->vc);
+}
+
 static void dw_edma_free_chan_resources(struct dma_chan *dchan)
 {
 	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
@@ -968,6 +981,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
 	dma->device_pause = dw_edma_device_pause;
 	dma->device_resume = dw_edma_device_resume;
 	dma->device_terminate_all = dw_edma_device_terminate_all;
+	dma->device_synchronize = dw_edma_device_synchronize;
 	dma->device_issue_pending = dw_edma_device_issue_pending;
 	dma->device_tx_status = dw_edma_device_tx_status;
 	dma->device_prep_slave_sg = dw_edma_device_prep_slave_sg;
-- 
2.51.0

[PATCH 3/7] dmaengine: dw-edma: Clean up vchan descriptors on termination

[Koichiro Den]

dw-edma resets channel state from terminate_all() paths, but pending
virt-dma descriptors can remain on the submitted and issued lists. A later
issue_pending() may then restart work that the client already terminated,
possibly into buffers that were already reused. Descriptors that are never
restarted leak instead.

Move issued and submitted descriptors to the terminated list whenever a
termination request completes. Also release virt-dma resources from
free_chan_resources().

If termination was deferred because the channel was still running, wait
until the STOP path deconfigures the channel before synchronizing or
freeing virt-dma resources. Otherwise dmaengine_terminate_sync() can return
before the deferred STOP cleanup has moved issued descriptors to the
terminated list and before the channel is known to have stopped.

The old free_chan_resources() loop usually broke as soon as terminate_all()
returned zero, so it did not effectively spin until the timeout. This wait
can now last until the existing timeout, so use cond_resched() instead of
busy-polling with cpu_relax(), and warn if the timeout expires.

Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v2:
  - Poll with usleep_range() (and include linux/delay.h for it) instead
    of a cond_resched() busy loop in the termination wait; each
    iteration does an MMIO read that is a non-posted round trip on
    remote setups.
  - Split out into this preparation series (was patch 04/17 of
    the dynamic LL appends v1).
  - Let dw_edma_free_chan_resources() reuse dw_edma_device_synchronize()
    instead of open-coding the same wait-and-synchronize sequence.

 drivers/dma/dw-edma/dw-edma-core.c | 79 ++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 14 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 4e0dc52397e2..1b493c104a5b 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -7,6 +7,7 @@
  */
 
 #include <linux/module.h>
+#include <linux/delay.h>
 #include <linux/device.h>
 #include <linux/kernel.h>
 #include <linux/dmaengine.h>
@@ -15,6 +16,7 @@
 #include <linux/irq.h>
 #include <linux/dma/edma.h>
 #include <linux/dma-mapping.h>
+#include <linux/sched.h>
 #include <linux/string_choices.h>
 
 #include "dw-edma-core.h"
@@ -208,6 +210,28 @@ static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
 	vchan_terminate_vdesc(vd);
 }
 
+static void dw_edma_terminate_vdesc_list(struct list_head *head)
+{
+	struct virt_dma_desc *vd, *_vd;
+
+	list_for_each_entry_safe(vd, _vd, head, node)
+		dw_edma_terminate_vdesc(vd);
+}
+
+/* Must be called with vc.lock held. */
+static void dw_edma_terminate_all_descs(struct dw_edma_chan *chan)
+{
+	/*
+	 * This order must not be reversed. Cookies are assigned when
+	 * descriptors are submitted, so desc_issued contains older cookies
+	 * than desc_submitted. Completing desc_submitted first could move
+	 * chan->vc.chan.completed_cookie backwards when desc_issued is
+	 * terminated afterwards.
+	 */
+	dw_edma_terminate_vdesc_list(&chan->vc.desc_issued);
+	dw_edma_terminate_vdesc_list(&chan->vc.desc_submitted);
+}
+
 static void dw_edma_device_caps(struct dma_chan *dchan,
 				struct dma_slave_caps *caps)
 {
@@ -313,20 +337,25 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
 static int dw_edma_device_terminate_all(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long flags;
 	int err = 0;
 
+	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (!chan->configured) {
-		/* Do nothing */
+		dw_edma_terminate_all_descs(chan);
 	} else if (chan->status == EDMA_ST_PAUSE) {
+		dw_edma_terminate_all_descs(chan);
 		chan->status = EDMA_ST_IDLE;
 		chan->configured = false;
 	} else if (chan->status == EDMA_ST_IDLE) {
+		dw_edma_terminate_all_descs(chan);
 		chan->configured = false;
 	} else if (dw_edma_core_ch_status(chan) == DMA_COMPLETE) {
 		/*
 		 * The channel is in a false BUSY state, probably didn't
 		 * receive or lost an interrupt
 		 */
+		dw_edma_terminate_all_descs(chan);
 		chan->status = EDMA_ST_IDLE;
 		chan->configured = false;
 	} else if (chan->request > EDMA_REQ_PAUSE) {
@@ -334,6 +363,7 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
 	} else {
 		chan->request = EDMA_REQ_STOP;
 	}
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
 	return err;
 }
@@ -680,7 +710,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 			break;
 
 		case EDMA_REQ_STOP:
-			dw_edma_terminate_vdesc(vd);
+			dw_edma_terminate_all_descs(chan);
 			chan->request = EDMA_REQ_NONE;
 			chan->status = EDMA_ST_IDLE;
 			break;
@@ -862,28 +892,49 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
 	return 0;
 }
 
+static void dw_edma_wait_termination(struct dma_chan *dchan)
+{
+	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
+	unsigned long flags;
+	bool configured = true;
+
+	/*
+	 * dw_edma_device_terminate_all() may defer cleanup to a later interrupt
+	 * while the channel is still running. Retry until the channel is
+	 * deconfigured, which marks that termination completed.
+	 */
+	while (time_before(jiffies, timeout)) {
+		dw_edma_device_terminate_all(dchan);
+
+		spin_lock_irqsave(&chan->vc.lock, flags);
+		configured = chan->configured;
+		spin_unlock_irqrestore(&chan->vc.lock, flags);
+		if (!configured)
+			return;
+
+		usleep_range(1000, 2000);
+		cond_resched();
+	}
+
+	dev_warn(chan->dw->chip->dev,
+		 "timeout waiting for channel termination\n");
+}
+
 static void dw_edma_device_synchronize(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 
+	dw_edma_wait_termination(dchan);
 	vchan_synchronize(&chan->vc);
 }
 
 static void dw_edma_free_chan_resources(struct dma_chan *dchan)
 {
-	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
-	int ret;
-
-	while (time_before(jiffies, timeout)) {
-		ret = dw_edma_device_terminate_all(dchan);
-		if (!ret)
-			break;
-
-		if (time_after_eq(jiffies, timeout))
-			return;
+	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 
-		cpu_relax();
-	}
+	dw_edma_device_synchronize(dchan);
+	vchan_free_chan_resources(&chan->vc);
 }
 
 static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
-- 
2.51.0

[PATCH 4/7] dmaengine: dw-edma: Serialize channel state checks

[Koichiro Den]

pause() and resume() read and update channel state without holding vc.lock,
while the interrupt handlers update the same state under it. Take the same
lock around those state checks so that request, status, and configured stay
consistent.

For example, pause() can observe EDMA_ST_BUSY right before the interrupt
handler completes the final descriptor and moves the channel to
EDMA_ST_IDLE, and then record EDMA_REQ_PAUSE on an already idle channel. No
further interrupt will acknowledge the request, and since issue_pending()
requires EDMA_REQ_NONE, the channel is wedged for good: terminate_all()
leaves the stale request behind, so even reconfiguring the channel does not
recover it.

issue_pending() already runs under vc.lock, but it tests configured before
taking it. Move that test under the lock as well, so that the decision to
start work is made against the current value rather than one observed
before a concurrent terminate_all() deconfigured the channel.

Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v2:
  - Split out into this preparation series (was patch 05/17 of
    the dynamic LL appends v1).
  - Collect Frank's Reviewed-by.

 drivers/dma/dw-edma/dw-edma-core.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 1b493c104a5b..5664421c6f15 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -301,8 +301,10 @@ static int dw_edma_device_config(struct dma_chan *dchan,
 static int dw_edma_device_pause(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long flags;
 	int err = 0;
 
+	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (!chan->configured)
 		err = -EPERM;
 	else if (chan->status != EDMA_ST_BUSY)
@@ -311,6 +313,7 @@ static int dw_edma_device_pause(struct dma_chan *dchan)
 		err = -EPERM;
 	else
 		chan->request = EDMA_REQ_PAUSE;
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
 	return err;
 }
@@ -318,8 +321,10 @@ static int dw_edma_device_pause(struct dma_chan *dchan)
 static int dw_edma_device_resume(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long flags;
 	int err = 0;
 
+	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (!chan->configured) {
 		err = -EPERM;
 	} else if (chan->status != EDMA_ST_PAUSE) {
@@ -330,6 +335,7 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
 		chan->status = EDMA_ST_BUSY;
 		dw_edma_start_transfer(chan);
 	}
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
 	return err;
 }
@@ -373,11 +379,9 @@ static void dw_edma_device_issue_pending(struct dma_chan *dchan)
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 	unsigned long flags;
 
-	if (!chan->configured)
-		return;
-
 	spin_lock_irqsave(&chan->vc.lock, flags);
-	if (vchan_issue_pending(&chan->vc) && chan->request == EDMA_REQ_NONE &&
+	if (chan->configured && vchan_issue_pending(&chan->vc) &&
+	    chan->request == EDMA_REQ_NONE &&
 	    chan->status == EDMA_ST_IDLE) {
 		chan->status = EDMA_ST_BUSY;
 		dw_edma_start_transfer(chan);
-- 
2.51.0

[PATCH 5/7] dmaengine: dw-edma-pcie: Drop redundant pci_free_irq_vectors()

[Koichiro Den]

dw_edma_pcie enables the PCI device with pcim_enable_device(), so IRQ
vectors allocated by pci_alloc_irq_vectors() are released by
pcim_msi_release() on device release. The driver should not call
pci_free_irq_vectors() manually.

Drop the redundant remove-time cleanup and rely on the managed PCI
device lifetime instead, as documented by commit 03e4905402ae ("PCI/MSI:
Clarify pci_free_irq_vectors() usage for managed devices").

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v2:
  - New patch in v2, posted as part of this preparation series.

 drivers/dma/dw-edma/dw-edma-pcie.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-pcie.c b/drivers/dma/dw-edma/dw-edma-pcie.c
index 791c46e8ae4c..5e81a433a957 100644
--- a/drivers/dma/dw-edma/dw-edma-pcie.c
+++ b/drivers/dma/dw-edma/dw-edma-pcie.c
@@ -555,9 +555,6 @@ static void dw_edma_pcie_remove(struct pci_dev *pdev)
 	err = dw_edma_remove(chip);
 	if (err)
 		pci_warn(pdev, "can't remove device properly: %d\n", err);
-
-	/* Freeing IRQs */
-	pci_free_irq_vectors(pdev);
 }
 
 static const struct pci_device_id dw_edma_pcie_id_table[] = {
-- 
2.51.0

[PATCH 6/7] dmaengine: dw-edma: Snapshot the v0 interrupt status once per handler pass

[Koichiro Den]

The v0 interrupt handler reads the interrupt status register twice per
invocation, once through the DONE accessor and once through the ABORT
accessor, although both fields live in the same 32-bit register. On
remote setups (dw-edma-pcie) each read is a non-posted round trip across
the PCIe link costing on the order of a microsecond, and with one
completion interrupt per element the duplicate adds up. As an example,
profiling the R-Car S4 remote path put the handler at ~7us per
invocation, dominated by such reads.

Read the register once and derive the DONE and ABORT views from the
snapshot. No abort is lost to this because the pass only clears status
bits it observed, so an abort raised after the snapshot keeps its status
and its own interrupt delivery brings it to the next pass. A second
abort on an observed channel cannot race the clear either, as an aborted
channel stays halted until software restarts it, and any restart follows
the abort() handling, which comes after
dw_edma_v0_core_clear_abort_int().

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v2:
  - New patch in v2, posted as part of this preparation series.

 drivers/dma/dw-edma/dw-edma-v0-core.c | 28 +++++++++++++--------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index cfdd6463252e..377812eaa110 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -218,18 +218,6 @@ static void dw_edma_v0_core_clear_abort_int(struct dw_edma_chan *chan)
 		  FIELD_PREP(EDMA_V0_ABORT_INT_MASK, BIT(chan->id)));
 }
 
-static u32 dw_edma_v0_core_status_done_int(struct dw_edma *dw, enum dw_edma_dir dir)
-{
-	return FIELD_GET(EDMA_V0_DONE_INT_MASK,
-			 GET_RW_32(dw, dir, int_status));
-}
-
-static u32 dw_edma_v0_core_status_abort_int(struct dw_edma *dw, enum dw_edma_dir dir)
-{
-	return FIELD_GET(EDMA_V0_ABORT_INT_MASK,
-			 GET_RW_32(dw, dir, int_status));
-}
-
 static irqreturn_t
 dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 			   dw_edma_handler_t done, dw_edma_handler_t abort)
@@ -239,7 +227,7 @@ dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 	irqreturn_t ret = IRQ_NONE;
 	struct dw_edma_chan *chan;
 	unsigned long off;
-	u32 mask;
+	u32 mask, sts;
 
 	if (dir == EDMA_DIR_WRITE) {
 		total = dw->wr_ch_cnt;
@@ -251,7 +239,17 @@ dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 		mask = dw_irq->rd_mask;
 	}
 
-	val = dw_edma_v0_core_status_done_int(dw, dir);
+	/*
+	 * DONE and ABORT status share one register, and on remote setups
+	 * every read is a non-posted round trip across the PCIe link. Take
+	 * one snapshot and derive both views from it. An abort raised
+	 * after the snapshot is deferred, not lost: only bits observed in
+	 * the snapshot are ever cleared below, so its status survives for
+	 * the next invocation, which its own interrupt delivery triggers.
+	 */
+	sts = GET_RW_32(dw, dir, int_status);
+
+	val = FIELD_GET(EDMA_V0_DONE_INT_MASK, sts);
 	val &= mask;
 	for_each_set_bit(pos, &val, total) {
 		chan = &dw->chan[pos + off];
@@ -262,7 +260,7 @@ dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 		ret = IRQ_HANDLED;
 	}
 
-	val = dw_edma_v0_core_status_abort_int(dw, dir);
+	val = FIELD_GET(EDMA_V0_ABORT_INT_MASK, sts);
 	val &= mask;
 	for_each_set_bit(pos, &val, total) {
 		chan = &dw->chan[pos + off];
-- 
2.51.0

[PATCH 7/7] dmaengine: dw-edma: Defer channel IRQ handling to workqueue

[Koichiro Den]

On some SoCs (e.g. R-Car S4) the endpoint-side eDMA raises a single
fixed SPI that is hardwired to CPU0 and covers every read and write
channel. Handling channel events directly in that hard IRQ context
serializes the completion processing of all channels on one CPU:
descriptor recycling and refill, client callbacks (the vchan tasklet
runs on the scheduling CPU) and the doorbell writes all funnel through
CPU0, while the handler additionally spins on each channel's vc.lock.
Especially under multi-channels heavy load, the contention becomes a
performance bottleneck.

Keep the hard IRQ handler minimal and have it just clear the status and
dispatch, defer the per-channel processing to work items. A work item
per channel preserves per-channel ordering while letting the channels be
processed in parallel on any CPU.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v2:
  - New patch in v2, posted as part of this preparation series.

 drivers/dma/dw-edma/dw-edma-core.c | 73 ++++++++++++++++++++++++++++--
 drivers/dma/dw-edma/dw-edma-core.h | 12 +++++
 2 files changed, 80 insertions(+), 5 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 5664421c6f15..704d8f9746e8 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -31,6 +31,11 @@ struct dw_edma_desc *vd2dw_edma_desc(struct virt_dma_desc *vd)
 	return container_of(vd, struct dw_edma_desc, vd);
 }
 
+enum dw_edma_irq_event {
+	DW_EDMA_IRQ_DONE	= BIT(0),
+	DW_EDMA_IRQ_ABORT	= BIT(1),
+};
+
 static inline
 u64 dw_edma_get_pci_address(struct dw_edma_chan *chan, phys_addr_t cpu_addr)
 {
@@ -748,6 +753,44 @@ static void dw_edma_abort_interrupt(struct dw_edma_chan *chan)
 	chan->status = EDMA_ST_IDLE;
 }
 
+static void dw_edma_irq_work(struct work_struct *work)
+{
+	struct dw_edma_chan *chan = container_of(work, struct dw_edma_chan,
+						 irq_work);
+	unsigned int events;
+
+	do {
+		events = atomic_xchg(&chan->irq_pending, 0);
+
+		if (events & DW_EDMA_IRQ_DONE)
+			dw_edma_done_interrupt(chan);
+		if (events & DW_EDMA_IRQ_ABORT)
+			dw_edma_abort_interrupt(chan);
+		/*
+		 * Correctness does not depend on this loop: queue_work() can
+		 * requeue once the work item starts running. Staying here just
+		 * coalesces back-to-back channel events into one wakeup.
+		 */
+	} while (atomic_read(&chan->irq_pending));
+}
+
+static void dw_edma_queue_irq_work(struct dw_edma_chan *chan,
+				   enum dw_edma_irq_event event)
+{
+	atomic_or(event, &chan->irq_pending);
+	queue_work(chan->dw->wq, &chan->irq_work);
+}
+
+static void dw_edma_done_interrupt_deferred(struct dw_edma_chan *chan)
+{
+	dw_edma_queue_irq_work(chan, DW_EDMA_IRQ_DONE);
+}
+
+static void dw_edma_abort_interrupt_deferred(struct dw_edma_chan *chan)
+{
+	dw_edma_queue_irq_work(chan, DW_EDMA_IRQ_ABORT);
+}
+
 static void dw_edma_emul_irq_ack(struct irq_data *d)
 {
 	struct dw_edma *dw = irq_data_get_irq_chip_data(d);
@@ -842,8 +885,8 @@ static inline irqreturn_t dw_edma_interrupt_write_inner(int irq, void *data)
 	struct dw_edma_irq *dw_irq = data;
 
 	return dw_edma_core_handle_int(dw_irq, EDMA_DIR_WRITE,
-				       dw_edma_done_interrupt,
-				       dw_edma_abort_interrupt);
+				       dw_edma_done_interrupt_deferred,
+				       dw_edma_abort_interrupt_deferred);
 }
 
 static inline irqreturn_t dw_edma_interrupt_read_inner(int irq, void *data)
@@ -851,8 +894,8 @@ static inline irqreturn_t dw_edma_interrupt_read_inner(int irq, void *data)
 	struct dw_edma_irq *dw_irq = data;
 
 	return dw_edma_core_handle_int(dw_irq, EDMA_DIR_READ,
-				       dw_edma_done_interrupt,
-				       dw_edma_abort_interrupt);
+				       dw_edma_done_interrupt_deferred,
+				       dw_edma_abort_interrupt_deferred);
 }
 
 static inline irqreturn_t dw_edma_interrupt_write(int irq, void *data)
@@ -930,6 +973,7 @@ static void dw_edma_device_synchronize(struct dma_chan *dchan)
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 
 	dw_edma_wait_termination(dchan);
+	cancel_work_sync(&chan->irq_work);
 	vchan_synchronize(&chan->vc);
 }
 
@@ -972,6 +1016,8 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
 		chan->configured = false;
 		chan->request = EDMA_REQ_NONE;
 		chan->status = EDMA_ST_IDLE;
+		INIT_WORK(&chan->irq_work, dw_edma_irq_work);
+		atomic_set(&chan->irq_pending, 0);
 
 		if (chan->dir == EDMA_DIR_WRITE)
 			chan->ll_max = (chip->ll_region_wr[chan->id].sz / EDMA_LL_SZ);
@@ -1185,10 +1231,21 @@ int dw_edma_probe(struct dw_edma_chip *chip)
 	/* Disable eDMA, only to establish the ideal initial conditions */
 	dw_edma_core_off(dw);
 
+	/*
+	 * Deferred IRQ works are queued from the hard IRQ handlers, so the
+	 * workqueue must exist before any IRQ is requested.
+	 */
+	dw->wq = alloc_workqueue("dw-edma:%s", WQ_UNBOUND | WQ_HIGHPRI, 0,
+				 dev_name(chip->dev));
+	if (!dw->wq)
+		return -ENOMEM;
+
 	/* Request IRQs */
 	err = dw_edma_irq_request(dw, &wr_alloc, &rd_alloc);
-	if (err)
+	if (err) {
+		destroy_workqueue(dw->wq);
 		return err;
+	}
 
 	/* Allocate a dedicated virtual IRQ for interrupt-emulation doorbells */
 	err = dw_edma_emul_irq_alloc(dw);
@@ -1211,6 +1268,7 @@ int dw_edma_probe(struct dw_edma_chip *chip)
 	for (i = (dw->nr_irqs - 1); i >= 0; i--)
 		free_irq(chip->ops->irq_vector(dev, i), &dw->irq[i]);
 	dw_edma_emul_irq_free(dw);
+	destroy_workqueue(dw->wq);
 
 	return err;
 }
@@ -1235,6 +1293,11 @@ int dw_edma_remove(struct dw_edma_chip *chip)
 		free_irq(chip->ops->irq_vector(dev, i), &dw->irq[i]);
 	dw_edma_emul_irq_free(dw);
 
+	for (i = 0; i < dw->wr_ch_cnt + dw->rd_ch_cnt; i++)
+		cancel_work_sync(&dw->chan[i].irq_work);
+
+	destroy_workqueue(dw->wq);
+
 	/* Deregister eDMA device */
 	dma_async_device_unregister(&dw->dma);
 	list_for_each_entry_safe(chan, _chan, &dw->dma.channels,
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 6474cacf7195..a6a9ed09fe1b 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -9,8 +9,10 @@
 #ifndef _DW_EDMA_CORE_H
 #define _DW_EDMA_CORE_H
 
+#include <linux/atomic.h>
 #include <linux/msi.h>
 #include <linux/dma/edma.h>
+#include <linux/workqueue.h>
 
 #include "../virt-dma.h"
 
@@ -87,6 +89,9 @@ struct dw_edma_chan {
 
 	struct dma_slave_config		config;
 	bool				non_ll;
+
+	struct work_struct		irq_work;
+	atomic_t			irq_pending;
 };
 
 struct dw_edma_irq {
@@ -109,6 +114,13 @@ struct dw_edma {
 
 	struct dw_edma_chan		*chan;
 
+	/*
+	 * Deferred channel IRQ handling. WQ_HIGHPRI keeps
+	 * completion processing from starving behind saturated user load;
+	 * WQ_UNBOUND spreads per-channel works across CPUs.
+	 */
+	struct workqueue_struct		*wq;
+
 	raw_spinlock_t			lock;		/* Protect v0 shared registers */
 
 	struct dw_edma_chip             *chip;
-- 
2.51.0

Re: [PATCH 1/7] dmaengine: dw-edma: Fix HDMA channel status register access

[Frank Li]

On Fri, Jul 10, 2026 at 05:08:57PM +0900, Koichiro Den wrote:
> GET_CH_32() takes the direction before the channel ID, but
> dw_hdma_v0_core_ch_status() passed them in the opposite order. This can
> make the status callback read another HDMA channel status register.
>
> Use the same argument order as the other HDMA register accesses.
>
> Fixes: e74c39573d35 ("dmaengine: dw-edma: Add support for native HDMA")
> Cc: stable@vger.kernel.org
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---

Do you miss version number at subject?

Frank

> Changes in v2:
>   - Split out into this preparation series (was patch 02/17 of the
>     dynamic LL appends v1); no changes to the patch itself.
>   - Collect Frank's Reviewed-by.
>
>  drivers/dma/dw-edma/dw-hdma-v0-core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
> index 632abb8b481c..2beec876b184 100644
> --- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
> +++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
> @@ -79,7 +79,7 @@ static enum dma_status dw_hdma_v0_core_ch_status(struct dw_edma_chan *chan)
>  	u32 tmp;
>
>  	tmp = FIELD_GET(HDMA_V0_CH_STATUS_MASK,
> -			GET_CH_32(dw, chan->id, chan->dir, ch_stat));
> +			GET_CH_32(dw, chan->dir, chan->id, ch_stat));
>
>  	if (tmp == 1)
>  		return DMA_IN_PROGRESS;
> --
> 2.51.0
>

Re: [PATCH 2/7] dmaengine: dw-edma: Terminate STOP requests without callbacks

[Frank Li]

On Fri, Jul 10, 2026 at 05:08:58PM +0900, Koichiro Den wrote:
> The STOP request path handles device_terminate_all(). The DMA Engine
> client documentation says in the "Terminate APIs" section of
> Documentation/driver-api/dmaengine/client.rst:
>
> "No callback functions will be called for any incomplete transfers."
>
> dw-edma used vchan_cookie_complete() for a stopped descriptor. This
> queues the descriptor on the completed list and schedules its callback.
> A late callback after dmaengine_terminate_sync() can dereference
> callback state, such as a request object, that the client has already
> freed.
>
> Move the stopped descriptor to the terminated list. Complete the cookie
> before doing so, so cookie polling observes that the transfer is no
> longer in flight, but do not schedule the completion callback. Add a
> synchronize callback so virt-dma can release terminated descriptors.
>
> Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---

Reviewed-by: Frank Li <Frank.Li@nxp.com>

> Changes in v2:
>   - Split out into this preparation series (was patch 03/17 of
>     the dynamic LL appends v1).
>   - No changes.
>
>  drivers/dma/dw-edma/dw-edma-core.c | 18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index 89a4c498a17b..4e0dc52397e2 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -201,6 +201,13 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
>  	return 1;
>  }
>
> +static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
> +{
> +	list_del(&vd->node);
> +	dma_cookie_complete(&vd->tx);
> +	vchan_terminate_vdesc(vd);
> +}
> +
>  static void dw_edma_device_caps(struct dma_chan *dchan,
>  				struct dma_slave_caps *caps)
>  {
> @@ -673,8 +680,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
>  			break;
>
>  		case EDMA_REQ_STOP:
> -			list_del(&vd->node);
> -			vchan_cookie_complete(vd);
> +			dw_edma_terminate_vdesc(vd);
>  			chan->request = EDMA_REQ_NONE;
>  			chan->status = EDMA_ST_IDLE;
>  			break;
> @@ -856,6 +862,13 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
>  	return 0;
>  }
>
> +static void dw_edma_device_synchronize(struct dma_chan *dchan)
> +{
> +	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +
> +	vchan_synchronize(&chan->vc);
> +}
> +
>  static void dw_edma_free_chan_resources(struct dma_chan *dchan)
>  {
>  	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
> @@ -968,6 +981,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
>  	dma->device_pause = dw_edma_device_pause;
>  	dma->device_resume = dw_edma_device_resume;
>  	dma->device_terminate_all = dw_edma_device_terminate_all;
> +	dma->device_synchronize = dw_edma_device_synchronize;
>  	dma->device_issue_pending = dw_edma_device_issue_pending;
>  	dma->device_tx_status = dw_edma_device_tx_status;
>  	dma->device_prep_slave_sg = dw_edma_device_prep_slave_sg;
> --
> 2.51.0
>

Re: [PATCH 3/7] dmaengine: dw-edma: Clean up vchan descriptors on termination

[Frank Li]

On Fri, Jul 10, 2026 at 05:08:59PM +0900, Koichiro Den wrote:
> dw-edma resets channel state from terminate_all() paths, but pending
> virt-dma descriptors can remain on the submitted and issued lists. A later
> issue_pending() may then restart work that the client already terminated,
> possibly into buffers that were already reused. Descriptors that are never
> restarted leak instead.
>
> Move issued and submitted descriptors to the terminated list whenever a
> termination request completes. Also release virt-dma resources from
> free_chan_resources().
>
> If termination was deferred because the channel was still running, wait
> until the STOP path deconfigures the channel before synchronizing or
> freeing virt-dma resources. Otherwise dmaengine_terminate_sync() can return
> before the deferred STOP cleanup has moved issued descriptors to the
> terminated list and before the channel is known to have stopped.
>
> The old free_chan_resources() loop usually broke as soon as terminate_all()
> returned zero, so it did not effectively spin until the timeout. This wait
> can now last until the existing timeout, so use cond_resched() instead of
> busy-polling with cpu_relax(), and warn if the timeout expires.
>
> Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---

Reviewed-by: Frank Li <Frank.Li@nxp.com>

> Changes in v2:
>   - Poll with usleep_range() (and include linux/delay.h for it) instead
>     of a cond_resched() busy loop in the termination wait; each
>     iteration does an MMIO read that is a non-posted round trip on
>     remote setups.
>   - Split out into this preparation series (was patch 04/17 of
>     the dynamic LL appends v1).
>   - Let dw_edma_free_chan_resources() reuse dw_edma_device_synchronize()
>     instead of open-coding the same wait-and-synchronize sequence.
>
>  drivers/dma/dw-edma/dw-edma-core.c | 79 ++++++++++++++++++++++++------
>  1 file changed, 65 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index 4e0dc52397e2..1b493c104a5b 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -7,6 +7,7 @@
>   */
>
>  #include <linux/module.h>
> +#include <linux/delay.h>
>  #include <linux/device.h>
>  #include <linux/kernel.h>
>  #include <linux/dmaengine.h>
> @@ -15,6 +16,7 @@
>  #include <linux/irq.h>
>  #include <linux/dma/edma.h>
>  #include <linux/dma-mapping.h>
> +#include <linux/sched.h>
>  #include <linux/string_choices.h>
>
>  #include "dw-edma-core.h"
> @@ -208,6 +210,28 @@ static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
>  	vchan_terminate_vdesc(vd);
>  }
>
> +static void dw_edma_terminate_vdesc_list(struct list_head *head)
> +{
> +	struct virt_dma_desc *vd, *_vd;
> +
> +	list_for_each_entry_safe(vd, _vd, head, node)
> +		dw_edma_terminate_vdesc(vd);
> +}
> +
> +/* Must be called with vc.lock held. */
> +static void dw_edma_terminate_all_descs(struct dw_edma_chan *chan)
> +{
> +	/*
> +	 * This order must not be reversed. Cookies are assigned when
> +	 * descriptors are submitted, so desc_issued contains older cookies
> +	 * than desc_submitted. Completing desc_submitted first could move
> +	 * chan->vc.chan.completed_cookie backwards when desc_issued is
> +	 * terminated afterwards.
> +	 */
> +	dw_edma_terminate_vdesc_list(&chan->vc.desc_issued);
> +	dw_edma_terminate_vdesc_list(&chan->vc.desc_submitted);
> +}
> +
>  static void dw_edma_device_caps(struct dma_chan *dchan,
>  				struct dma_slave_caps *caps)
>  {
> @@ -313,20 +337,25 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
>  static int dw_edma_device_terminate_all(struct dma_chan *dchan)
>  {
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +	unsigned long flags;
>  	int err = 0;
>
> +	spin_lock_irqsave(&chan->vc.lock, flags);
>  	if (!chan->configured) {
> -		/* Do nothing */
> +		dw_edma_terminate_all_descs(chan);
>  	} else if (chan->status == EDMA_ST_PAUSE) {
> +		dw_edma_terminate_all_descs(chan);
>  		chan->status = EDMA_ST_IDLE;
>  		chan->configured = false;
>  	} else if (chan->status == EDMA_ST_IDLE) {
> +		dw_edma_terminate_all_descs(chan);
>  		chan->configured = false;
>  	} else if (dw_edma_core_ch_status(chan) == DMA_COMPLETE) {
>  		/*
>  		 * The channel is in a false BUSY state, probably didn't
>  		 * receive or lost an interrupt
>  		 */
> +		dw_edma_terminate_all_descs(chan);
>  		chan->status = EDMA_ST_IDLE;
>  		chan->configured = false;
>  	} else if (chan->request > EDMA_REQ_PAUSE) {
> @@ -334,6 +363,7 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
>  	} else {
>  		chan->request = EDMA_REQ_STOP;
>  	}
> +	spin_unlock_irqrestore(&chan->vc.lock, flags);
>
>  	return err;
>  }
> @@ -680,7 +710,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
>  			break;
>
>  		case EDMA_REQ_STOP:
> -			dw_edma_terminate_vdesc(vd);
> +			dw_edma_terminate_all_descs(chan);
>  			chan->request = EDMA_REQ_NONE;
>  			chan->status = EDMA_ST_IDLE;
>  			break;
> @@ -862,28 +892,49 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
>  	return 0;
>  }
>
> +static void dw_edma_wait_termination(struct dma_chan *dchan)
> +{
> +	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
> +	unsigned long flags;
> +	bool configured = true;
> +
> +	/*
> +	 * dw_edma_device_terminate_all() may defer cleanup to a later interrupt
> +	 * while the channel is still running. Retry until the channel is
> +	 * deconfigured, which marks that termination completed.
> +	 */
> +	while (time_before(jiffies, timeout)) {
> +		dw_edma_device_terminate_all(dchan);
> +
> +		spin_lock_irqsave(&chan->vc.lock, flags);
> +		configured = chan->configured;
> +		spin_unlock_irqrestore(&chan->vc.lock, flags);
> +		if (!configured)
> +			return;
> +
> +		usleep_range(1000, 2000);
> +		cond_resched();
> +	}
> +
> +	dev_warn(chan->dw->chip->dev,
> +		 "timeout waiting for channel termination\n");
> +}
> +
>  static void dw_edma_device_synchronize(struct dma_chan *dchan)
>  {
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
>
> +	dw_edma_wait_termination(dchan);
>  	vchan_synchronize(&chan->vc);
>  }
>
>  static void dw_edma_free_chan_resources(struct dma_chan *dchan)
>  {
> -	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
> -	int ret;
> -
> -	while (time_before(jiffies, timeout)) {
> -		ret = dw_edma_device_terminate_all(dchan);
> -		if (!ret)
> -			break;
> -
> -		if (time_after_eq(jiffies, timeout))
> -			return;
> +	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
>
> -		cpu_relax();
> -	}
> +	dw_edma_device_synchronize(dchan);
> +	vchan_free_chan_resources(&chan->vc);
>  }
>
>  static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
> --
> 2.51.0
>

Re: [PATCH 5/7] dmaengine: dw-edma-pcie: Drop redundant pci_free_irq_vectors()

[Frank Li]

On Fri, Jul 10, 2026 at 05:09:01PM +0900, Koichiro Den wrote:
> dw_edma_pcie enables the PCI device with pcim_enable_device(), so IRQ
> vectors allocated by pci_alloc_irq_vectors() are released by
> pcim_msi_release() on device release. The driver should not call
> pci_free_irq_vectors() manually.
>
> Drop the redundant remove-time cleanup and rely on the managed PCI
> device lifetime instead, as documented by commit 03e4905402ae ("PCI/MSI:
> Clarify pci_free_irq_vectors() usage for managed devices").
>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---

Reviewed-by: Frank Li <Frank.Li@nxp.com>

> Changes in v2:
>   - New patch in v2, posted as part of this preparation series.
>
>  drivers/dma/dw-edma/dw-edma-pcie.c | 3 ---
>  1 file changed, 3 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-pcie.c b/drivers/dma/dw-edma/dw-edma-pcie.c
> index 791c46e8ae4c..5e81a433a957 100644
> --- a/drivers/dma/dw-edma/dw-edma-pcie.c
> +++ b/drivers/dma/dw-edma/dw-edma-pcie.c
> @@ -555,9 +555,6 @@ static void dw_edma_pcie_remove(struct pci_dev *pdev)
>  	err = dw_edma_remove(chip);
>  	if (err)
>  		pci_warn(pdev, "can't remove device properly: %d\n", err);
> -
> -	/* Freeing IRQs */
> -	pci_free_irq_vectors(pdev);
>  }
>
>  static const struct pci_device_id dw_edma_pcie_id_table[] = {
> --
> 2.51.0
>

Re: [PATCH 6/7] dmaengine: dw-edma: Snapshot the v0 interrupt status once per handler pass

[Frank Li]

On Fri, Jul 10, 2026 at 05:09:02PM +0900, Koichiro Den wrote:
> The v0 interrupt handler reads the interrupt status register twice per
> invocation, once through the DONE accessor and once through the ABORT
> accessor, although both fields live in the same 32-bit register. On
> remote setups (dw-edma-pcie) each read is a non-posted round trip across
> the PCIe link costing on the order of a microsecond, and with one
> completion interrupt per element the duplicate adds up. As an example,
> profiling the R-Car S4 remote path put the handler at ~7us per
> invocation, dominated by such reads.
>
> Read the register once and derive the DONE and ABORT views from the
> snapshot. No abort is lost to this because the pass only clears status
> bits it observed, so an abort raised after the snapshot keeps its status
> and its own interrupt delivery brings it to the next pass. A second
> abort on an observed channel cannot race the clear either, as an aborted
> channel stays halted until software restarts it, and any restart follows
> the abort() handling, which comes after
> dw_edma_v0_core_clear_abort_int().
>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---

Reviewed-by: Frank Li <Frank.Li@nxp.com>

> Changes in v2:
>   - New patch in v2, posted as part of this preparation series.
>
>  drivers/dma/dw-edma/dw-edma-v0-core.c | 28 +++++++++++++--------------
>  1 file changed, 13 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
> index cfdd6463252e..377812eaa110 100644
> --- a/drivers/dma/dw-edma/dw-edma-v0-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
> @@ -218,18 +218,6 @@ static void dw_edma_v0_core_clear_abort_int(struct dw_edma_chan *chan)
>  		  FIELD_PREP(EDMA_V0_ABORT_INT_MASK, BIT(chan->id)));
>  }
>
> -static u32 dw_edma_v0_core_status_done_int(struct dw_edma *dw, enum dw_edma_dir dir)
> -{
> -	return FIELD_GET(EDMA_V0_DONE_INT_MASK,
> -			 GET_RW_32(dw, dir, int_status));
> -}
> -
> -static u32 dw_edma_v0_core_status_abort_int(struct dw_edma *dw, enum dw_edma_dir dir)
> -{
> -	return FIELD_GET(EDMA_V0_ABORT_INT_MASK,
> -			 GET_RW_32(dw, dir, int_status));
> -}
> -
>  static irqreturn_t
>  dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
>  			   dw_edma_handler_t done, dw_edma_handler_t abort)
> @@ -239,7 +227,7 @@ dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
>  	irqreturn_t ret = IRQ_NONE;
>  	struct dw_edma_chan *chan;
>  	unsigned long off;
> -	u32 mask;
> +	u32 mask, sts;
>
>  	if (dir == EDMA_DIR_WRITE) {
>  		total = dw->wr_ch_cnt;
> @@ -251,7 +239,17 @@ dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
>  		mask = dw_irq->rd_mask;
>  	}
>
> -	val = dw_edma_v0_core_status_done_int(dw, dir);
> +	/*
> +	 * DONE and ABORT status share one register, and on remote setups
> +	 * every read is a non-posted round trip across the PCIe link. Take
> +	 * one snapshot and derive both views from it. An abort raised
> +	 * after the snapshot is deferred, not lost: only bits observed in
> +	 * the snapshot are ever cleared below, so its status survives for
> +	 * the next invocation, which its own interrupt delivery triggers.
> +	 */
> +	sts = GET_RW_32(dw, dir, int_status);
> +
> +	val = FIELD_GET(EDMA_V0_DONE_INT_MASK, sts);
>  	val &= mask;
>  	for_each_set_bit(pos, &val, total) {
>  		chan = &dw->chan[pos + off];
> @@ -262,7 +260,7 @@ dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
>  		ret = IRQ_HANDLED;
>  	}
>
> -	val = dw_edma_v0_core_status_abort_int(dw, dir);
> +	val = FIELD_GET(EDMA_V0_ABORT_INT_MASK, sts);
>  	val &= mask;
>  	for_each_set_bit(pos, &val, total) {
>  		chan = &dw->chan[pos + off];
> --
> 2.51.0
>

Re: [PATCH 7/7] dmaengine: dw-edma: Defer channel IRQ handling to workqueue

[Frank Li]

On Fri, Jul 10, 2026 at 05:09:03PM +0900, Koichiro Den wrote:
> On some SoCs (e.g. R-Car S4) the endpoint-side eDMA raises a single
> fixed SPI that is hardwired to CPU0 and covers every read and write
> channel. Handling channel events directly in that hard IRQ context
> serializes the completion processing of all channels on one CPU:
> descriptor recycling and refill, client callbacks (the vchan tasklet
> runs on the scheduling CPU) and the doorbell writes all funnel through
> CPU0, while the handler additionally spins on each channel's vc.lock.
> Especially under multi-channels heavy load, the contention becomes a
> performance bottleneck.
>
> Keep the hard IRQ handler minimal and have it just clear the status and
> dispatch, defer the per-channel processing to work items. A work item
> per channel preserves per-channel ordering while letting the channels be
> processed in parallel on any CPU.
>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---

It looks good, allen try convert bh tasklet to workqueue.
https://lore.kernel.org/dmaengine/CAOMdWSLVk136RzEyiN76zqk65VLwYms0hCDi5Kww9FQppie12A@mail.gmail.com/

Please double check if it can apply to dw-edma?

Frank

> Changes in v2:
>   - New patch in v2, posted as part of this preparation series.
>
>  drivers/dma/dw-edma/dw-edma-core.c | 73 ++++++++++++++++++++++++++++--
>  drivers/dma/dw-edma/dw-edma-core.h | 12 +++++
>  2 files changed, 80 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index 5664421c6f15..704d8f9746e8 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -31,6 +31,11 @@ struct dw_edma_desc *vd2dw_edma_desc(struct virt_dma_desc *vd)
>  	return container_of(vd, struct dw_edma_desc, vd);
>  }
>
> +enum dw_edma_irq_event {
> +	DW_EDMA_IRQ_DONE	= BIT(0),
> +	DW_EDMA_IRQ_ABORT	= BIT(1),
> +};
> +
>  static inline
>  u64 dw_edma_get_pci_address(struct dw_edma_chan *chan, phys_addr_t cpu_addr)
>  {
> @@ -748,6 +753,44 @@ static void dw_edma_abort_interrupt(struct dw_edma_chan *chan)
>  	chan->status = EDMA_ST_IDLE;
>  }
>
> +static void dw_edma_irq_work(struct work_struct *work)
> +{
> +	struct dw_edma_chan *chan = container_of(work, struct dw_edma_chan,
> +						 irq_work);
> +	unsigned int events;
> +
> +	do {
> +		events = atomic_xchg(&chan->irq_pending, 0);
> +
> +		if (events & DW_EDMA_IRQ_DONE)
> +			dw_edma_done_interrupt(chan);
> +		if (events & DW_EDMA_IRQ_ABORT)
> +			dw_edma_abort_interrupt(chan);
> +		/*
> +		 * Correctness does not depend on this loop: queue_work() can
> +		 * requeue once the work item starts running. Staying here just
> +		 * coalesces back-to-back channel events into one wakeup.
> +		 */
> +	} while (atomic_read(&chan->irq_pending));
> +}
> +
> +static void dw_edma_queue_irq_work(struct dw_edma_chan *chan,
> +				   enum dw_edma_irq_event event)
> +{
> +	atomic_or(event, &chan->irq_pending);
> +	queue_work(chan->dw->wq, &chan->irq_work);
> +}
> +
> +static void dw_edma_done_interrupt_deferred(struct dw_edma_chan *chan)
> +{
> +	dw_edma_queue_irq_work(chan, DW_EDMA_IRQ_DONE);
> +}
> +
> +static void dw_edma_abort_interrupt_deferred(struct dw_edma_chan *chan)
> +{
> +	dw_edma_queue_irq_work(chan, DW_EDMA_IRQ_ABORT);
> +}
> +
>  static void dw_edma_emul_irq_ack(struct irq_data *d)
>  {
>  	struct dw_edma *dw = irq_data_get_irq_chip_data(d);
> @@ -842,8 +885,8 @@ static inline irqreturn_t dw_edma_interrupt_write_inner(int irq, void *data)
>  	struct dw_edma_irq *dw_irq = data;
>
>  	return dw_edma_core_handle_int(dw_irq, EDMA_DIR_WRITE,
> -				       dw_edma_done_interrupt,
> -				       dw_edma_abort_interrupt);
> +				       dw_edma_done_interrupt_deferred,
> +				       dw_edma_abort_interrupt_deferred);
>  }
>
>  static inline irqreturn_t dw_edma_interrupt_read_inner(int irq, void *data)
> @@ -851,8 +894,8 @@ static inline irqreturn_t dw_edma_interrupt_read_inner(int irq, void *data)
>  	struct dw_edma_irq *dw_irq = data;
>
>  	return dw_edma_core_handle_int(dw_irq, EDMA_DIR_READ,
> -				       dw_edma_done_interrupt,
> -				       dw_edma_abort_interrupt);
> +				       dw_edma_done_interrupt_deferred,
> +				       dw_edma_abort_interrupt_deferred);
>  }
>
>  static inline irqreturn_t dw_edma_interrupt_write(int irq, void *data)
> @@ -930,6 +973,7 @@ static void dw_edma_device_synchronize(struct dma_chan *dchan)
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
>
>  	dw_edma_wait_termination(dchan);
> +	cancel_work_sync(&chan->irq_work);
>  	vchan_synchronize(&chan->vc);
>  }
>
> @@ -972,6 +1016,8 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
>  		chan->configured = false;
>  		chan->request = EDMA_REQ_NONE;
>  		chan->status = EDMA_ST_IDLE;
> +		INIT_WORK(&chan->irq_work, dw_edma_irq_work);
> +		atomic_set(&chan->irq_pending, 0);
>
>  		if (chan->dir == EDMA_DIR_WRITE)
>  			chan->ll_max = (chip->ll_region_wr[chan->id].sz / EDMA_LL_SZ);
> @@ -1185,10 +1231,21 @@ int dw_edma_probe(struct dw_edma_chip *chip)
>  	/* Disable eDMA, only to establish the ideal initial conditions */
>  	dw_edma_core_off(dw);
>
> +	/*
> +	 * Deferred IRQ works are queued from the hard IRQ handlers, so the
> +	 * workqueue must exist before any IRQ is requested.
> +	 */
> +	dw->wq = alloc_workqueue("dw-edma:%s", WQ_UNBOUND | WQ_HIGHPRI, 0,
> +				 dev_name(chip->dev));
> +	if (!dw->wq)
> +		return -ENOMEM;
> +
>  	/* Request IRQs */
>  	err = dw_edma_irq_request(dw, &wr_alloc, &rd_alloc);
> -	if (err)
> +	if (err) {
> +		destroy_workqueue(dw->wq);
>  		return err;
> +	}
>
>  	/* Allocate a dedicated virtual IRQ for interrupt-emulation doorbells */
>  	err = dw_edma_emul_irq_alloc(dw);
> @@ -1211,6 +1268,7 @@ int dw_edma_probe(struct dw_edma_chip *chip)
>  	for (i = (dw->nr_irqs - 1); i >= 0; i--)
>  		free_irq(chip->ops->irq_vector(dev, i), &dw->irq[i]);
>  	dw_edma_emul_irq_free(dw);
> +	destroy_workqueue(dw->wq);
>
>  	return err;
>  }
> @@ -1235,6 +1293,11 @@ int dw_edma_remove(struct dw_edma_chip *chip)
>  		free_irq(chip->ops->irq_vector(dev, i), &dw->irq[i]);
>  	dw_edma_emul_irq_free(dw);
>
> +	for (i = 0; i < dw->wr_ch_cnt + dw->rd_ch_cnt; i++)
> +		cancel_work_sync(&dw->chan[i].irq_work);
> +
> +	destroy_workqueue(dw->wq);
> +
>  	/* Deregister eDMA device */
>  	dma_async_device_unregister(&dw->dma);
>  	list_for_each_entry_safe(chan, _chan, &dw->dma.channels,
> diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
> index 6474cacf7195..a6a9ed09fe1b 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.h
> +++ b/drivers/dma/dw-edma/dw-edma-core.h
> @@ -9,8 +9,10 @@
>  #ifndef _DW_EDMA_CORE_H
>  #define _DW_EDMA_CORE_H
>
> +#include <linux/atomic.h>
>  #include <linux/msi.h>
>  #include <linux/dma/edma.h>
> +#include <linux/workqueue.h>
>
>  #include "../virt-dma.h"
>
> @@ -87,6 +89,9 @@ struct dw_edma_chan {
>
>  	struct dma_slave_config		config;
>  	bool				non_ll;
> +
> +	struct work_struct		irq_work;
> +	atomic_t			irq_pending;
>  };
>
>  struct dw_edma_irq {
> @@ -109,6 +114,13 @@ struct dw_edma {
>
>  	struct dw_edma_chan		*chan;
>
> +	/*
> +	 * Deferred channel IRQ handling. WQ_HIGHPRI keeps
> +	 * completion processing from starving behind saturated user load;
> +	 * WQ_UNBOUND spreads per-channel works across CPUs.
> +	 */
> +	struct workqueue_struct		*wq;
> +
>  	raw_spinlock_t			lock;		/* Protect v0 shared registers */
>
>  	struct dw_edma_chip             *chip;
> --
> 2.51.0
>