[PATCH 00/17] dmaengine: dw-edma: Support dynamic LL appends

[PATCH 00/17] dmaengine: dw-edma: Support dynamic LL appends

[Koichiro Den]

Hi,

This series is a reworked version of Frank's earlier RFT series:

  https://lore.kernel.org/dmaengine/20260109-edma_dymatic-v1-0-9a98c9c98536@nxp.com/

After discussing the HDMA test results with Frank, I am sending this as a
standalone series that keeps the main dynamic-append direction, while adding the
fixes and HDMA handling needed to make it work reliably on both eDMA and HDMA.

Several patches are kept from, or based on, Frank's RFT series; the individual
patches carry the corresponding attribution.

The series has been tested on both eDMA and HDMA systems. Both completed the fio
test set reliably; performance results are shown below.


Dependencies
============

1). [PATCH v7 0/9] dmaengine: Add new API to combine configuration and descriptor preparation
    https://lore.kernel.org/dmaengine/20260521-dma_prep_config-v7-0-1f73f4899883@nxp.com/

2). [PATCH v2 00/11] dmaengine: dw-edma: flatten desc structions and simple code
    https://lore.kernel.org/dmaengine/20260109-edma_ll-v2-0-5c0b27b2c664@nxp.com/


Performance measurements
========================

"Before" means the dependency series applied, without this series.
"After" means the same tree plus this series.

The fio test cases follow the set used in Frank's original RFT series.
Each full fio test set was run three times in alternating order (B-A-B-A-B-A),
with runtime=30s and ramp_time=5s. The tables below report mean bandwidth; the
detailed per-test rows also include standard deviation.

Note:
- These results are from one eDMA platform and one HDMA platform, so the exact
  deltas should NOT be read as generic numbers for all dw-edma integrations.
- Both endpoint setups used nvmet_pci_epf with a namespace backed by a
  null_blk device.

Summary by group (BW delta %)

              all    read   write    qd32      q1  small 4K  large >=128K
  eDMA      +54.6   +46.5   +66.3   +56.1   +53.5     +82.0         +46.3
  HDMA       +9.0    +5.5   +14.1   +14.9    -0.7     +24.5          +4.3

The eDMA setup shows broad improvement across the test set. On HDMA, the main
gains are in high queue-depth and small-block write cases; low queue-depth cases
are mostly neutral, with some run-to-run noise. For HDMA, watermark interrupts
are needed to obtain a reliable running HDMA_LLP_* progress point. They can be
mostly overhead for low queue-depth workloads where the current descriptor fits
in the LL ring and there is no later descriptor to append.


eDMA:
  - Testbed:
    * Endpoint: RK3588 (Rock 5B)
      controller IP version: v5.60a
      ll_max: 170

  - Summary by group (BW delta %)

    all          n=26 mean= +54.6 median= +38.4 min= +16.3 max=+119.0
    read         n=14 mean= +46.5 median= +37.5 min= +18.7 max=+119.0
    write        n=11 mean= +66.3 median= +68.1 min= +16.3 max=+117.2
    qd32         n=16 mean= +56.1 median= +46.8 min= +18.7 max=+117.2
    q1           n= 9 mean= +53.5 median= +36.8 min= +16.3 max=+119.0
    small 4K     n= 6 mean= +82.0 median= +93.6 min= +18.7 max=+117.2
    large >=128K n=20 mean= +46.3 median= +37.6 min= +16.3 max=+119.0

  - Before mean -> After mean (MiB/s)

    Case                         Before             After              Delta
    ---------------------------  -----------------  -----------------  ------
    Rnd read     4KB q1  1j          22.7 (sd 7.7)     48.3 (sd 11.3)  +112.8%
    Rnd read     4KB q32 1j        206.3 (sd 23.8)    245.0 (sd 21.7)   +18.7%
    Rnd read     4KB q32 4j        213.3 (sd 28.0)    332.7 (sd 45.6)   +55.9%
    Rnd read   128KB q1  1j       512.7 (sd 193.6)   644.0 (sd 152.8)   +25.6%
    Rnd read   128KB q32 1j       2285.7 (sd 15.5)    3071.7 (sd 4.2)   +34.4%
    Rnd read   128KB q32 4j        2392.0 (sd 6.1)    3290.0 (sd 1.0)   +37.5%
    Rnd read   512KB q1  1j         634.3 (sd 7.8)    788.7 (sd 15.2)   +24.3%
    Rnd read   512KB q32 1j        2388.7 (sd 5.5)    3282.0 (sd 2.6)   +37.4%
    Rnd read   512KB q32 4j        2391.7 (sd 5.5)    3293.0 (sd 0.0)   +37.7%
    Rnd write    4KB q1  1j         24.4 (sd 10.2)     42.8 (sd 13.2)   +75.8%
    Rnd write    4KB q32 1j        109.0 (sd 13.0)    230.3 (sd 27.1)  +111.3%
    Rnd write    4KB q32 4j        110.3 (sd 14.4)    239.7 (sd 34.4)  +117.2%
    Rnd write  128KB q1  1j        339.0 (sd 41.1)   498.7 (sd 102.9)   +47.1%
    Rnd write  128KB q32 1j       1027.3 (sd 33.5)   1617.0 (sd 14.8)   +57.4%
    Rnd write  128KB q32 4j        951.3 (sd 72.6)    1599.0 (sd 3.6)   +68.1%
    Seq read   128KB q1  1j       379.7 (sd 120.1)    831.3 (sd 89.9)  +119.0%
    Seq read   128KB q32 1j        2291.7 (sd 6.1)   3091.3 (sd 22.8)   +34.9%
    Seq read   512KB q1  1j        644.7 (sd 34.4)    882.0 (sd 28.5)   +36.8%
    Seq read   512KB q32 1j        2387.7 (sd 5.7)    3284.0 (sd 2.6)   +37.5%
    Seq read     1MB q32 1j        2390.0 (sd 5.3)    3292.3 (sd 2.1)   +37.8%
    Seq write  128KB q1  1j        354.0 (sd 88.4)    438.0 (sd 65.1)   +23.7%
    Seq write  128KB q32 1j        934.3 (sd 46.0)   1620.0 (sd 15.6)   +73.4%
    Seq write  512KB q1  1j        552.7 (sd 14.6)    642.7 (sd 38.1)   +16.3%
    Seq write  512KB q32 1j       1041.0 (sd 39.5)    1621.3 (sd 1.5)   +55.7%
    Seq write    1MB q32 1j        808.3 (sd 22.7)    1479.7 (sd 3.5)   +83.1%
    Rnd rdwr  4K..1MB q8  4j       846.7 (sd 18.8)   1177.7 (sd 23.1)   +39.1%

HDMA:
  - Testbed:
    * Endpoint: SpacemiT K3
      controller IP version: v6.30a
      ll_max: 170

  - Summary by group (BW delta %)

    all          n=26 mean=  +9.0 median=  +6.9 min= -15.2 max= +50.2
    read         n=14 mean=  +5.5 median=  +6.4 min= -15.2 max= +24.0
    write        n=11 mean= +14.1 median=  +9.0 min=  -0.2 max= +50.2
    qd32         n=16 mean= +14.9 median=  +9.1 min=  +5.7 max= +50.2
    q1           n= 9 mean=  -0.7 median=  +0.2 min= -15.2 max=  +5.2
    small 4K     n= 6 mean= +24.5 median= +21.5 min=  -0.2 max= +50.2
    large >=128K n=20 mean=  +4.3 median=  +6.4 min= -15.2 max=  +9.8

  - Before mean -> After mean (MiB/s)

    Case                         Before             After              Delta
    ---------------------------  -----------------  -----------------  ------
    Rnd read     4KB q1  1j          68.5 (sd 5.7)      72.0 (sd 6.8)    +5.1%
    Rnd read     4KB q32 1j        310.7 (sd 38.0)    385.3 (sd 43.6)   +24.0%
    Rnd read     4KB q32 4j        324.0 (sd 45.1)     385.7 (sd 9.5)   +19.0%
    Rnd read   128KB q1  1j        737.7 (sd 63.3)    746.0 (sd 47.1)    +1.1%
    Rnd read   128KB q32 1j       1513.0 (sd 24.0)    1617.0 (sd 2.0)    +6.9%
    Rnd read   128KB q32 4j        1552.7 (sd 7.0)   1641.0 (sd 29.9)    +5.7%
    Rnd read   512KB q1  1j        828.3 (sd 16.9)    815.7 (sd 14.0)    -1.5%
    Rnd read   512KB q32 1j        1550.0 (sd 8.5)   1661.7 (sd 14.3)    +7.2%
    Rnd read   512KB q32 4j       1547.3 (sd 20.4)   1670.0 (sd 27.0)    +7.9%
    Rnd write    4KB q1  1j          67.2 (sd 5.1)      67.1 (sd 5.5)    -0.2%
    Rnd write    4KB q32 1j         207.7 (sd 6.8)     309.7 (sd 3.8)   +49.1%
    Rnd write    4KB q32 4j         208.0 (sd 5.6)     312.3 (sd 4.0)   +50.2%
    Rnd write  128KB q1  1j        545.0 (sd 42.5)    573.3 (sd 45.7)    +5.2%
    Rnd write  128KB q32 1j       1251.3 (sd 16.0)    1363.3 (sd 6.7)    +9.0%
    Rnd write  128KB q32 4j       1251.0 (sd 17.1)    1365.3 (sd 4.9)    +9.1%
    Seq read   128KB q1  1j        803.3 (sd 78.2)   681.0 (sd 110.1)   -15.2%
    Seq read   128KB q32 1j       1513.3 (sd 23.5)    1618.3 (sd 4.0)    +6.9%
    Seq read   512KB q1  1j        846.7 (sd 26.9)    797.7 (sd 73.9)    -5.8%
    Seq read   512KB q32 1j       1522.0 (sd 36.2)    1671.0 (sd 1.7)    +9.8%
    Seq read     1MB q32 1j       1544.0 (sd 21.8)   1636.3 (sd 25.1)    +6.0%
    Seq write  128KB q1  1j        544.3 (sd 13.3)    572.3 (sd 28.4)    +5.1%
    Seq write  128KB q32 1j       1251.3 (sd 15.5)    1364.3 (sd 4.9)    +9.0%
    Seq write  512KB q1  1j        772.7 (sd 23.0)    774.3 (sd 64.1)    +0.2%
    Seq write  512KB q32 1j       1251.3 (sd 17.0)    1365.0 (sd 5.2)    +9.1%
    Seq write    1MB q32 1j       1250.3 (sd 16.5)    1366.0 (sd 5.3)    +9.3%
    Rnd rdwr  4K..1MB q8  4j        875.0 (sd 9.0)     884.3 (sd 4.5)    +1.1%



Best regards,
Koichiro


Frank Li (5):
  dmaengine: dw-edma: Add dw_edma_core_ll_cur_idx() to get current LL
    entry index
  dmaengine: dw-edma: Move dw_hdma_set_callback_result() up
  dmaengine: dw-edma: Make DMA link list work as a circular buffer
  dmaengine: dw-edma: Dynamically append requests while running
  dmaengine: dw-edma: Add trace support

Koichiro Den (12):
  dmaengine: dw-edma: Fix residue burst index in tx_status()
  dmaengine: dw-edma: Fix HDMA channel status register access
  dmaengine: dw-edma: Terminate STOP requests without callbacks
  dmaengine: dw-edma: Clean up vchan descriptors on termination
  dmaengine: dw-edma: Serialize channel state checks
  dmaengine: dw-edma: Add LL interrupt placement policy
  dmaengine: dw-edma: Reclaim issued descriptors from LL progress
  dmaengine: dw-edma: Use HDMA watermarks as progress events
  dmaengine: dw-edma: Clear LL data entries on reset
  dmaengine: dw-edma: Dispatch DONE interrupts by channel request
  dmaengine: dw-edma: Reset LL state after terminate and abort
  dmaengine: dw-edma: Recover stopped HDMA from tx_status

 drivers/dma/dw-edma/Makefile          |   3 +
 drivers/dma/dw-edma/dw-edma-core.c    | 577 +++++++++++++++++++++-----
 drivers/dma/dw-edma/dw-edma-core.h    |  63 ++-
 drivers/dma/dw-edma/dw-edma-trace.c   |   4 +
 drivers/dma/dw-edma/dw-edma-trace.h   | 150 +++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c |  50 ++-
 drivers/dma/dw-edma/dw-hdma-v0-core.c | 125 +++++-
 drivers/dma/dw-edma/dw-hdma-v0-regs.h |   1 +
 8 files changed, 847 insertions(+), 126 deletions(-)
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h

-- 
2.51.0

[PATCH 01/17] dmaengine: dw-edma: Fix residue burst index in tx_status()

[Koichiro Den]

dw_edma_device_tx_status() uses desc->done_burst to subtract the
completed byte count from the descriptor size. done_burst is a count of
completed bursts, not the zero-based index of the last completed burst.

Index desc->burst[] with done_burst - 1. Otherwise tx_status() reads the
next burst's cumulative transfer size, which under-reports the residue
and can become a one-past-the-end access when all bursts have completed.

While at it, return early when txstate is NULL and drop the redundant
desc check after vd2dw_edma_desc(). These are minor clean-ups since
dma_set_residue() already tolerates a NULL state, and vd2dw_edma_desc()
is only reached for a valid vdesc.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
@Frank, if you plan to respin 20260109-edma_ll-v2-0-5c0b27b2c664@nxp.com
and agree with this patch, consider folding this fix into your patch
when submitting your v3.

 drivers/dma/dw-edma/dw-edma-core.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 1c8aef5e03b0..d99b6256660a 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -244,7 +244,7 @@ dw_edma_device_tx_status(struct dma_chan *dchan, dma_cookie_t cookie,
 		ret = DMA_PAUSED;
 
 	if (!txstate)
-		goto ret_residue;
+		return ret;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
 	vd = vchan_find_desc(&chan->vc, cookie);
@@ -252,12 +252,11 @@ dw_edma_device_tx_status(struct dma_chan *dchan, dma_cookie_t cookie,
 		desc = vd2dw_edma_desc(vd);
 
 		residue = desc->alloc_sz;
-		if (desc && desc->done_burst)
-			residue -= desc->burst[desc->done_burst].xfer_sz;
+		if (desc->done_burst)
+			residue -= desc->burst[desc->done_burst - 1].xfer_sz;
 	}
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
-ret_residue:
 	dma_set_residue(txstate, residue);
 
 	return ret;
-- 
2.51.0

[PATCH 02/17] dmaengine: dw-edma: Fix HDMA channel status register access

[Koichiro Den]

GET_CH_32() takes the direction before the channel ID, but
dw_hdma_v0_core_ch_status() passed them in the opposite order. This can
make the status callback read another HDMA channel status register.

Use the same argument order as the other HDMA register accesses.

Fixes: e74c39573d35 ("dmaengine: dw-edma: Add support for native HDMA")
Cc: stable@vger.kernel.org
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Given Devendra's comment on
a28adc76-044b-4666-bda0-d7f9a8d52a63@amd.com,
I expect he will soon submit a very similar patch. If so, please prefer
his patch over this one if it works. I included this fix here since the
rest of this series makes this pre-existing bug easier to hit.

 drivers/dma/dw-edma/dw-hdma-v0-core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index 7f9fe3a6edd9..862375c8e4ba 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -79,7 +79,7 @@ static enum dma_status dw_hdma_v0_core_ch_status(struct dw_edma_chan *chan)
 	u32 tmp;
 
 	tmp = FIELD_GET(HDMA_V0_CH_STATUS_MASK,
-			GET_CH_32(dw, chan->id, chan->dir, ch_stat));
+			GET_CH_32(dw, chan->dir, chan->id, ch_stat));
 
 	if (tmp == 1)
 		return DMA_IN_PROGRESS;
-- 
2.51.0

[PATCH 03/17] dmaengine: dw-edma: Terminate STOP requests without callbacks

[Koichiro Den]

The STOP request path handles device_terminate_all(). The DMA Engine
client documentation says in the "Terminate APIs" section of
Documentation/driver-api/dmaengine/client.rst:

  "No callback functions will be called for any incomplete transfers."

dw-edma used vchan_cookie_complete() for a stopped descriptor. This
queues the descriptor on the completed list and schedules its callback.
A late callback after dmaengine_terminate_sync() can dereference
callback state, such as a request object, that the client has already
freed.

Move stopped descriptors to the terminated list. Complete the cookie
before doing so, so cookie polling observes that the transfer is no
longer in flight, but do not schedule the completion callback. Add a
synchronize callback so virt-dma can release terminated descriptors.

Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index d99b6256660a..bedaee6d30ab 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -106,6 +106,13 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 	return 1;
 }
 
+static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
+{
+	list_del(&vd->node);
+	dma_cookie_complete(&vd->tx);
+	vchan_terminate_vdesc(vd);
+}
+
 static void dw_edma_device_caps(struct dma_chan *dchan,
 				struct dma_slave_caps *caps)
 {
@@ -537,8 +544,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 			break;
 
 		case EDMA_REQ_STOP:
-			list_del(&vd->node);
-			vchan_cookie_complete(vd);
+			dw_edma_terminate_vdesc(vd);
 			chan->request = EDMA_REQ_NONE;
 			chan->status = EDMA_ST_IDLE;
 			break;
@@ -610,6 +616,13 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
 	return 0;
 }
 
+static void dw_edma_device_synchronize(struct dma_chan *dchan)
+{
+	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+
+	vchan_synchronize(&chan->vc);
+}
+
 static void dw_edma_free_chan_resources(struct dma_chan *dchan)
 {
 	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
@@ -723,6 +736,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
 	dma->device_pause = dw_edma_device_pause;
 	dma->device_resume = dw_edma_device_resume;
 	dma->device_terminate_all = dw_edma_device_terminate_all;
+	dma->device_synchronize = dw_edma_device_synchronize;
 	dma->device_issue_pending = dw_edma_device_issue_pending;
 	dma->device_tx_status = dw_edma_device_tx_status;
 	dma->device_prep_slave_sg_config = dw_edma_device_prep_slave_sg_config;
-- 
2.51.0

[PATCH 04/17] dmaengine: dw-edma: Clean up vchan descriptors on termination

[Koichiro Den]

dw-edma resets channel state from terminate_all() paths, but pending
virt-dma descriptors can remain on the submitted and issued lists. A
later issue_pending() may then restart work that the client already
terminated, possibly into buffers that were already reused. Descriptors
that are never restarted leak instead.

Move issued and submitted descriptors to the terminated list whenever a
termination request completes. Also release virt-dma resources from
free_chan_resources().

If termination was deferred because the channel was still running, wait
until the STOP path deconfigures the channel before synchronizing or
freeing virt-dma resources. Otherwise dmaengine_terminate_sync() can
return before the deferred STOP cleanup has moved issued descriptors to
the terminated list and before the channel is known to have stopped.

The old free_chan_resources() loop usually broke as soon as
terminate_all() returned zero, so it did not effectively spin until the
timeout. This wait can now last until the existing timeout, so use
cond_resched() instead of busy-polling with cpu_relax(), and warn if the
timeout expires.

Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c | 78 ++++++++++++++++++++++++------
 1 file changed, 64 insertions(+), 14 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index bedaee6d30ab..2777dc0b2aed 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -15,6 +15,7 @@
 #include <linux/irq.h>
 #include <linux/dma/edma.h>
 #include <linux/dma-mapping.h>
+#include <linux/sched.h>
 #include <linux/string_choices.h>
 
 #include "dw-edma-core.h"
@@ -113,6 +114,28 @@ static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
 	vchan_terminate_vdesc(vd);
 }
 
+static void dw_edma_terminate_vdesc_list(struct list_head *head)
+{
+	struct virt_dma_desc *vd, *_vd;
+
+	list_for_each_entry_safe(vd, _vd, head, node)
+		dw_edma_terminate_vdesc(vd);
+}
+
+/* Must be called with vc.lock held. */
+static void dw_edma_terminate_all_descs(struct dw_edma_chan *chan)
+{
+	/*
+	 * This order must not be reversed. Cookies are assigned when
+	 * descriptors are submitted, so desc_issued contains older cookies
+	 * than desc_submitted. Completing desc_submitted first could move
+	 * chan->vc.chan.completed_cookie backwards when desc_issued is
+	 * terminated afterwards.
+	 */
+	dw_edma_terminate_vdesc_list(&chan->vc.desc_issued);
+	dw_edma_terminate_vdesc_list(&chan->vc.desc_submitted);
+}
+
 static void dw_edma_device_caps(struct dma_chan *dchan,
 				struct dma_slave_caps *caps)
 {
@@ -190,20 +213,25 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
 static int dw_edma_device_terminate_all(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long flags;
 	int err = 0;
 
+	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (!chan->configured) {
-		/* Do nothing */
+		dw_edma_terminate_all_descs(chan);
 	} else if (chan->status == EDMA_ST_PAUSE) {
+		dw_edma_terminate_all_descs(chan);
 		chan->status = EDMA_ST_IDLE;
 		chan->configured = false;
 	} else if (chan->status == EDMA_ST_IDLE) {
+		dw_edma_terminate_all_descs(chan);
 		chan->configured = false;
 	} else if (dw_edma_core_ch_status(chan) == DMA_COMPLETE) {
 		/*
 		 * The channel is in a false BUSY state, probably didn't
 		 * receive or lost an interrupt
 		 */
+		dw_edma_terminate_all_descs(chan);
 		chan->status = EDMA_ST_IDLE;
 		chan->configured = false;
 	} else if (chan->request > EDMA_REQ_PAUSE) {
@@ -211,6 +239,7 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
 	} else {
 		chan->request = EDMA_REQ_STOP;
 	}
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
 	return err;
 }
@@ -544,7 +573,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 			break;
 
 		case EDMA_REQ_STOP:
-			dw_edma_terminate_vdesc(vd);
+			dw_edma_terminate_all_descs(chan);
 			chan->request = EDMA_REQ_NONE;
 			chan->status = EDMA_ST_IDLE;
 			break;
@@ -616,28 +645,49 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
 	return 0;
 }
 
+static void dw_edma_wait_termination(struct dma_chan *dchan)
+{
+	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
+	unsigned long flags;
+	bool configured = true;
+
+	/*
+	 * dw_edma_device_terminate_all() may defer cleanup to a later interrupt
+	 * while the channel is still running. Retry until the channel is
+	 * deconfigured, which marks that termination completed.
+	 */
+	while (time_before(jiffies, timeout)) {
+		dw_edma_device_terminate_all(dchan);
+
+		spin_lock_irqsave(&chan->vc.lock, flags);
+		configured = chan->configured;
+		spin_unlock_irqrestore(&chan->vc.lock, flags);
+		if (!configured)
+			return;
+
+		cond_resched();
+	}
+
+	dev_warn(chan->dw->chip->dev,
+		 "timeout waiting for channel termination\n");
+}
+
 static void dw_edma_device_synchronize(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 
+	dw_edma_wait_termination(dchan);
 	vchan_synchronize(&chan->vc);
 }
 
 static void dw_edma_free_chan_resources(struct dma_chan *dchan)
 {
-	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
-	int ret;
-
-	while (time_before(jiffies, timeout)) {
-		ret = dw_edma_device_terminate_all(dchan);
-		if (!ret)
-			break;
-
-		if (time_after_eq(jiffies, timeout))
-			return;
+	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 
-		cpu_relax();
-	}
+	dw_edma_wait_termination(dchan);
+	vchan_synchronize(&chan->vc);
+	vchan_free_chan_resources(&chan->vc);
 }
 
 static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
-- 
2.51.0

[PATCH 05/17] dmaengine: dw-edma: Serialize channel state checks

[Koichiro Den]

pause() and resume() read and update channel state without holding
vc.lock, while the interrupt handlers update the same state under it.
Take the same lock around those state checks so that request, status,
and configured stay consistent.

For example, pause() can observe EDMA_ST_BUSY right before the interrupt
handler completes the final descriptor and moves the channel to
EDMA_ST_IDLE, and then record EDMA_REQ_PAUSE on an already idle channel.
No further interrupt will acknowledge the request, and since
issue_pending() requires EDMA_REQ_NONE, the channel is wedged for good:
terminate_all() leaves the stale request behind, so even reconfiguring
the channel does not recover it.

issue_pending() already runs under vc.lock, but it tests configured
before taking it. Move that test under the lock as well, so that the
decision to start work is made against the current value rather than one
observed before a concurrent terminate_all() deconfigured the channel.

Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 2777dc0b2aed..489f7fe49840 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -177,8 +177,10 @@ dw_edma_device_get_config(struct dma_chan *dchan,
 static int dw_edma_device_pause(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long flags;
 	int err = 0;
 
+	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (!chan->configured)
 		err = -EPERM;
 	else if (chan->status != EDMA_ST_BUSY)
@@ -187,6 +189,7 @@ static int dw_edma_device_pause(struct dma_chan *dchan)
 		err = -EPERM;
 	else
 		chan->request = EDMA_REQ_PAUSE;
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
 	return err;
 }
@@ -194,8 +197,10 @@ static int dw_edma_device_pause(struct dma_chan *dchan)
 static int dw_edma_device_resume(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	unsigned long flags;
 	int err = 0;
 
+	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (!chan->configured) {
 		err = -EPERM;
 	} else if (chan->status != EDMA_ST_PAUSE) {
@@ -206,6 +211,7 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
 		chan->status = EDMA_ST_BUSY;
 		dw_edma_start_transfer(chan);
 	}
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
 	return err;
 }
@@ -249,11 +255,9 @@ static void dw_edma_device_issue_pending(struct dma_chan *dchan)
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 	unsigned long flags;
 
-	if (!chan->configured)
-		return;
-
 	spin_lock_irqsave(&chan->vc.lock, flags);
-	if (vchan_issue_pending(&chan->vc) && chan->request == EDMA_REQ_NONE &&
+	if (chan->configured && vchan_issue_pending(&chan->vc) &&
+	    chan->request == EDMA_REQ_NONE &&
 	    chan->status == EDMA_ST_IDLE) {
 		chan->status = EDMA_ST_BUSY;
 		dw_edma_start_transfer(chan);
-- 
2.51.0

[PATCH 06/17] dmaengine: dw-edma: Add dw_edma_core_ll_cur_idx() to get current LL entry index

[Koichiro Den]

From: Frank Li <Frank.Li@nxp.com>

Add dw_edma_core_ll_cur_idx() to get the current LL entry index and
prepare for dynamic addition of DMA requests while the DMA engine is
running.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Unchanged from the original submission from Frank:
https://lore.kernel.org/dmaengine/20260109-edma_dymatic-v1-1-9a98c9c98536@nxp.com/
except one trivial typo fix in the commit message (s/dymatic/dynamic/)
and commit title/message updates.

 drivers/dma/dw-edma/dw-edma-core.h    | 10 ++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c | 17 +++++++++++++++++
 drivers/dma/dw-edma/dw-hdma-v0-core.c | 17 +++++++++++++++++
 3 files changed, 44 insertions(+)

diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 31039eb85079..d68c4592c617 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -123,6 +123,7 @@ struct dw_edma_core_ops {
 	void (*ll_data)(struct dw_edma_chan *chan, struct dw_edma_burst *burst,
 			u32 idx, bool cb, bool irq);
 	void (*ll_link)(struct dw_edma_chan *chan, u32 idx, bool cb, u64 addr);
+	int (*ll_cur_idx)(struct dw_edma_chan *chan);
 	void (*ch_doorbell)(struct dw_edma_chan *chan);
 	void (*ch_enable)(struct dw_edma_chan *chan);
 	void (*ch_config)(struct dw_edma_chan *chan);
@@ -164,6 +165,15 @@ struct dw_edma_chan *dchan2dw_edma_chan(struct dma_chan *dchan)
 	return vc2dw_edma_chan(to_virt_chan(dchan));
 }
 
+/*
+ * Get current DMA running idx.
+ * < 0 means channel have not initialized or hardware reset by PCI link lost
+ */
+static inline int dw_edma_core_ll_cur_idx(struct dw_edma_chan *chan)
+{
+	return chan->dw->core->ll_cur_idx(chan);
+}
+
 static inline u64 dw_edma_core_get_ll_paddr(struct dw_edma_chan *chan)
 {
 	if (chan->dir == EDMA_DIR_WRITE)
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index 7b4591f984ad..edc71a4dbc79 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -504,6 +504,22 @@ static void dw_edma_v0_core_ch_doorbell(struct dw_edma_chan *chan)
 		  FIELD_PREP(EDMA_V0_DOORBELL_CH_MASK, chan->id));
 }
 
+static int dw_edma_v0_core_ll_cur_idx(struct dw_edma_chan *chan)
+{
+	u64 paddr;
+	u32 val;
+
+	/* LL region never cross 4G memory boundary, so only check low 32bit */
+	val = GET_CH_32(chan->dw, chan->dir, chan->id, llp.lsb);
+	paddr = dw_edma_core_get_ll_paddr(chan);
+
+	/* DMA have not setup or DMA engine reset because PCIe link lost */
+	if (!val)
+		return -EINVAL;
+
+	return (val - (paddr & 0xFFFFFFFF)) / EDMA_LL_SZ;
+}
+
 /* eDMA debugfs callbacks */
 static void dw_edma_v0_core_debugfs_on(struct dw_edma *dw)
 {
@@ -517,6 +533,7 @@ static const struct dw_edma_core_ops dw_edma_v0_core = {
 	.handle_int = dw_edma_v0_core_handle_int,
 	.ll_data = dw_edma_v0_core_ll_data,
 	.ll_link = dw_edma_v0_core_ll_link,
+	.ll_cur_idx = dw_edma_v0_core_ll_cur_idx,
 	.ch_doorbell = dw_edma_v0_core_ch_doorbell,
 	.ch_enable = dw_edma_v0_core_ch_enable,
 	.ch_config = dw_edma_v0_core_ch_config,
diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index 862375c8e4ba..677416f422ff 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -285,6 +285,22 @@ static void dw_hdma_v0_core_ch_doorbell(struct dw_edma_chan *chan)
 	SET_CH_32(dw, chan->dir, chan->id, doorbell, HDMA_V0_DOORBELL_START);
 }
 
+static int dw_hdma_v0_core_ll_cur_idx(struct dw_edma_chan *chan)
+{
+	u64 paddr;
+	u32 val;
+
+	/* LL region never cross 4G memory boundary, so only check low 32bit */
+	val = GET_CH_32(chan->dw, chan->dir, chan->id, llp.lsb);
+	paddr = dw_edma_core_get_ll_paddr(chan);
+
+	/* DMA have not setup or DMA engine reset because PCIe link lost */
+	if (!val)
+		return -EINVAL;
+
+	return (val - (paddr & 0xFFFFFFFF)) / EDMA_LL_SZ;
+}
+
 /* HDMA debugfs callbacks */
 static void dw_hdma_v0_core_debugfs_on(struct dw_edma *dw)
 {
@@ -298,6 +314,7 @@ static const struct dw_edma_core_ops dw_hdma_v0_core = {
 	.handle_int = dw_hdma_v0_core_handle_int,
 	.ll_data = dw_hdma_v0_core_ll_data,
 	.ll_link = dw_hdma_v0_core_ll_link,
+	.ll_cur_idx = dw_hdma_v0_core_ll_cur_idx,
 	.ch_doorbell = dw_hdma_v0_core_ch_doorbell,
 	.ch_enable = dw_hdma_v0_core_ch_enable,
 	.ch_config = dw_hdma_v0_core_ch_config,
-- 
2.51.0

[PATCH 07/17] dmaengine: dw-edma: Move dw_hdma_set_callback_result() up

[Koichiro Den]

From: Frank Li <Frank.Li@nxp.com>

Move dw_hdma_set_callback_result() before dw_edma_device_tx_status() to
avoid forward declare.

No functional change.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Unchanged from Frank's original submission:
https://lore.kernel.org/dmaengine/20260109-edma_dymatic-v1-2-9a98c9c98536@nxp.com/
except one trivial typo fix in the commit message (s/declear/declare/).

 drivers/dma/dw-edma/dw-edma-core.c | 50 +++++++++++++++---------------
 1 file changed, 25 insertions(+), 25 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 489f7fe49840..5e41b1aab450 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -107,6 +107,31 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 	return 1;
 }
 
+static void dw_hdma_set_callback_result(struct virt_dma_desc *vd,
+					enum dmaengine_tx_result result)
+{
+	u32 residue = 0;
+	struct dw_edma_desc *desc;
+	struct dmaengine_result *res;
+
+	if (!vd->tx.callback_result)
+		return;
+
+	desc = vd2dw_edma_desc(vd);
+	if (desc) {
+		residue = desc->alloc_sz;
+
+		if (result == DMA_TRANS_NOERROR)
+			residue -= desc->burst[desc->start_burst - 1].xfer_sz;
+		else if (desc->done_burst)
+			residue -= desc->burst[desc->done_burst - 1].xfer_sz;
+	}
+
+	res = &vd->tx_result;
+	res->result = result;
+	res->residue = residue;
+}
+
 static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
 {
 	list_del(&vd->node);
@@ -527,31 +552,6 @@ dw_edma_device_prep_interleaved_dma(struct dma_chan *dchan,
 	return dw_edma_device_transfer(&xfer, dw_edma_device_get_config(dchan, NULL));
 }
 
-static void dw_hdma_set_callback_result(struct virt_dma_desc *vd,
-					enum dmaengine_tx_result result)
-{
-	u32 residue = 0;
-	struct dw_edma_desc *desc;
-	struct dmaengine_result *res;
-
-	if (!vd->tx.callback_result)
-		return;
-
-	desc = vd2dw_edma_desc(vd);
-	if (desc) {
-		residue = desc->alloc_sz;
-
-		if (result == DMA_TRANS_NOERROR)
-			residue -= desc->burst[desc->start_burst - 1].xfer_sz;
-		else if (desc->done_burst)
-			residue -= desc->burst[desc->done_burst - 1].xfer_sz;
-	}
-
-	res = &vd->tx_result;
-	res->result = result;
-	res->residue = residue;
-}
-
 static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 {
 	struct dw_edma_desc *desc;
-- 
2.51.0

[PATCH 08/17] dmaengine: dw-edma: Make DMA link list work as a circular buffer

[Koichiro Den]

From: Frank Li <Frank.Li@nxp.com>

The existing code rebuilds the entire link list from the beginning and
resets the DMA link header for each transfer, which is unnecessary.

The DMA link list can be treated as a circular buffer, where new DMA
requests are appended at ll_head with the appropriate CB flags and ring
the doorbell, without rebuilding the whole list.

Switch to this circular-buffer model to prepare for dynamically adding
new requests while the DMA engine is running.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
[den: fixed partial-append start_burst accounting; refreshed the fixed
 link element before each new lap; dropped the unused first argument;
 fixed checkpatch.pl issues; polished doorbell wording]
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Based on the original submission from Frank:
https://lore.kernel.org/dmaengine/20260109-edma_dymatic-v1-3-9a98c9c98536@nxp.com/

 drivers/dma/dw-edma/dw-edma-core.c | 71 +++++++++++++++++++++++-------
 drivers/dma/dw-edma/dw-edma-core.h | 25 ++++++++++-
 2 files changed, 79 insertions(+), 17 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 5e41b1aab450..cac03c59bfe4 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -52,7 +52,6 @@ dw_edma_alloc_desc(struct dw_edma_chan *chan, u32 nburst)
 
 	desc->chan = chan;
 	desc->nburst = nburst;
-	desc->cb = true;
 
 	return desc;
 }
@@ -62,27 +61,64 @@ static void vchan_free_desc(struct virt_dma_desc *vdesc)
 	kfree(vd2dw_edma_desc(vdesc));
 }
 
-static void dw_edma_core_start(struct dw_edma_desc *desc, bool first)
+static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
+{
+	chan->ll_head = 0;
+	chan->ll_end = 0;
+	chan->cb = true;
+
+	dw_edma_core_ll_link(chan, chan->ll_max - 1, chan->cb,
+			     chan->ll_region.paddr);
+
+	dw_edma_core_ch_enable(chan);
+}
+
+static u32 dw_edma_core_get_free_num(struct dw_edma_chan *chan)
+{
+	/*
+	 * Max entries is ll_max - 1 because last one used for link back to
+	 * start of ll_region.
+	 */
+	return (chan->ll_end + chan->ll_max - 2 - chan->ll_head) %
+		(chan->ll_max - 1);
+}
+
+static void dw_edma_core_start(struct dw_edma_desc *desc)
 {
 	struct dw_edma_chan *chan = desc->chan;
 	u32 i = 0;
+	u32 free;
+
+	for (i = desc->start_burst; i < desc->nburst; i++) {
+		free = dw_edma_core_get_free_num(chan);
 
-	for (i = 0; i < desc->nburst; i++) {
-		if (i == chan->ll_max - 1)
+		if (!free)
 			break;
 
-		dw_edma_core_ll_data(chan, &desc->burst[i + desc->start_burst],
-				     i, desc->cb,
-				     i == desc->nburst - 1 || i == chan->ll_max - 2);
-	}
+		/*
+		 * Refresh the link element before filling the last data slot so
+		 * the next lap has the updated CB value.
+		 */
+		if (chan->ll_head == chan->ll_max - 2)
+			dw_edma_core_ll_link(chan, chan->ll_max - 1, chan->cb,
+					     chan->ll_region.paddr);
 
-	desc->done_burst = desc->start_burst;
-	desc->start_burst += i;
+		/* Enable irq for last free entry or last burst */
+		dw_edma_core_ll_data(chan, &desc->burst[i],
+				     chan->ll_head, chan->cb,
+				     i == desc->nburst - 1 || free == 1);
+
+		chan->ll_head++;
 
-	dw_edma_core_ll_link(chan, i, desc->cb, chan->ll_region.paddr);
+		if (chan->ll_head == chan->ll_max - 1) {
+			chan->cb = !chan->cb;
+			chan->ll_head = 0;
+		}
+	}
 
-	if (first)
-		dw_edma_core_ch_enable(chan);
+	desc->done_burst = desc->start_burst;
+	desc->start_burst = i;
+	desc->ll_end = chan->ll_head;
 
 	dw_edma_core_ch_doorbell(chan);
 }
@@ -91,6 +127,10 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 {
 	struct dw_edma_desc *desc;
 	struct virt_dma_desc *vd;
+	int index = dw_edma_core_ll_cur_idx(chan);
+
+	if (index < 0)
+		dw_edma_core_reset_ll(chan);
 
 	vd = vchan_next_desc(&chan->vc);
 	if (!vd)
@@ -100,9 +140,7 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 	if (!desc)
 		return 0;
 
-	dw_edma_core_start(desc, !desc->start_burst);
-
-	desc->cb = !desc->cb;
+	dw_edma_core_start(desc);
 
 	return 1;
 }
@@ -569,6 +607,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 							    DMA_TRANS_NOERROR);
 				list_del(&vd->node);
 				vchan_cookie_complete(vd);
+				chan->ll_end = desc->ll_end;
 			}
 
 			/* Continue transferring if there are remaining chunks or issued requests.
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index d68c4592c617..46af4ea3ae5f 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -60,9 +60,10 @@ struct dw_edma_desc {
 	u32				alloc_sz;
 	u32				xfer_sz;
 
+	u32				ll_end;
+
 	u32				done_burst;
 	u32				start_burst;
-	u8				cb;
 	u32				nburst;
 	struct dw_edma_burst            burst[] __counted_by(nburst);
 };
@@ -73,9 +74,31 @@ struct dw_edma_chan {
 	int				id;
 	enum dw_edma_dir		dir;
 
+	/*
+	 * New LL entries are appended at ll_head. Entries between ll_end
+	 * and ll_head, modulo the LL ring, are owned by DMA; the rest are
+	 * owned by software.
+	 *
+	 *   software-owned      DMA-owned       software-owned
+	 * +---------------+-------------------+---------------+
+	 * ^               ^                   ^
+	 * 0             ll_end              ll_head
+	 *
+	 * The link entry points back to the region start. ll_head == ll_end
+	 * means all entries are software-owned and previous DMA work is
+	 * done.
+	 *
+	 * Software always keeps at least one free entry, so the ring is
+	 * never completely DMA-owned.
+	 */
+	u32				ll_head;
+	u32				ll_end;
+
 	u32				ll_max;
 	struct dw_edma_region		ll_region;	/* Linked list */
 
+	bool				cb;
+
 	struct msi_msg			msi;
 
 	enum dw_edma_request		request;
-- 
2.51.0

[PATCH 09/17] dmaengine: dw-edma: Add LL interrupt placement policy

[Koichiro Den]

Move the linked-list (LL) interrupt placement decision behind a core
callback so eDMA and HDMA can use different policies.

Keep eDMA interrupts at descriptor ends and at the last free slot,
matching its existing DONE interrupt semantics.

For HDMA, use watermark placement as producer-consumer progress points.
Keep descriptor-end and the last data entry before the link element
unconditional: descriptor-end gives completion accounting a progress
point even for a single descriptor, while the last data entry provides
an end-of-lap producer-consumer checkpoint.

Add fixed-interval watermarks only when the current descriptor cannot
fit in the LL ring, or when another issued descriptor is waiting behind
it.  This keeps low-depth traffic from taking extra watermark interrupts
without making large rings mostly STOP-driven. The interval is a driver
coalescing policy, not a databook-mandated value.

This patch only decides where LL interrupt bits should be set. A later
HDMA patch wires those bits to the hardware watermark interrupt path.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c    |  9 ++++++--
 drivers/dma/dw-edma/dw-edma-core.h    |  1 +
 drivers/dma/dw-edma/dw-edma-v0-core.c | 10 +++++++++
 drivers/dma/dw-edma/dw-hdma-v0-core.c | 31 +++++++++++++++++++++++++++
 4 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index cac03c59bfe4..2165f2fa5398 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -83,6 +83,12 @@ static u32 dw_edma_core_get_free_num(struct dw_edma_chan *chan)
 		(chan->ll_max - 1);
 }
 
+static bool dw_edma_core_enable_ll_irq(struct dw_edma_desc *desc, u32 i,
+				       u32 free)
+{
+	return desc->chan->dw->core->ll_irq(desc, i, free);
+}
+
 static void dw_edma_core_start(struct dw_edma_desc *desc)
 {
 	struct dw_edma_chan *chan = desc->chan;
@@ -103,10 +109,9 @@ static void dw_edma_core_start(struct dw_edma_desc *desc)
 			dw_edma_core_ll_link(chan, chan->ll_max - 1, chan->cb,
 					     chan->ll_region.paddr);
 
-		/* Enable irq for last free entry or last burst */
 		dw_edma_core_ll_data(chan, &desc->burst[i],
 				     chan->ll_head, chan->cb,
-				     i == desc->nburst - 1 || free == 1);
+				     dw_edma_core_enable_ll_irq(desc, i, free));
 
 		chan->ll_head++;
 
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 46af4ea3ae5f..ea9f4292c40e 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -147,6 +147,7 @@ struct dw_edma_core_ops {
 			u32 idx, bool cb, bool irq);
 	void (*ll_link)(struct dw_edma_chan *chan, u32 idx, bool cb, u64 addr);
 	int (*ll_cur_idx)(struct dw_edma_chan *chan);
+	bool (*ll_irq)(struct dw_edma_desc *desc, u32 i, u32 free);
 	void (*ch_doorbell)(struct dw_edma_chan *chan);
 	void (*ch_enable)(struct dw_edma_chan *chan);
 	void (*ch_config)(struct dw_edma_chan *chan);
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index edc71a4dbc79..47faedd14dc2 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -520,6 +520,15 @@ static int dw_edma_v0_core_ll_cur_idx(struct dw_edma_chan *chan)
 	return (val - (paddr & 0xFFFFFFFF)) / EDMA_LL_SZ;
 }
 
+static bool dw_edma_v0_core_ll_irq(struct dw_edma_desc *desc, u32 i, u32 free)
+{
+	/*
+	 * eDMA reports LL interrupts through DONE. Keep them at
+	 * descriptor ends, plus the last free slot to refill the ring.
+	 */
+	return i == desc->nburst - 1 || free == 1;
+}
+
 /* eDMA debugfs callbacks */
 static void dw_edma_v0_core_debugfs_on(struct dw_edma *dw)
 {
@@ -534,6 +543,7 @@ static const struct dw_edma_core_ops dw_edma_v0_core = {
 	.ll_data = dw_edma_v0_core_ll_data,
 	.ll_link = dw_edma_v0_core_ll_link,
 	.ll_cur_idx = dw_edma_v0_core_ll_cur_idx,
+	.ll_irq = dw_edma_v0_core_ll_irq,
 	.ch_doorbell = dw_edma_v0_core_ch_doorbell,
 	.ch_enable = dw_edma_v0_core_ch_enable,
 	.ch_config = dw_edma_v0_core_ch_config,
diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index 677416f422ff..b9e193774714 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -13,6 +13,9 @@
 #include "dw-hdma-v0-regs.h"
 #include "dw-hdma-v0-debugfs.h"
 
+/* Empirically chosen; can become debugfs-tunable if needed. */
+#define HDMA_V0_WATERMARK_INTERVAL			4
+
 enum dw_hdma_control {
 	DW_HDMA_V0_CB					= BIT(0),
 	DW_HDMA_V0_TCB					= BIT(1),
@@ -301,6 +304,33 @@ static int dw_hdma_v0_core_ll_cur_idx(struct dw_edma_chan *chan)
 	return (val - (paddr & 0xFFFFFFFF)) / EDMA_LL_SZ;
 }
 
+static bool dw_hdma_v0_core_ll_irq(struct dw_edma_desc *desc, u32 i, u32 free)
+{
+	struct dw_edma_chan *chan = desc->chan;
+	bool needs_progress;
+
+	/*
+	 * Keep descriptor-end and the last data entry before the link element
+	 * unconditional: descriptor-end gives completion accounting a progress
+	 * point even for a single descriptor, while the last data entry provides
+	 * an end-of-lap producer-consumer checkpoint.
+	 */
+	if (i == desc->nburst - 1 || chan->ll_head == chan->ll_max - 2)
+		return true;
+
+	/*
+	 * Additional fixed-interval watermarks keep large LL rings from becoming
+	 * mostly STOP-driven. They are useful only when there is more work to
+	 * feed or the current descriptor cannot fit in the LL ring without
+	 * progress.
+	 */
+	needs_progress = desc->nburst > chan->ll_max - 2 ||
+			 !list_is_last(&desc->vd.node, &chan->vc.desc_issued);
+
+	return needs_progress && chan->ll_head &&
+	       chan->ll_head % HDMA_V0_WATERMARK_INTERVAL == 0;
+}
+
 /* HDMA debugfs callbacks */
 static void dw_hdma_v0_core_debugfs_on(struct dw_edma *dw)
 {
@@ -315,6 +345,7 @@ static const struct dw_edma_core_ops dw_hdma_v0_core = {
 	.ll_data = dw_hdma_v0_core_ll_data,
 	.ll_link = dw_hdma_v0_core_ll_link,
 	.ll_cur_idx = dw_hdma_v0_core_ll_cur_idx,
+	.ll_irq = dw_hdma_v0_core_ll_irq,
 	.ch_doorbell = dw_hdma_v0_core_ch_doorbell,
 	.ch_enable = dw_hdma_v0_core_ch_enable,
 	.ch_config = dw_hdma_v0_core_ch_config,
-- 
2.51.0

[PATCH 10/17] dmaengine: dw-edma: Reclaim issued descriptors from LL progress

[Koichiro Den]

Dynamic append can leave LL entries for more than one issued descriptor
in the ring at the same time. A DONE or progress interrupt therefore
cannot complete work by looking only at the first issued descriptor.

Track the hardware consumption point separately from descriptor
completion. ll_done follows the LL pointer reported by hardware; ll_end
remains the boundary of the last completed descriptor. Measure free
space from ll_done, so consumed LL entries can be reused even before the
descriptor that owns them has completed. This keeps descriptors larger
than the LL ring moving.

Keep start_burst as the appended boundary and done_burst as the consumed
boundary. Complete issued descriptors whose fully appended LL range is
covered by the latest progress segment. The ring keeps one data entry
free, so a reported physical LL index is unique within the current
ll_done..ll_head producer window.

For eDMA, LL element LIE/RIE interrupts are reported through the DONE
interrupt status, and the documented producer-consumer flow reads
DMA_LLP_* on such interrupts to recycle elements up to the reported
location. HDMA STOP handling also uses this common accounting. The
following patch wires HDMA watermark interrupts as running progress
events.

[den: dw_edma_ll_clean_pending() naming and core idea are borrowed from
 20260109-edma_dymatic-v1-3-9a98c9c98536@nxp.com]
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c | 111 +++++++++++++++++++++++++++--
 drivers/dma/dw-edma/dw-edma-core.h |  18 ++---
 2 files changed, 116 insertions(+), 13 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 2165f2fa5398..839bafc762a1 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -65,6 +65,7 @@ static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
 {
 	chan->ll_head = 0;
 	chan->ll_end = 0;
+	chan->ll_done = 0;
 	chan->cb = true;
 
 	dw_edma_core_ll_link(chan, chan->ll_max - 1, chan->cb,
@@ -73,13 +74,34 @@ static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
 	dw_edma_core_ch_enable(chan);
 }
 
+static u32 dw_edma_core_get_ll_data_cnt(struct dw_edma_chan *chan)
+{
+	return chan->ll_max - 1;
+}
+
+static u32 dw_edma_core_get_ll_dist(struct dw_edma_chan *chan, u32 from, u32 to)
+{
+	u32 cnt = dw_edma_core_get_ll_data_cnt(chan);
+
+	return (to + cnt - from) % cnt;
+}
+
+static u32 dw_edma_core_get_used_num(struct dw_edma_chan *chan)
+{
+	return dw_edma_core_get_ll_dist(chan, chan->ll_done, chan->ll_head);
+}
+
 static u32 dw_edma_core_get_free_num(struct dw_edma_chan *chan)
 {
 	/*
+	 * Measure occupancy from ll_done, not ll_end. Entries consumed by the
+	 * hardware can be reused even if the descriptor that owns them has not
+	 * completed yet; this lets descriptors larger than the ring move forward.
+	 *
 	 * Max entries is ll_max - 1 because last one used for link back to
 	 * start of ll_region.
 	 */
-	return (chan->ll_end + chan->ll_max - 2 - chan->ll_head) %
+	return (chan->ll_done + chan->ll_max - 2 - chan->ll_head) %
 		(chan->ll_max - 1);
 }
 
@@ -89,6 +111,29 @@ static bool dw_edma_core_enable_ll_irq(struct dw_edma_desc *desc, u32 i,
 	return desc->chan->dw->core->ll_irq(desc, i, free);
 }
 
+static bool dw_edma_ll_advance(struct dw_edma_chan *chan, int idx, u32 *old_done)
+{
+	u32 cnt = dw_edma_core_get_ll_data_cnt(chan);
+	u32 done;
+
+	if (idx < 0 || (u32)idx >= cnt)
+		return false;
+
+	done = dw_edma_core_get_ll_dist(chan, chan->ll_done, idx);
+	if (!done || done > dw_edma_core_get_used_num(chan))
+		return false;
+
+	*old_done = chan->ll_done;
+	chan->ll_done = idx;
+
+	return true;
+}
+
+static bool dw_edma_ll_pending(struct dw_edma_chan *chan)
+{
+	return dw_edma_core_get_used_num(chan);
+}
+
 static void dw_edma_core_start(struct dw_edma_desc *desc)
 {
 	struct dw_edma_chan *chan = desc->chan;
@@ -121,7 +166,6 @@ static void dw_edma_core_start(struct dw_edma_desc *desc)
 		}
 	}
 
-	desc->done_burst = desc->start_burst;
 	desc->start_burst = i;
 	desc->ll_end = chan->ll_head;
 
@@ -164,9 +208,7 @@ static void dw_hdma_set_callback_result(struct virt_dma_desc *vd,
 	if (desc) {
 		residue = desc->alloc_sz;
 
-		if (result == DMA_TRANS_NOERROR)
-			residue -= desc->burst[desc->start_burst - 1].xfer_sz;
-		else if (desc->done_burst)
+		if (desc->done_burst)
 			residue -= desc->burst[desc->done_burst - 1].xfer_sz;
 	}
 
@@ -204,6 +246,59 @@ static void dw_edma_terminate_all_descs(struct dw_edma_chan *chan)
 	dw_edma_terminate_vdesc_list(&chan->vc.desc_submitted);
 }
 
+/* Must be called with vc.lock held. */
+static void dw_edma_ll_clean_pending(struct dw_edma_chan *chan, u32 old_done)
+{
+	struct virt_dma_desc *vd, *_vd;
+	u32 done = dw_edma_core_get_ll_dist(chan, old_done, chan->ll_done);
+
+	list_for_each_entry_safe(vd, _vd, &chan->vc.desc_issued, node) {
+		struct dw_edma_desc *desc = vd2dw_edma_desc(vd);
+		u32 consumed, started;
+
+		if (!done)
+			break;
+
+		/*
+		 * start_burst is the append boundary. done_burst is the
+		 * hardware-consumed boundary reported through LL progress.
+		 */
+		started = desc->start_burst - desc->done_burst;
+		if (!started)
+			break;
+
+		consumed = min(done, started);
+		desc->done_burst += consumed;
+		done -= consumed;
+
+		/*
+		 * Descriptors are appended in list order, so later descriptors
+		 * cannot be complete if this one has not been fully consumed.
+		 */
+		if (desc->done_burst != desc->nburst)
+			break;
+
+		/* Hardware has consumed this descriptor's LL entries. */
+		dw_hdma_set_callback_result(vd, DMA_TRANS_NOERROR);
+		list_del(&vd->node);
+		vchan_cookie_complete(vd);
+		chan->ll_end = desc->ll_end;
+	}
+}
+
+/* Must be called with vc.lock held. */
+static bool dw_edma_ll_recycle(struct dw_edma_chan *chan, int idx)
+{
+	u32 old_done;
+
+	if (!dw_edma_ll_advance(chan, idx, &old_done))
+		return false;
+
+	dw_edma_ll_clean_pending(chan, old_done);
+
+	return true;
+}
+
 static void dw_edma_device_caps(struct dma_chan *dchan,
 				struct dma_slave_caps *caps)
 {
@@ -600,8 +695,11 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 	struct dw_edma_desc *desc;
 	struct virt_dma_desc *vd;
 	unsigned long flags;
+	int idx;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
+	idx = dw_edma_core_ll_cur_idx(chan);
+	dw_edma_ll_recycle(chan, idx);
 	vd = vchan_next_desc(&chan->vc);
 	if (vd) {
 		switch (chan->request) {
@@ -634,6 +732,9 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 		default:
 			break;
 		}
+	} else if (chan->request == EDMA_REQ_NONE) {
+		chan->status = dw_edma_ll_pending(chan) ?
+			       EDMA_ST_BUSY : EDMA_ST_IDLE;
 	}
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index ea9f4292c40e..dbc4af0eab59 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -75,24 +75,26 @@ struct dw_edma_chan {
 	enum dw_edma_dir		dir;
 
 	/*
-	 * New LL entries are appended at ll_head. Entries between ll_end
-	 * and ll_head, modulo the LL ring, are owned by DMA; the rest are
-	 * owned by software.
+	 * New LL entries are appended at ll_head. Entries between ll_done and
+	 * ll_head, modulo the LL ring, are owned by DMA; the rest have already
+	 * been consumed and may be overwritten by software. ll_end trails behind
+	 * at the boundary of the last completed descriptor.
 	 *
 	 *   software-owned      DMA-owned       software-owned
 	 * +---------------+-------------------+---------------+
 	 * ^               ^                   ^
-	 * 0             ll_end              ll_head
+	 * 0            ll_done             ll_head
 	 *
-	 * The link entry points back to the region start. ll_head == ll_end
-	 * means all entries are software-owned and previous DMA work is
-	 * done.
+	 * The link entry points back to the region start. No DMA-owned entries
+	 * remain once ll_done catches up with ll_head.
 	 *
 	 * Software always keeps at least one free entry, so the ring is
-	 * never completely DMA-owned.
+	 * never completely DMA-owned. That keeps a hardware-reported physical
+	 * LL index unique within the current ll_done..ll_head producer window.
 	 */
 	u32				ll_head;
 	u32				ll_end;
+	u32				ll_done;
 
 	u32				ll_max;
 	struct dw_edma_region		ll_region;	/* Linked list */
-- 
2.51.0

[PATCH 11/17] dmaengine: dw-edma: Use HDMA watermarks as progress events

[Koichiro Den]

HDMA updates its running LL pointer at watermark points, so route HDMA
watermark interrupts to a progress handler instead of treating progress
as a STOP-only event.

Wire HDMA watermark interrupt setup, clearing, MSI address programming,
and LL LWIE/RWIE bits into the core interrupt path. STOP remains the
HDMA done event, while WATERMARK only reports running progress.

The previous patch decides where LL interrupt bits are placed. This
patch makes HDMA honor those bits as hardware watermark events and
dispatch them to the common progress handler.

The progress handler reuses the common LL recycling helper to reclaim
completed descriptors and refill the ring while the channel keeps
running.  Do this only while no channel request is pending; STOP and
PAUSE requests are handled by their request paths, not as normal running
progress.

The legacy eDMA dispatcher keeps routing its interrupt status only to
DONE and ABORT handlers; the new progress handler is used only by HDMA
watermark interrupts.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c    | 27 ++++++++++++
 drivers/dma/dw-edma/dw-edma-core.h    |  9 ++--
 drivers/dma/dw-edma/dw-edma-v0-core.c |  4 +-
 drivers/dma/dw-edma/dw-hdma-v0-core.c | 59 +++++++++++++++++++++++----
 drivers/dma/dw-edma/dw-hdma-v0-regs.h |  1 +
 5 files changed, 87 insertions(+), 13 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 839bafc762a1..a289d8f8cc17 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -299,6 +299,15 @@ static bool dw_edma_ll_recycle(struct dw_edma_chan *chan, int idx)
 	return true;
 }
 
+/* Must be called with vc.lock held. */
+static bool dw_edma_ll_recycle_and_refill(struct dw_edma_chan *chan, int idx)
+{
+	if (!dw_edma_ll_recycle(chan, idx))
+		return false;
+
+	return dw_edma_start_transfer(chan);
+}
+
 static void dw_edma_device_caps(struct dma_chan *dchan,
 				struct dma_slave_caps *caps)
 {
@@ -739,6 +748,22 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
 
+static void dw_edma_progress_interrupt(struct dw_edma_chan *chan)
+{
+	unsigned long flags;
+	int idx;
+
+	spin_lock_irqsave(&chan->vc.lock, flags);
+	idx = dw_edma_core_ll_cur_idx(chan);
+	if (chan->request == EDMA_REQ_NONE && chan->status != EDMA_ST_PAUSE) {
+		dw_edma_ll_recycle_and_refill(chan, idx);
+		chan->status = dw_edma_ll_pending(chan) ?
+			       EDMA_ST_BUSY : EDMA_ST_IDLE;
+	}
+
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
+}
+
 static void dw_edma_abort_interrupt(struct dw_edma_chan *chan)
 {
 	struct virt_dma_desc *vd;
@@ -762,6 +787,7 @@ static inline irqreturn_t dw_edma_interrupt_write(int irq, void *data)
 
 	return dw_edma_core_handle_int(dw_irq, EDMA_DIR_WRITE,
 				       dw_edma_done_interrupt,
+				       dw_edma_progress_interrupt,
 				       dw_edma_abort_interrupt);
 }
 
@@ -771,6 +797,7 @@ static inline irqreturn_t dw_edma_interrupt_read(int irq, void *data)
 
 	return dw_edma_core_handle_int(dw_irq, EDMA_DIR_READ,
 				       dw_edma_done_interrupt,
+				       dw_edma_progress_interrupt,
 				       dw_edma_abort_interrupt);
 }
 
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index dbc4af0eab59..9bd0a5f2f08b 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -144,7 +144,9 @@ struct dw_edma_core_ops {
 	u16 (*ch_count)(struct dw_edma *dw, enum dw_edma_dir dir);
 	enum dma_status (*ch_status)(struct dw_edma_chan *chan);
 	irqreturn_t (*handle_int)(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
-				  dw_edma_handler_t done, dw_edma_handler_t abort);
+				  dw_edma_handler_t done,
+				  dw_edma_handler_t progress,
+				  dw_edma_handler_t abort);
 	void (*ll_data)(struct dw_edma_chan *chan, struct dw_edma_burst *burst,
 			u32 idx, bool cb, bool irq);
 	void (*ll_link)(struct dw_edma_chan *chan, u32 idx, bool cb, u64 addr);
@@ -228,9 +230,10 @@ enum dma_status dw_edma_core_ch_status(struct dw_edma_chan *chan)
 
 static inline irqreturn_t
 dw_edma_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
-			dw_edma_handler_t done, dw_edma_handler_t abort)
+			dw_edma_handler_t done, dw_edma_handler_t progress,
+			dw_edma_handler_t abort)
 {
-	return dw_irq->dw->core->handle_int(dw_irq, dir, done, abort);
+	return dw_irq->dw->core->handle_int(dw_irq, dir, done, progress, abort);
 }
 
 static inline
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index 47faedd14dc2..dfe0483896d3 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -232,7 +232,9 @@ static u32 dw_edma_v0_core_status_abort_int(struct dw_edma *dw, enum dw_edma_dir
 
 static irqreturn_t
 dw_edma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
-			   dw_edma_handler_t done, dw_edma_handler_t abort)
+			   dw_edma_handler_t done,
+			   dw_edma_handler_t progress,
+			   dw_edma_handler_t abort)
 {
 	struct dw_edma *dw = dw_irq->dw;
 	unsigned long total, pos, val;
diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index b9e193774714..9f5b11350f23 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -59,9 +59,13 @@ static void dw_hdma_v0_core_off(struct dw_edma *dw)
 
 	for (id = 0; id < HDMA_V0_MAX_NR_CH; id++) {
 		SET_BOTH_CH_32(dw, id, int_setup,
-			       HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+			       HDMA_V0_STOP_INT_MASK |
+			       HDMA_V0_WATERMARK_INT_MASK |
+			       HDMA_V0_ABORT_INT_MASK);
 		SET_BOTH_CH_32(dw, id, int_clear,
-			       HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+			       HDMA_V0_STOP_INT_MASK |
+			       HDMA_V0_WATERMARK_INT_MASK |
+			       HDMA_V0_ABORT_INT_MASK);
 		SET_BOTH_CH_32(dw, id, ch_en, 0);
 	}
 }
@@ -99,6 +103,14 @@ static void dw_hdma_v0_core_clear_done_int(struct dw_edma_chan *chan)
 	SET_CH_32(dw, chan->dir, chan->id, int_clear, HDMA_V0_STOP_INT_MASK);
 }
 
+static void dw_hdma_v0_core_clear_watermark_int(struct dw_edma_chan *chan)
+{
+	struct dw_edma *dw = chan->dw;
+
+	SET_CH_32(dw, chan->dir, chan->id, int_clear,
+		  HDMA_V0_WATERMARK_INT_MASK);
+}
+
 static void dw_hdma_v0_core_clear_abort_int(struct dw_edma_chan *chan)
 {
 	struct dw_edma *dw = chan->dw;
@@ -115,7 +127,9 @@ static u32 dw_hdma_v0_core_status_int(struct dw_edma_chan *chan)
 
 static irqreturn_t
 dw_hdma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
-			   dw_edma_handler_t done, dw_edma_handler_t abort)
+			   dw_edma_handler_t done,
+			   dw_edma_handler_t progress,
+			   dw_edma_handler_t abort)
 {
 	struct dw_edma *dw = dw_irq->dw;
 	unsigned long total, pos, val;
@@ -134,16 +148,29 @@ dw_hdma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 	}
 
 	for_each_set_bit(pos, &mask, total) {
+		bool has_stop, has_watermark;
+
 		chan = &dw->chan[pos + off];
 
 		val = dw_hdma_v0_core_status_int(chan);
-		if (FIELD_GET(HDMA_V0_STOP_INT_MASK, val)) {
-			dw_hdma_v0_core_clear_done_int(chan);
-			done(chan);
+		has_stop = FIELD_GET(HDMA_V0_STOP_INT_MASK, val);
+		has_watermark = FIELD_GET(HDMA_V0_WATERMARK_INT_MASK, val);
 
+		if (has_watermark) {
+			dw_hdma_v0_core_clear_watermark_int(chan);
 			ret = IRQ_HANDLED;
 		}
 
+		if (has_stop) {
+			dw_hdma_v0_core_clear_done_int(chan);
+			ret = IRQ_HANDLED;
+		}
+
+		if (has_stop)
+			done(chan);
+		else if (has_watermark)
+			progress(chan);
+
 		if (FIELD_GET(HDMA_V0_ABORT_INT_MASK, val)) {
 			dw_hdma_v0_core_clear_abort_int(chan);
 			abort(chan);
@@ -204,10 +231,12 @@ static void dw_hdma_v0_core_ch_enable(struct dw_edma_chan *chan)
 
 	/* Enable engine */
 	SET_CH_32(dw, chan->dir, chan->id, ch_en, BIT(0));
-	/* Interrupt unmask - stop, abort */
+	/* Interrupt unmask - stop, watermark, abort */
 	tmp = GET_CH_32(dw, chan->dir, chan->id, int_setup);
-	tmp &= ~(HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
-	/* Interrupt enable - stop, abort */
+	tmp &= ~(HDMA_V0_STOP_INT_MASK | HDMA_V0_WATERMARK_INT_MASK |
+		 HDMA_V0_ABORT_INT_MASK);
+	/* Interrupt enable - stop, abort. */
+	/* Watermark is enabled per LL element. */
 	tmp |= HDMA_V0_LOCAL_STOP_INT_EN | HDMA_V0_LOCAL_ABORT_INT_EN;
 	if (!(dw->chip->flags & DW_EDMA_CHIP_LOCAL))
 		tmp |= HDMA_V0_REMOTE_STOP_INT_EN | HDMA_V0_REMOTE_ABORT_INT_EN;
@@ -247,6 +276,11 @@ static void dw_hdma_v0_core_ch_config(struct dw_edma_chan *chan)
 	/* MSI done addr - low, high */
 	SET_CH_32(dw, chan->dir, chan->id, msi_stop.lsb, chan->msi.address_lo);
 	SET_CH_32(dw, chan->dir, chan->id, msi_stop.msb, chan->msi.address_hi);
+	/* MSI watermark addr - low, high */
+	SET_CH_32(dw, chan->dir, chan->id, msi_watermark.lsb,
+		  chan->msi.address_lo);
+	SET_CH_32(dw, chan->dir, chan->id, msi_watermark.msb,
+		  chan->msi.address_hi);
 	/* MSI abort addr - low, high */
 	SET_CH_32(dw, chan->dir, chan->id, msi_abort.lsb, chan->msi.address_lo);
 	SET_CH_32(dw, chan->dir, chan->id, msi_abort.msb, chan->msi.address_hi);
@@ -263,6 +297,13 @@ dw_hdma_v0_core_ll_data(struct dw_edma_chan *chan, struct dw_edma_burst *burst,
 	if (cb)
 		control |= DW_HDMA_V0_CB;
 
+	if (irq) {
+		control |= DW_HDMA_V0_LWIE;
+
+		if (!(chan->dw->chip->flags & DW_EDMA_CHIP_LOCAL))
+			control |= DW_HDMA_V0_RWIE;
+	}
+
 	dw_hdma_v0_write_ll_data(chan, idx, control, burst->sz, burst->sar,
 				 burst->dar);
 }
diff --git a/drivers/dma/dw-edma/dw-hdma-v0-regs.h b/drivers/dma/dw-edma/dw-hdma-v0-regs.h
index eab5fd7177e5..bf13e451b1a9 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-regs.h
+++ b/drivers/dma/dw-edma/dw-hdma-v0-regs.h
@@ -17,6 +17,7 @@
 #define HDMA_V0_LOCAL_STOP_INT_EN		BIT(4)
 #define HDMA_V0_REMOTE_STOP_INT_EN		BIT(3)
 #define HDMA_V0_ABORT_INT_MASK			BIT(2)
+#define HDMA_V0_WATERMARK_INT_MASK		BIT(1)
 #define HDMA_V0_STOP_INT_MASK			BIT(0)
 #define HDMA_V0_LINKLIST_EN			BIT(0)
 #define HDMA_V0_CONSUMER_CYCLE_STAT		BIT(1)
-- 
2.51.0

[PATCH 12/17] dmaengine: dw-edma: Clear LL data entries on reset

[Koichiro Den]

The LL memory has always been reused, but the circular LL ring makes old
data entries reachable again after the software state is reset.
Resetting ll_head, ll_end and cb is not enough if a data entry from an
earlier lap still carries a control word with a CB value matching the
next lap.

Add a core operation to clear a data entry control word, and scrub all
data entries when the LL ring is reset.

Only the control word needs to be cleared. The reset path starts the
next ring cycle with CB=1, so a zeroed control word leaves stale data
entries with CB cleared and prevents them from being consumed before
software rewrites the slot with the current CB.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c    |  5 +++++
 drivers/dma/dw-edma/dw-edma-core.h    |  6 ++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c | 16 ++++++++++++++++
 drivers/dma/dw-edma/dw-hdma-v0-core.c | 16 ++++++++++++++++
 4 files changed, 43 insertions(+)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index a289d8f8cc17..e76d8e0c6fa8 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -63,9 +63,14 @@ static void vchan_free_desc(struct virt_dma_desc *vdesc)
 
 static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
 {
+	u32 i;
+
 	chan->ll_head = 0;
 	chan->ll_end = 0;
 	chan->ll_done = 0;
+	/* Drop stale CB bits before reusing the circular LL ring. */
+	for (i = 0; i < chan->ll_max - 1; i++)
+		dw_edma_core_ll_clear(chan, i);
 	chan->cb = true;
 
 	dw_edma_core_ll_link(chan, chan->ll_max - 1, chan->cb,
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 9bd0a5f2f08b..1252d264c1ca 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -150,6 +150,7 @@ struct dw_edma_core_ops {
 	void (*ll_data)(struct dw_edma_chan *chan, struct dw_edma_burst *burst,
 			u32 idx, bool cb, bool irq);
 	void (*ll_link)(struct dw_edma_chan *chan, u32 idx, bool cb, u64 addr);
+	void (*ll_clear)(struct dw_edma_chan *chan, u32 idx);
 	int (*ll_cur_idx)(struct dw_edma_chan *chan);
 	bool (*ll_irq)(struct dw_edma_desc *desc, u32 i, u32 free);
 	void (*ch_doorbell)(struct dw_edma_chan *chan);
@@ -255,6 +256,11 @@ dw_edma_core_ll_link(struct dw_edma_chan *chan, u32 idx, bool cb, u64 addr)
 	chan->dw->core->ll_link(chan, idx, cb, addr);
 }
 
+static inline void dw_edma_core_ll_clear(struct dw_edma_chan *chan, u32 idx)
+{
+	chan->dw->core->ll_clear(chan, idx);
+}
+
 static inline void dw_edma_core_ch_doorbell(struct dw_edma_chan *chan)
 {
 	chan->dw->core->ch_doorbell(chan);
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index dfe0483896d3..265eefbf2ead 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -495,6 +495,21 @@ dw_edma_v0_core_ll_link(struct dw_edma_chan *chan, u32 idx, bool cb, u64 addr)
 	dw_edma_v0_write_ll_link(chan, idx, control, chan->ll_region.paddr);
 }
 
+static void dw_edma_v0_core_ll_clear(struct dw_edma_chan *chan, u32 idx)
+{
+	ptrdiff_t ofs = idx * sizeof(struct dw_edma_v0_lli);
+
+	if (chan->dw->chip->flags & DW_EDMA_CHIP_LOCAL) {
+		struct dw_edma_v0_lli *lli = chan->ll_region.vaddr.mem + ofs;
+
+		lli->control = 0;
+	} else {
+		struct dw_edma_v0_lli __iomem *lli = chan->ll_region.vaddr.io + ofs;
+
+		writel(0, &lli->control);
+	}
+}
+
 static void dw_edma_v0_core_ch_doorbell(struct dw_edma_chan *chan)
 {
 	struct dw_edma *dw = chan->dw;
@@ -544,6 +559,7 @@ static const struct dw_edma_core_ops dw_edma_v0_core = {
 	.handle_int = dw_edma_v0_core_handle_int,
 	.ll_data = dw_edma_v0_core_ll_data,
 	.ll_link = dw_edma_v0_core_ll_link,
+	.ll_clear = dw_edma_v0_core_ll_clear,
 	.ll_cur_idx = dw_edma_v0_core_ll_cur_idx,
 	.ll_irq = dw_edma_v0_core_ll_irq,
 	.ch_doorbell = dw_edma_v0_core_ch_doorbell,
diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index 9f5b11350f23..ad5e8201eb63 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -319,6 +319,21 @@ dw_hdma_v0_core_ll_link(struct dw_edma_chan *chan, u32 idx, bool cb, u64 addr)
 	dw_hdma_v0_write_ll_link(chan, idx, control, chan->ll_region.paddr);
 }
 
+static void dw_hdma_v0_core_ll_clear(struct dw_edma_chan *chan, u32 idx)
+{
+	ptrdiff_t ofs = idx * sizeof(struct dw_hdma_v0_lli);
+
+	if (chan->dw->chip->flags & DW_EDMA_CHIP_LOCAL) {
+		struct dw_hdma_v0_lli *lli = chan->ll_region.vaddr.mem + ofs;
+
+		lli->control = 0;
+	} else {
+		struct dw_hdma_v0_lli __iomem *lli = chan->ll_region.vaddr.io + ofs;
+
+		writel(0, &lli->control);
+	}
+}
+
 static void dw_hdma_v0_core_ch_doorbell(struct dw_edma_chan *chan)
 {
 	struct dw_edma *dw = chan->dw;
@@ -385,6 +400,7 @@ static const struct dw_edma_core_ops dw_hdma_v0_core = {
 	.handle_int = dw_hdma_v0_core_handle_int,
 	.ll_data = dw_hdma_v0_core_ll_data,
 	.ll_link = dw_hdma_v0_core_ll_link,
+	.ll_clear = dw_hdma_v0_core_ll_clear,
 	.ll_cur_idx = dw_hdma_v0_core_ll_cur_idx,
 	.ll_irq = dw_hdma_v0_core_ll_irq,
 	.ch_doorbell = dw_hdma_v0_core_ch_doorbell,
-- 
2.51.0

[PATCH 13/17] dmaengine: dw-edma: Dispatch DONE interrupts by channel request

[Koichiro Den]

Handle the channel request first in the DONE interrupt path, then look
at the issued descriptor list only in the cases that need it.

This keeps the existing behavior, including the current STOP and PAUSE
handling when no issued descriptor is present.

No functional change intended.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Note: this patch only reshapes the code to make the next patch easier to
review.

 drivers/dma/dw-edma/dw-edma-core.c | 34 ++++++++++++++++++------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index e76d8e0c6fa8..ae38ff0a8b83 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -715,9 +715,10 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 	idx = dw_edma_core_ll_cur_idx(chan);
 	dw_edma_ll_recycle(chan, idx);
 	vd = vchan_next_desc(&chan->vc);
-	if (vd) {
-		switch (chan->request) {
-		case EDMA_REQ_NONE:
+
+	switch (chan->request) {
+	case EDMA_REQ_NONE:
+		if (vd) {
 			desc = vd2dw_edma_desc(vd);
 			if (desc->start_burst >= desc->nburst) {
 				dw_hdma_set_callback_result(vd,
@@ -730,26 +731,31 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 			/* Continue transferring if there are remaining chunks or issued requests.
 			 */
 			chan->status = dw_edma_start_transfer(chan) ? EDMA_ST_BUSY : EDMA_ST_IDLE;
-			break;
+		} else {
+			chan->status = dw_edma_ll_pending(chan) ?
+				       EDMA_ST_BUSY : EDMA_ST_IDLE;
+		}
+		break;
 
-		case EDMA_REQ_STOP:
+	case EDMA_REQ_STOP:
+		if (vd) {
 			dw_edma_terminate_all_descs(chan);
 			chan->request = EDMA_REQ_NONE;
 			chan->status = EDMA_ST_IDLE;
-			break;
+		}
+		break;
 
-		case EDMA_REQ_PAUSE:
+	case EDMA_REQ_PAUSE:
+		if (vd) {
 			chan->request = EDMA_REQ_NONE;
 			chan->status = EDMA_ST_PAUSE;
-			break;
-
-		default:
-			break;
 		}
-	} else if (chan->request == EDMA_REQ_NONE) {
-		chan->status = dw_edma_ll_pending(chan) ?
-			       EDMA_ST_BUSY : EDMA_ST_IDLE;
+		break;
+
+	default:
+		break;
 	}
+
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
 
-- 
2.51.0

[PATCH 14/17] dmaengine: dw-edma: Reset LL state after terminate and abort

[Koichiro Den]

Termination or abort can leave appended entries in the circular LL ring.
Reset the LL state before the channel is reused, so the next transfer
starts from a clean ring instead of stale producer state or old LL
entries.

Since this now resets the hardware-visible LL ring, terminate_all() must
base the decision on the hardware channel state, not only on the
driver's software status. If the channel is still running, defer the
reset to STOP acknowledgment. If the hardware has already stopped, no
later interrupt will acknowledge a pending STOP or PAUSE request, so
clean up immediately.

Once STOP has been requested, do not recycle LL progress as successful
completion. Move pending descriptors to the terminated list without
callbacks and deconfigure the channel. For abort interrupts, complete
all issued descriptors with DMA_TRANS_ABORTED, reset the LL state, and
leave the channel configured for reuse.

Also handle STOP and PAUSE acknowledgments even if the issued list has
already become empty.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c | 60 +++++++++++++++---------------
 1 file changed, 29 insertions(+), 31 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index ae38ff0a8b83..4036adafedfa 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -397,34 +397,33 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 	unsigned long flags;
-	int err = 0;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (!chan->configured) {
 		dw_edma_terminate_all_descs(chan);
-	} else if (chan->status == EDMA_ST_PAUSE) {
-		dw_edma_terminate_all_descs(chan);
-		chan->status = EDMA_ST_IDLE;
-		chan->configured = false;
-	} else if (chan->status == EDMA_ST_IDLE) {
-		dw_edma_terminate_all_descs(chan);
-		chan->configured = false;
-	} else if (dw_edma_core_ch_status(chan) == DMA_COMPLETE) {
+	} else if (dw_edma_core_ch_status(chan) == DMA_IN_PROGRESS) {
+		/*
+		 * Defer the cleanup to the STOP interrupt. This also keeps a
+		 * pending STOP request idempotent and promotes a pending
+		 * PAUSE request to STOP.
+		 */
+		chan->request = EDMA_REQ_STOP;
+	} else {
 		/*
-		 * The channel is in a false BUSY state, probably didn't
-		 * receive or lost an interrupt
+		 * The channel is not running from the hardware point of view:
+		 * it has either never started or already stopped. No later
+		 * interrupt will clean up descriptors for us, nor can it
+		 * acknowledge a pending STOP or PAUSE request.
 		 */
 		dw_edma_terminate_all_descs(chan);
+		dw_edma_core_reset_ll(chan);
+		chan->request = EDMA_REQ_NONE;
 		chan->status = EDMA_ST_IDLE;
 		chan->configured = false;
-	} else if (chan->request > EDMA_REQ_PAUSE) {
-		err = -EPERM;
-	} else {
-		chan->request = EDMA_REQ_STOP;
 	}
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
-	return err;
+	return 0;
 }
 
 static void dw_edma_device_issue_pending(struct dma_chan *dchan)
@@ -713,11 +712,11 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
 	idx = dw_edma_core_ll_cur_idx(chan);
-	dw_edma_ll_recycle(chan, idx);
-	vd = vchan_next_desc(&chan->vc);
 
 	switch (chan->request) {
 	case EDMA_REQ_NONE:
+		dw_edma_ll_recycle(chan, idx);
+		vd = vchan_next_desc(&chan->vc);
 		if (vd) {
 			desc = vd2dw_edma_desc(vd);
 			if (desc->start_burst >= desc->nburst) {
@@ -738,18 +737,17 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 		break;
 
 	case EDMA_REQ_STOP:
-		if (vd) {
-			dw_edma_terminate_all_descs(chan);
-			chan->request = EDMA_REQ_NONE;
-			chan->status = EDMA_ST_IDLE;
-		}
+		dw_edma_terminate_all_descs(chan);
+		dw_edma_core_reset_ll(chan);
+		chan->request = EDMA_REQ_NONE;
+		chan->status = EDMA_ST_IDLE;
+		chan->configured = false;
 		break;
 
 	case EDMA_REQ_PAUSE:
-		if (vd) {
-			chan->request = EDMA_REQ_NONE;
-			chan->status = EDMA_ST_PAUSE;
-		}
+		dw_edma_ll_recycle(chan, idx);
+		chan->request = EDMA_REQ_NONE;
+		chan->status = EDMA_ST_PAUSE;
 		break;
 
 	default:
@@ -777,19 +775,19 @@ static void dw_edma_progress_interrupt(struct dw_edma_chan *chan)
 
 static void dw_edma_abort_interrupt(struct dw_edma_chan *chan)
 {
-	struct virt_dma_desc *vd;
+	struct virt_dma_desc *vd, *_vd;
 	unsigned long flags;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
-	vd = vchan_next_desc(&chan->vc);
-	if (vd) {
+	list_for_each_entry_safe(vd, _vd, &chan->vc.desc_issued, node) {
 		dw_hdma_set_callback_result(vd, DMA_TRANS_ABORTED);
 		list_del(&vd->node);
 		vchan_cookie_complete(vd);
 	}
-	spin_unlock_irqrestore(&chan->vc.lock, flags);
+	dw_edma_core_reset_ll(chan);
 	chan->request = EDMA_REQ_NONE;
 	chan->status = EDMA_ST_IDLE;
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
 
 static inline irqreturn_t dw_edma_interrupt_write(int irq, void *data)
-- 
2.51.0

[PATCH 15/17] dmaengine: dw-edma: Dynamically append requests while running

[Koichiro Den]

From: Frank Li <Frank.Li@nxp.com>

Use the LL producer-consumer state to append issued descriptors while
the channel is still running, instead of waiting for the current work to
drain.

Walk the issued list and append entries for any descriptor that still
has unstarted bursts. A descriptor that has already been fully appended
may still be pending in hardware, so keep walking; later descriptors can
use LL entries that have already been reclaimed.

Do not use dw_edma_start_transfer() as a channel-liveness test. It only
reports whether this pass appended new LL entries. Keep the software
state tied to pending LL entries, and keep doorbell decisions separate
from that state.

Doorbell only when runnable LL work was appended or when a stopped
channel still has pending LL entries. Recheck and kick under vc.lock so
the doorbell is serialized with request/status updates and LL reset.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
Co-developed-by: Koichiro Den <den@valinux.co.jp>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from Frank's original submission:
20260109-edma_dymatic-v1-4-9a98c9c98536@nxp.com
- Move LL progress cleanup, including dw_edma_ll_clean_pending(), into
  the earlier LL progress accounting patch.
- Continue past fully appended descriptors when appending later issued
  descriptors.
- Treat dw_edma_start_transfer() as "entries appended", not channel
  liveness.
- Derive BUSY/IDLE from pending LL entries.
- Tighten doorbelling in issue_pending(), resume(), and DONE handling so
  STOP/PAUSE paths are not re-kicked, and keep the kick under vc.lock.
- Keep Frank's eDMA ll_cur_idx() re-doorbell workaround for now; a later
  patch moves stopped-pending recovery into common tx_status() handling.
- Rephrase s/Need hold vc.lock/Must be called with vc.lock held./ for
  consistency.

 drivers/dma/dw-edma/dw-edma-core.c    | 121 ++++++++++++++++++++------
 drivers/dma/dw-edma/dw-edma-core.h    |   6 ++
 drivers/dma/dw-edma/dw-edma-v0-core.c |  22 ++++-
 3 files changed, 121 insertions(+), 28 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 4036adafedfa..477fc63e2778 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -84,6 +84,11 @@ static u32 dw_edma_core_get_ll_data_cnt(struct dw_edma_chan *chan)
 	return chan->ll_max - 1;
 }
 
+static bool dw_edma_core_has_flags(struct dw_edma_chan *chan, u32 flags)
+{
+	return chan->dw->core->flags & flags;
+}
+
 static u32 dw_edma_core_get_ll_dist(struct dw_edma_chan *chan, u32 from, u32 to)
 {
 	u32 cnt = dw_edma_core_get_ll_data_cnt(chan);
@@ -139,6 +144,31 @@ static bool dw_edma_ll_pending(struct dw_edma_chan *chan)
 	return dw_edma_core_get_used_num(chan);
 }
 
+static bool dw_edma_ll_stopped_pending(struct dw_edma_chan *chan)
+{
+	return dw_edma_ll_pending(chan) &&
+	       dw_edma_core_ch_status(chan) == DMA_COMPLETE;
+}
+
+static bool dw_edma_ll_recoverable_pending(struct dw_edma_chan *chan)
+{
+	return chan->request == EDMA_REQ_NONE &&
+	       chan->status != EDMA_ST_PAUSE &&
+	       dw_edma_ll_stopped_pending(chan);
+}
+
+/* Must be called with vc.lock held. */
+static void
+dw_edma_core_ch_doorbell_recheck(struct dw_edma_chan *chan, bool doorbell)
+{
+	if (!doorbell && !dw_edma_ll_recoverable_pending(chan))
+		return;
+
+	/* Serialize the kick with channel state changes and LL reset. */
+	dw_edma_core_ch_doorbell(chan);
+}
+
+/* Must be called with vc.lock held. */
 static void dw_edma_core_start(struct dw_edma_desc *desc)
 {
 	struct dw_edma_chan *chan = desc->chan;
@@ -173,30 +203,36 @@ static void dw_edma_core_start(struct dw_edma_desc *desc)
 
 	desc->start_burst = i;
 	desc->ll_end = chan->ll_head;
-
-	dw_edma_core_ch_doorbell(chan);
 }
 
+/* Must be called with vc.lock held. */
 static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 {
 	struct dw_edma_desc *desc;
 	struct virt_dma_desc *vd;
 	int index = dw_edma_core_ll_cur_idx(chan);
+	int ret = 0;
 
 	if (index < 0)
 		dw_edma_core_reset_ll(chan);
 
-	vd = vchan_next_desc(&chan->vc);
-	if (!vd)
-		return 0;
+	list_for_each_entry(vd, &chan->vc.desc_issued, node) {
+		if (!dw_edma_core_get_free_num(chan))
+			return ret;
 
-	desc = vd2dw_edma_desc(vd);
-	if (!desc)
-		return 0;
+		desc = vd2dw_edma_desc(vd);
 
-	dw_edma_core_start(desc);
+		/*
+		 * Fully appended descriptors may still be pending. Keep walking
+		 * so later descriptors can use newly freed LL entries.
+		 */
+		if (desc->start_burst == desc->nburst)
+			continue;
+		dw_edma_core_start(desc);
+		ret = 1;
+	}
 
-	return 1;
+	return ret;
 }
 
 static void dw_hdma_set_callback_result(struct virt_dma_desc *vd,
@@ -375,6 +411,7 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 	unsigned long flags;
+	bool doorbell = false;
 	int err = 0;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
@@ -385,8 +422,10 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
 	} else if (chan->request != EDMA_REQ_NONE) {
 		err = -EPERM;
 	} else {
-		chan->status = EDMA_ST_BUSY;
 		dw_edma_start_transfer(chan);
+		doorbell = dw_edma_ll_pending(chan);
+		chan->status = doorbell ? EDMA_ST_BUSY : EDMA_ST_IDLE;
+		dw_edma_core_ch_doorbell_recheck(chan, doorbell);
 	}
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
@@ -430,13 +469,19 @@ static void dw_edma_device_issue_pending(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
 	unsigned long flags;
+	bool doorbell = false;
+	bool pending;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
-	if (chan->configured && vchan_issue_pending(&chan->vc) &&
-	    chan->request == EDMA_REQ_NONE &&
-	    chan->status == EDMA_ST_IDLE) {
+	if (!chan->configured)
+		pending = false;
+	else
+		pending = vchan_issue_pending(&chan->vc);
+	if (pending && chan->request == EDMA_REQ_NONE &&
+	    chan->status != EDMA_ST_PAUSE) {
 		chan->status = EDMA_ST_BUSY;
-		dw_edma_start_transfer(chan);
+		doorbell = dw_edma_start_transfer(chan);
+		dw_edma_core_ch_doorbell_recheck(chan, doorbell);
 	}
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
@@ -451,7 +496,18 @@ dw_edma_device_tx_status(struct dma_chan *dchan, dma_cookie_t cookie,
 	unsigned long flags;
 	enum dma_status ret;
 	u32 residue = 0;
+	int idx;
 
+	ret = dma_cookie_status(dchan, cookie, txstate);
+	if (ret == DMA_COMPLETE)
+		return ret;
+
+	spin_lock_irqsave(&chan->vc.lock, flags);
+	idx = dw_edma_core_ll_cur_idx(chan);
+	dw_edma_ll_recycle_and_refill(chan, idx);
+	spin_unlock_irqrestore(&chan->vc.lock, flags);
+
+	/* check again because dw_edma_ll_clean_pending() may update cookie */
 	ret = dma_cookie_status(dchan, cookie, txstate);
 	if (ret == DMA_COMPLETE)
 		return ret;
@@ -705,9 +761,9 @@ dw_edma_device_prep_interleaved_dma(struct dma_chan *dchan,
 
 static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 {
-	struct dw_edma_desc *desc;
 	struct virt_dma_desc *vd;
 	unsigned long flags;
+	bool doorbell = false;
 	int idx;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
@@ -718,22 +774,23 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 		dw_edma_ll_recycle(chan, idx);
 		vd = vchan_next_desc(&chan->vc);
 		if (vd) {
-			desc = vd2dw_edma_desc(vd);
-			if (desc->start_burst >= desc->nburst) {
-				dw_hdma_set_callback_result(vd,
-							    DMA_TRANS_NOERROR);
-				list_del(&vd->node);
-				vchan_cookie_complete(vd);
-				chan->ll_end = desc->ll_end;
-			}
-
-			/* Continue transferring if there are remaining chunks or issued requests.
+			/*
+			 * dw_edma_start_transfer() reports whether new entries
+			 * were appended. Channel liveness follows the LL
+			 * producer/consumer state.
 			 */
-			chan->status = dw_edma_start_transfer(chan) ? EDMA_ST_BUSY : EDMA_ST_IDLE;
+			if (dw_edma_core_has_flags(chan, DW_EDMA_CORE_FLAG_DONE_IRQ_DOORBELL))
+				doorbell = true;
+			doorbell |= dw_edma_start_transfer(chan);
+			chan->status = dw_edma_ll_pending(chan) ?
+					EDMA_ST_BUSY : EDMA_ST_IDLE;
 		} else {
 			chan->status = dw_edma_ll_pending(chan) ?
 				       EDMA_ST_BUSY : EDMA_ST_IDLE;
 		}
+
+		if (!doorbell && dw_edma_ll_recoverable_pending(chan))
+			doorbell = true;
 		break;
 
 	case EDMA_REQ_STOP:
@@ -753,6 +810,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 	default:
 		break;
 	}
+	dw_edma_core_ch_doorbell_recheck(chan, doorbell);
 
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
@@ -760,6 +818,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 static void dw_edma_progress_interrupt(struct dw_edma_chan *chan)
 {
 	unsigned long flags;
+	bool doorbell = false;
 	int idx;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
@@ -768,7 +827,15 @@ static void dw_edma_progress_interrupt(struct dw_edma_chan *chan)
 		dw_edma_ll_recycle_and_refill(chan, idx);
 		chan->status = dw_edma_ll_pending(chan) ?
 			       EDMA_ST_BUSY : EDMA_ST_IDLE;
+
+		/*
+		 * The channel may have stopped after the progress point was
+		 * sampled. Re-kick it if LL work remains pending.
+		 */
+		if (dw_edma_ll_recoverable_pending(chan))
+			doorbell = true;
 	}
+	dw_edma_core_ch_doorbell_recheck(chan, doorbell);
 
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 1252d264c1ca..27a0521c989c 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -9,11 +9,15 @@
 #ifndef _DW_EDMA_CORE_H
 #define _DW_EDMA_CORE_H
 
+#include <linux/bits.h>
 #include <linux/msi.h>
 #include <linux/dma/edma.h>
 
 #include "../virt-dma.h"
 
+/* Force a doorbell after DONE IRQ handling to recover lost starts. */
+#define DW_EDMA_CORE_FLAG_DONE_IRQ_DOORBELL		BIT(0)
+
 #define EDMA_LL_SZ					24
 
 enum dw_edma_dir {
@@ -140,6 +144,8 @@ struct dw_edma {
 typedef void (*dw_edma_handler_t)(struct dw_edma_chan *);
 
 struct dw_edma_core_ops {
+	u32 flags;
+
 	void (*off)(struct dw_edma *dw);
 	u16 (*ch_count)(struct dw_edma *dw, enum dw_edma_dir dir);
 	enum dma_status (*ch_status)(struct dw_edma_chan *chan);
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index 265eefbf2ead..a5ffb0e77602 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -516,7 +516,6 @@ static void dw_edma_v0_core_ch_doorbell(struct dw_edma_chan *chan)
 
 	dw_edma_v0_sync_ll_data(chan);
 
-	/* Doorbell */
 	SET_RW_32(dw, chan->dir, doorbell,
 		  FIELD_PREP(EDMA_V0_DOORBELL_CH_MASK, chan->id));
 }
@@ -534,6 +533,26 @@ static int dw_edma_v0_core_ll_cur_idx(struct dw_edma_chan *chan)
 	if (!val)
 		return -EINVAL;
 
+	/*
+	 * Doorbell will be missed if DMA engine running, so last update
+	 * descriptor have not fetched by DMA engine, so DMA engine stop.
+	 *
+	 *	Most like issue happen at
+	 *
+	 *	  DMA Engine		|	SW
+	 *        ======================================
+	 *  1     send Read req for LL
+	 *  2					update LL
+	 *  3					doorbell
+	 *  4	  *Missed doorbell*
+	 *  5     Get old LL data
+	 *  6     DMA stop
+	 *
+	 * Workaround: Push doorbell again when found DMA stop.
+	 */
+	if (dw_edma_v0_core_ch_status(chan) != DMA_IN_PROGRESS)
+		dw_edma_v0_core_ch_doorbell(chan);
+
 	return (val - (paddr & 0xFFFFFFFF)) / EDMA_LL_SZ;
 }
 
@@ -553,6 +572,7 @@ static void dw_edma_v0_core_debugfs_on(struct dw_edma *dw)
 }
 
 static const struct dw_edma_core_ops dw_edma_v0_core = {
+	.flags = DW_EDMA_CORE_FLAG_DONE_IRQ_DOORBELL,
 	.off = dw_edma_v0_core_off,
 	.ch_count = dw_edma_v0_core_ch_count,
 	.ch_status = dw_edma_v0_core_ch_status,
-- 
2.51.0

[PATCH 16/17] dmaengine: dw-edma: Recover stopped HDMA from tx_status

[Koichiro Den]

HDMA can stop after software has appended new LL entries. If that
happens without another interrupt, synchronous users polling tx_status()
can wait until timeout even though the channel has pending work.

Let tx_status() re-doorbell when the channel is stopped with pending LL
entries. Use the same recoverable-pending guard as the interrupt path so
paused channels are not re-kicked, and issue the recovery kick under
vc.lock so it is serialized with request/status updates and LL reset.

Do not use tx_status() as the normal running-progress path for HDMA.
HDMA has watermark interrupts for that. Polling the running LLP from
every dma_sync_wait() iteration adds MMIO and list-walk overhead, and
can duplicate the IRQ progress path. Keep LL polling in tx_status() for
eDMA, which has no separate progress interrupt, and for HDMA only when
stopped-pending recovery is needed.

For cores that do poll LL progress from tx_status(), avoid a second LL
progress pass in the stopped-pending recovery path. If the channel
stopped just after the first sample, re-doorbelling is enough for this
tx_status() call; a later interrupt or poll can observe further
progress.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/dw-edma-core.c    | 35 ++++++++++++++++++++++++---
 drivers/dma/dw-edma/dw-edma-core.h    |  2 ++
 drivers/dma/dw-edma/dw-edma-v0-core.c | 23 ++----------------
 3 files changed, 36 insertions(+), 24 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 477fc63e2778..acf6cc8147a6 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -495,16 +495,45 @@ dw_edma_device_tx_status(struct dma_chan *dchan, dma_cookie_t cookie,
 	struct virt_dma_desc *vd;
 	unsigned long flags;
 	enum dma_status ret;
+	bool doorbell = false;
+	bool progress_sampled = false;
 	u32 residue = 0;
-	int idx;
+	int idx = -EINVAL;
 
 	ret = dma_cookie_status(dchan, cookie, txstate);
 	if (ret == DMA_COMPLETE)
 		return ret;
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
-	idx = dw_edma_core_ll_cur_idx(chan);
-	dw_edma_ll_recycle_and_refill(chan, idx);
+	/*
+	 * eDMA has no separate progress interrupt, so tx_status() polls the
+	 * running LLP. HDMA uses watermark interrupts for normal progress and
+	 * only samples LLP here for the stopped-pending recovery below.
+	 */
+	if (chan->request == EDMA_REQ_NONE &&
+	    dw_edma_core_has_flags(chan, DW_EDMA_CORE_FLAG_TX_STATUS_POLL)) {
+		idx = dw_edma_core_ll_cur_idx(chan);
+		dw_edma_ll_recycle_and_refill(chan, idx);
+		progress_sampled = true;
+	}
+
+	if (dw_edma_ll_recoverable_pending(chan)) {
+		/*
+		 * Cores without tx_status() polling, such as HDMA, need a
+		 * stopped-channel LLP sample for recovery. Polling cores already
+		 * sampled progress above, so avoid a second pass here.
+		 */
+		if (!progress_sampled) {
+			idx = dw_edma_core_ll_cur_idx(chan);
+			dw_edma_ll_recycle_and_refill(chan, idx);
+		}
+
+		if (dw_edma_ll_pending(chan)) {
+			doorbell = true;
+			chan->status = EDMA_ST_BUSY;
+		}
+	}
+	dw_edma_core_ch_doorbell_recheck(chan, doorbell);
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
 	/* check again because dw_edma_ll_clean_pending() may update cookie */
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 27a0521c989c..1bacefb10a3b 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -17,6 +17,8 @@
 
 /* Force a doorbell after DONE IRQ handling to recover lost starts. */
 #define DW_EDMA_CORE_FLAG_DONE_IRQ_DOORBELL		BIT(0)
+/* Poll LL progress from tx_status() for cores without progress IRQs. */
+#define DW_EDMA_CORE_FLAG_TX_STATUS_POLL		BIT(1)
 
 #define EDMA_LL_SZ					24
 
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index a5ffb0e77602..15e4779cbc01 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -533,26 +533,6 @@ static int dw_edma_v0_core_ll_cur_idx(struct dw_edma_chan *chan)
 	if (!val)
 		return -EINVAL;
 
-	/*
-	 * Doorbell will be missed if DMA engine running, so last update
-	 * descriptor have not fetched by DMA engine, so DMA engine stop.
-	 *
-	 *	Most like issue happen at
-	 *
-	 *	  DMA Engine		|	SW
-	 *        ======================================
-	 *  1     send Read req for LL
-	 *  2					update LL
-	 *  3					doorbell
-	 *  4	  *Missed doorbell*
-	 *  5     Get old LL data
-	 *  6     DMA stop
-	 *
-	 * Workaround: Push doorbell again when found DMA stop.
-	 */
-	if (dw_edma_v0_core_ch_status(chan) != DMA_IN_PROGRESS)
-		dw_edma_v0_core_ch_doorbell(chan);
-
 	return (val - (paddr & 0xFFFFFFFF)) / EDMA_LL_SZ;
 }
 
@@ -572,7 +552,8 @@ static void dw_edma_v0_core_debugfs_on(struct dw_edma *dw)
 }
 
 static const struct dw_edma_core_ops dw_edma_v0_core = {
-	.flags = DW_EDMA_CORE_FLAG_DONE_IRQ_DOORBELL,
+	.flags = DW_EDMA_CORE_FLAG_DONE_IRQ_DOORBELL |
+		 DW_EDMA_CORE_FLAG_TX_STATUS_POLL,
 	.off = dw_edma_v0_core_off,
 	.ch_count = dw_edma_v0_core_ch_count,
 	.ch_status = dw_edma_v0_core_ch_status,
-- 
2.51.0

[PATCH 17/17] dmaengine: dw-edma: Add trace support

[Koichiro Den]

From: Frank Li <Frank.Li@nxp.com>

Add tracepoints for the linked-list fill, start, interrupt, tx_status(),
and completion paths. These are useful when debugging dynamic
linked-list appends and HDMA watermark progress handling. The LL
progress trace records ll_head, ll_end, and ll_done so it shows both the
descriptor completion boundary and the hardware consumption point.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
[den: dropped needless new-lines and unused edma_terminate_all event;
 included ll_done in LL progress traces; added trace_edma_irq() in
 dw_edma_progress_interrupt; trace descriptor completion before vchan
 clears the cookie; fixed minor checkpatch.pl cosmetic issues; updated
 commit message]
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
 drivers/dma/dw-edma/Makefile        |   3 +
 drivers/dma/dw-edma/dw-edma-core.c  |  13 +++
 drivers/dma/dw-edma/dw-edma-core.h  |   2 +
 drivers/dma/dw-edma/dw-edma-trace.c |   4 +
 drivers/dma/dw-edma/dw-edma-trace.h | 150 ++++++++++++++++++++++++++++
 5 files changed, 172 insertions(+)
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h

diff --git a/drivers/dma/dw-edma/Makefile b/drivers/dma/dw-edma/Makefile
index 83ab58f87760..3e31e7d92f3e 100644
--- a/drivers/dma/dw-edma/Makefile
+++ b/drivers/dma/dw-edma/Makefile
@@ -1,9 +1,12 @@
 # SPDX-License-Identifier: GPL-2.0
 
+dw-edma-trace-$(CONFIG_TRACING)	:= dw-edma-trace.o
+CFLAGS_dw-edma-trace.o		:= -I$(src)
 obj-$(CONFIG_DW_EDMA)		+= dw-edma.o
 dw-edma-$(CONFIG_DEBUG_FS)	:= dw-edma-v0-debugfs.o	\
 				   dw-hdma-v0-debugfs.o
 dw-edma-objs			:= dw-edma-core.o	\
 				   dw-edma-v0-core.o	\
+				   ${dw-edma-trace-y} \
 				   dw-hdma-v0-core.o $(dw-edma-y)
 obj-$(CONFIG_DW_EDMA_PCIE)	+= dw-edma-pcie.o
diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index acf6cc8147a6..ad7e2a2fd685 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -193,6 +193,12 @@ static void dw_edma_core_start(struct dw_edma_desc *desc)
 				     chan->ll_head, chan->cb,
 				     dw_edma_core_enable_ll_irq(desc, i, free));
 
+		trace_edma_fill_ll(chan, chan->ll_head,
+				   desc->vd.tx.cookie,
+				   desc->burst[i].sar,
+				   desc->burst[i].dar, desc->burst[i].sz,
+				   chan->cb);
+
 		chan->ll_head++;
 
 		if (chan->ll_head == chan->ll_max - 1) {
@@ -228,6 +234,8 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 		 */
 		if (desc->start_burst == desc->nburst)
 			continue;
+
+		trace_edma_start_desc(desc);
 		dw_edma_core_start(desc);
 		ret = 1;
 	}
@@ -322,6 +330,7 @@ static void dw_edma_ll_clean_pending(struct dw_edma_chan *chan, u32 old_done)
 		/* Hardware has consumed this descriptor's LL entries. */
 		dw_hdma_set_callback_result(vd, DMA_TRANS_NOERROR);
 		list_del(&vd->node);
+		trace_edma_complete_desc(desc);
 		vchan_cookie_complete(vd);
 		chan->ll_end = desc->ll_end;
 	}
@@ -536,6 +545,8 @@ dw_edma_device_tx_status(struct dma_chan *dchan, dma_cookie_t cookie,
 	dw_edma_core_ch_doorbell_recheck(chan, doorbell);
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 
+	trace_edma_tx_status_info(chan, idx);
+
 	/* check again because dw_edma_ll_clean_pending() may update cookie */
 	ret = dma_cookie_status(dchan, cookie, txstate);
 	if (ret == DMA_COMPLETE)
@@ -797,6 +808,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
 	idx = dw_edma_core_ll_cur_idx(chan);
+	trace_edma_irq(chan, idx);
 
 	switch (chan->request) {
 	case EDMA_REQ_NONE:
@@ -852,6 +864,7 @@ static void dw_edma_progress_interrupt(struct dw_edma_chan *chan)
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
 	idx = dw_edma_core_ll_cur_idx(chan);
+	trace_edma_irq(chan, idx);
 	if (chan->request == EDMA_REQ_NONE && chan->status != EDMA_ST_PAUSE) {
 		dw_edma_ll_recycle_and_refill(chan, idx);
 		chan->status = dw_edma_ll_pending(chan) ?
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 1bacefb10a3b..a72469c0d262 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -143,6 +143,8 @@ struct dw_edma {
 	const struct dw_edma_core_ops	*core;
 };
 
+#include "dw-edma-trace.h"
+
 typedef void (*dw_edma_handler_t)(struct dw_edma_chan *);
 
 struct dw_edma_core_ops {
diff --git a/drivers/dma/dw-edma/dw-edma-trace.c b/drivers/dma/dw-edma/dw-edma-trace.c
new file mode 100644
index 000000000000..2620ad61a943
--- /dev/null
+++ b/drivers/dma/dw-edma/dw-edma-trace.c
@@ -0,0 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define CREATE_TRACE_POINTS
+#include "dw-edma-core.h"
diff --git a/drivers/dma/dw-edma/dw-edma-trace.h b/drivers/dma/dw-edma/dw-edma-trace.h
new file mode 100644
index 000000000000..a908096156d4
--- /dev/null
+++ b/drivers/dma/dw-edma/dw-edma-trace.h
@@ -0,0 +1,150 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright 2023 NXP.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM dw_edma
+
+#if !defined(__LINUX_DW_EDMA_TRACE) || defined(TRACE_HEADER_MULTI_READ)
+#define __LINUX_DW_EDMA_TRACE
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+
+DECLARE_EVENT_CLASS(edma_desc_info,
+	TP_PROTO(struct dw_edma_desc *desc),
+	TP_ARGS(desc),
+	TP_STRUCT__entry(
+		__field(u32, nburst)
+		__field(u32, start_burst)
+		__field(u32, ll_end)
+		__field(u32, cookie)
+		__field(u32, id)
+		__field(u8, dir)
+	),
+	TP_fast_assign(
+		__entry->nburst = desc->nburst,
+		__entry->start_burst = desc->start_burst,
+		__entry->ll_end = desc->ll_end,
+		__entry->id = desc->chan->id,
+		__entry->dir = desc->chan->dir,
+		__entry->cookie = desc->vd.tx.cookie;
+	),
+	TP_printk("chan %d%c desc %d, nburst %d, start_burst %d, ll_end %d",
+		__entry->id,
+		__entry->dir ? 'R' : 'W',
+		__entry->cookie,
+		__entry->nburst,
+		__entry->start_burst,
+		__entry->ll_end)
+);
+
+DEFINE_EVENT(edma_desc_info, edma_start_desc,
+	TP_PROTO(struct dw_edma_desc *desc),
+	TP_ARGS(desc)
+);
+
+DEFINE_EVENT(edma_desc_info, edma_complete_desc,
+	TP_PROTO(struct dw_edma_desc *desc),
+	TP_ARGS(desc)
+);
+
+DECLARE_EVENT_CLASS(edma_ll_info,
+	TP_PROTO(struct dw_edma_chan *chan, int idx),
+	TP_ARGS(chan, idx),
+	TP_STRUCT__entry(
+		__field(u32, head)
+		__field(u32, end)
+		__field(u32, done)
+		__field(u32, total)
+		__field(u32, index)
+		__field(u32, completed_cookie)
+		__field(u32, cookie)
+		__field(u32, id)
+		__field(u8, dir)
+	),
+	TP_fast_assign(
+		__entry->head = chan->ll_head,
+		__entry->end = chan->ll_end,
+		__entry->done = chan->ll_done,
+		__entry->total = chan->ll_max,
+		__entry->index = idx,
+		__entry->completed_cookie = chan->vc.chan.completed_cookie,
+		__entry->cookie = chan->vc.chan.cookie,
+		__entry->id = chan->id,
+		__entry->dir = chan->dir;
+	),
+	TP_printk("chan %d%c head: %d end: %d done: %d: dma cur index: %d, complete cookie: %d, cookie: %d",
+		__entry->id,
+		__entry->dir ? 'R' : 'W',
+		__entry->head,
+		__entry->end,
+		__entry->done,
+		__entry->index,
+		__entry->completed_cookie,
+		__entry->cookie)
+);
+
+DEFINE_EVENT(edma_ll_info, edma_tx_status_info,
+	TP_PROTO(struct dw_edma_chan *chan, int idx),
+	TP_ARGS(chan, idx)
+);
+
+DEFINE_EVENT(edma_ll_info, edma_irq,
+	TP_PROTO(struct dw_edma_chan *chan, int idx),
+	TP_ARGS(chan, idx)
+);
+
+DECLARE_EVENT_CLASS(edma_log_ll,
+	TP_PROTO(struct dw_edma_chan *chan, u32 idx, u32 cookie, u64 src,
+		 u64 dest, u32 sz, bool flag),
+	TP_ARGS(chan, idx, cookie, src, dest, sz, flag),
+	TP_STRUCT__entry(
+		__field(u32, idx)
+		__field(u64, src)
+		__field(u64, dest)
+		__field(u32, sz)
+		__field(u32, id)
+		__field(u32, cookie)
+		__field(bool, flag)
+		__field(u8, dir)
+	),
+	TP_fast_assign(
+		__entry->idx = idx,
+		__entry->src = src,
+		__entry->dest = dest,
+		__entry->sz = sz,
+		__entry->id = chan->id,
+		__entry->dir = chan->dir,
+		__entry->cookie = cookie,
+		__entry->flag = flag;
+	),
+	TP_printk("chan %d%c %d [%d] %c src: %08llx dest: %08llx sz: %04x",
+		__entry->id,
+		__entry->dir ? 'R' : 'W',
+		__entry->cookie,
+		__entry->idx,
+		__entry->flag ? 'C' : 'c',
+		__entry->src,
+		__entry->dest,
+		__entry->sz)
+);
+
+DEFINE_EVENT(edma_log_ll, edma_fill_ll,
+	TP_PROTO(struct dw_edma_chan *chan, u32 idx, u32 cookie, u64 src,
+		 u64 dest, u32 sz, bool flag),
+	TP_ARGS(chan, idx, cookie, src, dest, sz, flag)
+);
+
+#endif
+
+/* this part must be outside header guard */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE dw-edma-trace
+
+#include <trace/define_trace.h>
-- 
2.51.0

Re: [PATCH 01/17] dmaengine: dw-edma: Fix residue burst index in tx_status()

[Frank Li]

On Tue, Jun 16, 2026 at 12:40:55AM +0900, Koichiro Den wrote:
> dw_edma_device_tx_status() uses desc->done_burst to subtract the
> completed byte count from the descriptor size. done_burst is a count of
> completed bursts, not the zero-based index of the last completed burst.
>
> Index desc->burst[] with done_burst - 1. Otherwise tx_status() reads the
> next burst's cumulative transfer size, which under-reports the residue
> and can become a one-past-the-end access when all bursts have completed.
>
> While at it, return early when txstate is NULL and drop the redundant
> desc check after vd2dw_edma_desc(). These are minor clean-ups since
> dma_set_residue() already tolerates a NULL state, and vd2dw_edma_desc()
> is only reached for a valid vdesc.
>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
> @Frank, if you plan to respin 20260109-edma_ll-v2-0-5c0b27b2c664@nxp.com
> and agree with this patch, consider folding this fix into your patch
> when submitting your v3.

Okay, after config and prep pick, I will squash this into above one

Frank
>
>  drivers/dma/dw-edma/dw-edma-core.c | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index 1c8aef5e03b0..d99b6256660a 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -244,7 +244,7 @@ dw_edma_device_tx_status(struct dma_chan *dchan, dma_cookie_t cookie,
>  		ret = DMA_PAUSED;
>
>  	if (!txstate)
> -		goto ret_residue;
> +		return ret;
>
>  	spin_lock_irqsave(&chan->vc.lock, flags);
>  	vd = vchan_find_desc(&chan->vc, cookie);
> @@ -252,12 +252,11 @@ dw_edma_device_tx_status(struct dma_chan *dchan, dma_cookie_t cookie,
>  		desc = vd2dw_edma_desc(vd);
>
>  		residue = desc->alloc_sz;
> -		if (desc && desc->done_burst)
> -			residue -= desc->burst[desc->done_burst].xfer_sz;
> +		if (desc->done_burst)
> +			residue -= desc->burst[desc->done_burst - 1].xfer_sz;
>  	}
>  	spin_unlock_irqrestore(&chan->vc.lock, flags);
>
> -ret_residue:
>  	dma_set_residue(txstate, residue);
>
>  	return ret;
> --
> 2.51.0
>

Re: [PATCH 02/17] dmaengine: dw-edma: Fix HDMA channel status register access

[Frank Li]

On Tue, Jun 16, 2026 at 12:40:56AM +0900, Koichiro Den wrote:
> GET_CH_32() takes the direction before the channel ID, but
> dw_hdma_v0_core_ch_status() passed them in the opposite order. This can
> make the status callback read another HDMA channel status register.
>
> Use the same argument order as the other HDMA register accesses.
>
> Fixes: e74c39573d35 ("dmaengine: dw-edma: Add support for native HDMA")
> Cc: stable@vger.kernel.org
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
> Given Devendra's comment on
> a28adc76-044b-4666-bda0-d7f9a8d52a63@amd.com,
> I expect he will soon submit a very similar patch. If so, please prefer
> his patch over this one if it works. I included this fix here since the
> rest of this series makes this pre-existing bug easier to hit.

I provide review-by here, so vnod can be free pick any one.

Reviewed-by: Frank Li <Frank.Li@nxp.com>
>
>  drivers/dma/dw-edma/dw-hdma-v0-core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
> index 7f9fe3a6edd9..862375c8e4ba 100644
> --- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
> +++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
> @@ -79,7 +79,7 @@ static enum dma_status dw_hdma_v0_core_ch_status(struct dw_edma_chan *chan)
>  	u32 tmp;
>
>  	tmp = FIELD_GET(HDMA_V0_CH_STATUS_MASK,
> -			GET_CH_32(dw, chan->id, chan->dir, ch_stat));
> +			GET_CH_32(dw, chan->dir, chan->id, ch_stat));
>
>  	if (tmp == 1)
>  		return DMA_IN_PROGRESS;
> --
> 2.51.0
>

Re: [PATCH 03/17] dmaengine: dw-edma: Terminate STOP requests without callbacks

[Frank Li]

On Tue, Jun 16, 2026 at 12:40:57AM +0900, Koichiro Den wrote:
> The STOP request path handles device_terminate_all(). The DMA Engine
> client documentation says in the "Terminate APIs" section of
> Documentation/driver-api/dmaengine/client.rst:
>
>   "No callback functions will be called for any incomplete transfers."
>
> dw-edma used vchan_cookie_complete() for a stopped descriptor. This
> queues the descriptor on the completed list and schedules its callback.
> A late callback after dmaengine_terminate_sync() can dereference
> callback state, such as a request object, that the client has already
> freed.
>
> Move stopped descriptors to the terminated list. Complete the cookie
> before doing so, so cookie polling observes that the transfer is no
> longer in flight, but do not schedule the completion callback. Add a
> synchronize callback so virt-dma can release terminated descriptors.
>
> Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
>  drivers/dma/dw-edma/dw-edma-core.c | 18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index d99b6256660a..bedaee6d30ab 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -106,6 +106,13 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
>  	return 1;
>  }
>
> +static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
> +{
> +	list_del(&vd->node);
> +	dma_cookie_complete(&vd->tx);
> +	vchan_terminate_vdesc(vd);

Is it vchan_terminate_vdesc() missing call dma_cookie_complete()?

> +}
> +
>  static void dw_edma_device_caps(struct dma_chan *dchan,
>  				struct dma_slave_caps *caps)
>  {
> @@ -537,8 +544,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
>  			break;
>
>  		case EDMA_REQ_STOP:
> -			list_del(&vd->node);
> -			vchan_cookie_complete(vd);
> +			dw_edma_terminate_vdesc(vd);
>  			chan->request = EDMA_REQ_NONE;
>  			chan->status = EDMA_ST_IDLE;
>  			break;
> @@ -610,6 +616,13 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
>  	return 0;
>  }
>
> +static void dw_edma_device_synchronize(struct dma_chan *dchan)
> +{
> +	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +
> +	vchan_synchronize(&chan->vc);
> +}
> +
>  static void dw_edma_free_chan_resources(struct dma_chan *dchan)
>  {
>  	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
> @@ -723,6 +736,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
>  	dma->device_pause = dw_edma_device_pause;
>  	dma->device_resume = dw_edma_device_resume;
>  	dma->device_terminate_all = dw_edma_device_terminate_all;
> +	dma->device_synchronize = dw_edma_device_synchronize;

Can we provide generally call back like, vchan_synchroniz_dmachan(), or
change existing vchan_synchronize() to by using struct dma_chan *dchan.

avoid duplicate it every dmaegine drivers.

Frank

>  	dma->device_issue_pending = dw_edma_device_issue_pending;
>  	dma->device_tx_status = dw_edma_device_tx_status;
>  	dma->device_prep_slave_sg_config = dw_edma_device_prep_slave_sg_config;
> --
> 2.51.0
>

Re: [PATCH 04/17] dmaengine: dw-edma: Clean up vchan descriptors on termination

[Frank Li]

On Tue, Jun 16, 2026 at 12:40:58AM +0900, Koichiro Den wrote:
> dw-edma resets channel state from terminate_all() paths, but pending
> virt-dma descriptors can remain on the submitted and issued lists. A
> later issue_pending() may then restart work that the client already
> terminated, possibly into buffers that were already reused. Descriptors
> that are never restarted leak instead.
>
> Move issued and submitted descriptors to the terminated list whenever a
> termination request completes. Also release virt-dma resources from
> free_chan_resources().
>
> If termination was deferred because the channel was still running, wait
> until the STOP path deconfigures the channel before synchronizing or
> freeing virt-dma resources. Otherwise dmaengine_terminate_sync() can
> return before the deferred STOP cleanup has moved issued descriptors to
> the terminated list and before the channel is known to have stopped.
>
> The old free_chan_resources() loop usually broke as soon as
> terminate_all() returned zero, so it did not effectively spin until the
> timeout. This wait can now last until the existing timeout, so use
> cond_resched() instead of busy-polling with cpu_relax(), and warn if the
> timeout expires.
>
> Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
>  drivers/dma/dw-edma/dw-edma-core.c | 78 ++++++++++++++++++++++++------
>  1 file changed, 64 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index bedaee6d30ab..2777dc0b2aed 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -15,6 +15,7 @@
>  #include <linux/irq.h>
>  #include <linux/dma/edma.h>
>  #include <linux/dma-mapping.h>
> +#include <linux/sched.h>
>  #include <linux/string_choices.h>
>
>  #include "dw-edma-core.h"
> @@ -113,6 +114,28 @@ static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
>  	vchan_terminate_vdesc(vd);
>  }
>
> +static void dw_edma_terminate_vdesc_list(struct list_head *head)
> +{
> +	struct virt_dma_desc *vd, *_vd;
> +
> +	list_for_each_entry_safe(vd, _vd, head, node)
> +		dw_edma_terminate_vdesc(vd);
> +}
> +
> +/* Must be called with vc.lock held. */
> +static void dw_edma_terminate_all_descs(struct dw_edma_chan *chan)
> +{
> +	/*
> +	 * This order must not be reversed. Cookies are assigned when
> +	 * descriptors are submitted, so desc_issued contains older cookies
> +	 * than desc_submitted. Completing desc_submitted first could move
> +	 * chan->vc.chan.completed_cookie backwards when desc_issued is
> +	 * terminated afterwards.
> +	 */
> +	dw_edma_terminate_vdesc_list(&chan->vc.desc_issued);
> +	dw_edma_terminate_vdesc_list(&chan->vc.desc_submitted);
> +}
> +

Is it possible move every thing to temp termniate queue by hold lock, then
call dw_edma_terminate_vdesc(vd) outside lock.

Frank
>  static void dw_edma_device_caps(struct dma_chan *dchan,
>  				struct dma_slave_caps *caps)
>  {
> @@ -190,20 +213,25 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
>  static int dw_edma_device_terminate_all(struct dma_chan *dchan)
>  {
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +	unsigned long flags;
>  	int err = 0;
>
> +	spin_lock_irqsave(&chan->vc.lock, flags);
>  	if (!chan->configured) {
> -		/* Do nothing */
> +		dw_edma_terminate_all_descs(chan);
>  	} else if (chan->status == EDMA_ST_PAUSE) {
> +		dw_edma_terminate_all_descs(chan);
>  		chan->status = EDMA_ST_IDLE;
>  		chan->configured = false;
>  	} else if (chan->status == EDMA_ST_IDLE) {
> +		dw_edma_terminate_all_descs(chan);
>  		chan->configured = false;
>  	} else if (dw_edma_core_ch_status(chan) == DMA_COMPLETE) {
>  		/*
>  		 * The channel is in a false BUSY state, probably didn't
>  		 * receive or lost an interrupt
>  		 */
> +		dw_edma_terminate_all_descs(chan);
>  		chan->status = EDMA_ST_IDLE;
>  		chan->configured = false;
>  	} else if (chan->request > EDMA_REQ_PAUSE) {
> @@ -211,6 +239,7 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
>  	} else {
>  		chan->request = EDMA_REQ_STOP;
>  	}
> +	spin_unlock_irqrestore(&chan->vc.lock, flags);
>
>  	return err;
>  }
> @@ -544,7 +573,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
>  			break;
>
>  		case EDMA_REQ_STOP:
> -			dw_edma_terminate_vdesc(vd);
> +			dw_edma_terminate_all_descs(chan);
>  			chan->request = EDMA_REQ_NONE;
>  			chan->status = EDMA_ST_IDLE;
>  			break;
> @@ -616,28 +645,49 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
>  	return 0;
>  }
>
> +static void dw_edma_wait_termination(struct dma_chan *dchan)
> +{
> +	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
> +	unsigned long flags;
> +	bool configured = true;
> +
> +	/*
> +	 * dw_edma_device_terminate_all() may defer cleanup to a later interrupt
> +	 * while the channel is still running. Retry until the channel is
> +	 * deconfigured, which marks that termination completed.
> +	 */
> +	while (time_before(jiffies, timeout)) {
> +		dw_edma_device_terminate_all(dchan);
> +
> +		spin_lock_irqsave(&chan->vc.lock, flags);
> +		configured = chan->configured;
> +		spin_unlock_irqrestore(&chan->vc.lock, flags);
> +		if (!configured)
> +			return;
> +
> +		cond_resched();
> +	}
> +
> +	dev_warn(chan->dw->chip->dev,
> +		 "timeout waiting for channel termination\n");
> +}
> +
>  static void dw_edma_device_synchronize(struct dma_chan *dchan)
>  {
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
>
> +	dw_edma_wait_termination(dchan);
>  	vchan_synchronize(&chan->vc);
>  }
>
>  static void dw_edma_free_chan_resources(struct dma_chan *dchan)
>  {
> -	unsigned long timeout = jiffies + msecs_to_jiffies(5000);
> -	int ret;
> -
> -	while (time_before(jiffies, timeout)) {
> -		ret = dw_edma_device_terminate_all(dchan);
> -		if (!ret)
> -			break;
> -
> -		if (time_after_eq(jiffies, timeout))
> -			return;
> +	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
>
> -		cpu_relax();
> -	}
> +	dw_edma_wait_termination(dchan);
> +	vchan_synchronize(&chan->vc);
> +	vchan_free_chan_resources(&chan->vc);
>  }
>
>  static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
> --
> 2.51.0
>

Re: [PATCH 05/17] dmaengine: dw-edma: Serialize channel state checks

[Frank Li]

On Tue, Jun 16, 2026 at 12:40:59AM +0900, Koichiro Den wrote:
> pause() and resume() read and update channel state without holding
> vc.lock, while the interrupt handlers update the same state under it.
> Take the same lock around those state checks so that request, status,
> and configured stay consistent.
>
> For example, pause() can observe EDMA_ST_BUSY right before the interrupt
> handler completes the final descriptor and moves the channel to
> EDMA_ST_IDLE, and then record EDMA_REQ_PAUSE on an already idle channel.
> No further interrupt will acknowledge the request, and since
> issue_pending() requires EDMA_REQ_NONE, the channel is wedged for good:
> terminate_all() leaves the stale request behind, so even reconfiguring
> the channel does not recover it.
>
> issue_pending() already runs under vc.lock, but it tests configured
> before taking it. Move that test under the lock as well, so that the
> decision to start work is made against the current value rather than one
> observed before a concurrent terminate_all() deconfigured the channel.
>
> Fixes: e63d79d1ffcd ("dmaengine: Add Synopsys eDMA IP core driver")
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---

Reviewed-by: Frank Li <Frank.Li@nxp.com>

>  drivers/dma/dw-edma/dw-edma-core.c | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index 2777dc0b2aed..489f7fe49840 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -177,8 +177,10 @@ dw_edma_device_get_config(struct dma_chan *dchan,
>  static int dw_edma_device_pause(struct dma_chan *dchan)
>  {
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +	unsigned long flags;
>  	int err = 0;
>
> +	spin_lock_irqsave(&chan->vc.lock, flags);
>  	if (!chan->configured)
>  		err = -EPERM;
>  	else if (chan->status != EDMA_ST_BUSY)
> @@ -187,6 +189,7 @@ static int dw_edma_device_pause(struct dma_chan *dchan)
>  		err = -EPERM;
>  	else
>  		chan->request = EDMA_REQ_PAUSE;
> +	spin_unlock_irqrestore(&chan->vc.lock, flags);
>
>  	return err;
>  }
> @@ -194,8 +197,10 @@ static int dw_edma_device_pause(struct dma_chan *dchan)
>  static int dw_edma_device_resume(struct dma_chan *dchan)
>  {
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
> +	unsigned long flags;
>  	int err = 0;
>
> +	spin_lock_irqsave(&chan->vc.lock, flags);
>  	if (!chan->configured) {
>  		err = -EPERM;
>  	} else if (chan->status != EDMA_ST_PAUSE) {
> @@ -206,6 +211,7 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
>  		chan->status = EDMA_ST_BUSY;
>  		dw_edma_start_transfer(chan);
>  	}
> +	spin_unlock_irqrestore(&chan->vc.lock, flags);
>
>  	return err;
>  }
> @@ -249,11 +255,9 @@ static void dw_edma_device_issue_pending(struct dma_chan *dchan)
>  	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
>  	unsigned long flags;
>
> -	if (!chan->configured)
> -		return;
> -
>  	spin_lock_irqsave(&chan->vc.lock, flags);
> -	if (vchan_issue_pending(&chan->vc) && chan->request == EDMA_REQ_NONE &&
> +	if (chan->configured && vchan_issue_pending(&chan->vc) &&
> +	    chan->request == EDMA_REQ_NONE &&
>  	    chan->status == EDMA_ST_IDLE) {
>  		chan->status = EDMA_ST_BUSY;
>  		dw_edma_start_transfer(chan);
> --
> 2.51.0
>