[PATCH v8 0/7] dmaengine: dw-edma: Support dynamic LL appends

[PATCH v8 0/7] dmaengine: dw-edma: Support dynamic LL appends

[Koichiro Den]

Hi,

This series lets dw-edma append requests to a running circular linked
list (LL) using PCS-CCS-CB-TCB Producer-Consumer Synchronization. This
reduces stop/start cycles between descriptors.

v8 fixes the premature LL completion reported by Sashiko on v7 patch 6
and picks up Frank's Reviewed-by tags (dropped on patches 2 and 4 due to
updates). I reran the tests, with additional tuning on K3 and S4 to
reduce run-to-run variation. See below.

Based on dmaengine-7.4-rc1 (aff5e6a77fc6).


v8 (Re-)testing
===============

I used three setups for testing:

  Id   Endpoint                  Link       Host
  ---  ------------------------  ---------  -----------------
  (A)  RK3588 ROCK 5B            Gen3 x4    CIX Sky1 Orion O6
  (B)  SpacemiT K3 [1] Pico-ITX  Gen3 x4    CIX Sky1 Orion O6
  (C)  R-Car S4 Spider           Gen4 x2    R-Car S4 Spider

1. pci-epf-test:

   For all (A)~(C), same PASS/FAIL/SKIP set with and without this series.

2. nvmet-pci-epf:

   I ran fio on the host against a null_blk target, with runtime=30s,
   ramp_time=5s. Each setup was tested five times without the series
   (base) and five times with it (new), alternating as
   base1->new1->base2->new2->...->base5->new5. The table shows median
   throughput in MiB/s.

                           (A) RK3588       (B) K3           (C) R-Car S4
                           (5.60a eDMA)     (6.30a HDMA)     (5.20a eDMA)
   ----------------------  ---------------  ---------------  -----------------
   rand read     4K q1       36->  35  -3%    62->  52 -16%  12.1->11.7  -3%
   rand read     4K q32     180-> 162 -10%   169-> 164  -3%  48.4->70.7 +46%
   rand read     4K q32/4   188-> 160 -15%   169-> 164  -3%  66.1->76.9 +16%
   rand read   128K q1      718-> 661  -8%   942-> 950  +1%   207-> 203  -2%
   rand read   128K q32      82->1199  x15  1140->1590 +39%   696-> 917 +32%
   rand read   128K q32/4    51->1248  x24  1810->2428 +34%  1243->1328  +7%
   rand read   512K q1     1039->1034   0%  1281->1287   0%   143-> 196 +37%
   rand read   512K q32      58->1312  x23  1800->2368 +32%  1217->1294  +6%
   rand read   512K q32/4    46-> 953  x21  1816->2528 +39%  1288->1371  +6%
   rand write    4K q1       34->  26 -22%    58->  48 -17%  11.7->11.3  -3%
   rand write    4K q32      70-> 148 x2.1   117-> 220 +88%  40.6->47.1 +16%
   rand write    4K q32/4    69-> 151 x2.2   115-> 219 +90%  43.3->62.0 +43%
   rand write  128K q1      466-> 474  +2%   758-> 781  +3%   142-> 144  +1%
   rand write  128K q32     888->1310 +48%  1686->1921 +14%   557-> 696 +25%
   rand write  128K q32/4   900->1287 +43%  1686->1920 +14%   733-> 860 +17%
   seq read    128K q1      663-> 621  -6%   905-> 836  -8%   203-> 207  +2%
   seq read    128K q32      69->1184  x17  1150->1590 +38%   699-> 938 +34%
   seq read    512K q1      948->1033  +9%  1304->1276  -2%   146-> 193 +32%
   seq read    512K q32      55->1335  x24  1804->2345 +30%  1072->1167  +9%
   seq read      1M q32      49->1021  x21  1841->2484 +35%  1249->1330  +6%
   seq write   128K q1      504-> 464  -8%   810-> 833  +3%   138-> 140  +1%
   seq write   128K q32     897->1307 +46%  1689->1922 +14%   565-> 704 +25%
   seq write   512K q1      569-> 715 +26%   271-> 306 +13%   113-> 172 +52%
   seq write   512K q32     899->1294 +44%  1687->1921 +14%   737-> 861 +17%
   seq write     1M q32     843->1194 +42%  1685->1918 +14%   747-> 876 +17%
   rw read    4K-1M q8/4    193-> 529 x2.7   531-> 727 +37%   453-> 499 +10%
   rw write   4K-1M q8/4    198-> 530 x2.7   532-> 728 +37%   453-> 500 +10%
   ("/4" in "q32/4" means 4 jobs.)

   More briefly, geometric mean of the new/base ratios across all runs
   in each group (5 runs x N cases):

                      (A) RK3588        (B) K3             (C) S4
     ---------------  ----------------- ------------------ ----------
     QD32 (16 cases)  x4.46             +28.2%             +20.0%
                      (x4.06 .. x4.81)  (+27.2% .. +28.7%) (+16.9% .. +22.8%)

     QD1   (9 cases)  -0.9%             +0.2%              +11.5%
                      (-10.8% .. +7.3%) (-3.1% .. +4.6%)   (+5.1% .. +16.5%)

     rw q8 (2 cases)  x2.70             +36.8%             +10.6%
                      (x2.48 .. x3.13)  (+34.9% .. +38.2%) (+9.7% .. +12.4%)

   Parentheses show the min and max of the 5 per-round geomeans, with
   each round comprising one base/new pair.

   Note for (B) and (C):
   - To reduce run-to-run variation, both base and new kernels used
     nohz=off on the endpoint, and the DMA IRQ was pinned to CPU4.

   Note for (A) and (C):
   - commit efadda401c5c ("PCI: dwc: Set DMA_BYPASS bit in outbound
     IATU CTRL2 register"), which is not in dmaengine-7.4-rc1 yet, was
     applied to both the base and the new kernels. Without it:
     - (A) often hit NVMe I/O timeouts in the 4K q32 write cases with
       the series applied. fio still exits successfully and the
       numbers look plausible, so this is easy to miss unless you
       check dmesg.
     - (C) was unstable in general.

3. Terminate/pause test during dma load

   Following Sashiko's report (against v6):
   https://lore.kernel.org/r/rbkljnmrf4ctc7rj6qlwhvhemvzvxx6pdxjxsdc2bmicez56ms@bgshyuvsjuqh/

   On (A) and (B), I used a locally modified nvmet-pci-epf to
   periodically terminate DMA transfers or pause/resume the channels
   under fio load. Terminated transfers were retried. Baseline,
   terminate-only, pause/resume-only, combined and post-injection tests
   passed CRC32C write/readback verification against a memory-backed
   null_blk target.

[1] Based on Ubuntu-riscv-7.3.0-9.9.1 + some other out-of-tree patches,
    especially to make EP mode work on K3.


Note
====

Much of the complexity in v1-v5 came from working around channel stalls
on R-Car S4 with the out-of-tree direct-DMA NTB transport. A channel
could remain stopped and ignore further doorbells without reporting an
error. I have not reproduced this particular behavior on other SoCs with
different DWC IP revisions, or with the in-tree DMA clients I tested on
S4. I have not obtained further information to explain it, so it remains
unresolved. More details:
https://lore.kernel.org/r/tkyu5rsxfa7i6rxxw6qxmmwke5qnlxptwx4j2s24fhftsmmf26@gamef2ec4wwm/

I substantially simplified the series since v6 because I was concerned
that keeping all those workarounds would make it too difficult to review
and leave it pending indefinitely. I continue investigating the S4
stalls separately, alongside the work to upstream the more demanding DMA
clients that expose them.


Best regards,
Koichiro Den
---
v7->v8 changes:
  - Read LLP before checking for a stop to avoid premature completion
    (Sashiko).
  - Resample after a stopped result, and use the same sample for
    completion and termination.
  - Reclaim delayed completions while paused without restarting the
    channel.
  - Pick up Frank's R-b on patches 3 and 5 (dropped from patches 2/4 due
    to code updates).
  - Update test numbers after rerunning with K3/S4 jitter tuning.
  - Link to v7: https://lore.kernel.org/r/20261104161227.1742005-1-den@valinux.co.jp/

v6->v7 changes:
  - Avoid premature LL reset on stale STOPPED status during termination
    (new patch 2).
  - Factor out descriptor completion (new patch 3, Frank).
  - Handle STOP before LL reclamation to avoid scheduling callbacks
    (Sashiko).
  - Restart stopped LL channels to drain published entries during
    termination.
  - Use modulo arithmetic for the running LL boundary and add lockdep
    assertions (Frank).
  - Keep HDMA watermark interrupts masked for non-LL channels (Sashiko).
  - Use the dw_edma_ prefix for tracepoints and the event class
    (Sashiko).
  - Add Frank's Reviewed-by to patch 1.
  - Link to v6: https://lore.kernel.org/all/20261030150742.2586776-1-den@valinux.co.jp/

v5->v6 changes:
  - Drop the recovery code and the defensive doorbell handling added for
    the R-Car S4 stall: IRQ-paired LLP snapshots and event
    serialization, the delayed stopped-channel recheck, tx_status()
    reclaim, the LL interrupt placement policy, engine reset and stall
    recovery. Empirically, the stall so far reproduces only on S4 (5.20a
    (GA)) with out-of-tree direct-DMA NTB transport.
  - Read progress in the deferred handler instead. A channel counts as
    stopped when its status is STOPPED and the transfer size is zero, on
    both eDMA and native HDMA.
  - Fold the STOP/PAUSE drain into the progress handler.
  - Rebase on the circular LL groundwork in dmaengine-7.4-rc1. 14
    patches -> 5.

For earlier changes and hardware observations, see v5:
https://lore.kernel.org/r/20260812155721.2807506-1-den@valinux.co.jp/


Frank Li (2):
  dmaengine: dw-edma: Dynamically append requests while running
  dmaengine: dw-edma: Add trace support

Koichiro Den (5):
  dmaengine: dw-edma: Add a channel transfer size operation
  dmaengine: dw-edma: Avoid resetting an LL on stale stopped status
  dmaengine: dw-edma: Factor out descriptor completion
  dmaengine: dw-edma: Reclaim LL entries from hardware progress
  dmaengine: dw-edma: Use HDMA watermark interrupts as LL progress

 drivers/dma/dw-edma/Makefile          |   2 +
 drivers/dma/dw-edma/dw-edma-core.c    | 286 ++++++++++++++++++++++----
 drivers/dma/dw-edma/dw-edma-core.h    |   8 +
 drivers/dma/dw-edma/dw-edma-trace.c   |   4 +
 drivers/dma/dw-edma/dw-edma-trace.h   | 117 +++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c |   6 +
 drivers/dma/dw-edma/dw-hdma-v0-core.c |  68 +++++-
 drivers/dma/dw-edma/dw-hdma-v0-regs.h |   1 +
 8 files changed, 442 insertions(+), 50 deletions(-)
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h


base-commit: aff5e6a77fc69a7c490246465940d18422d7c631
-- 
2.54.0

[PATCH v8 1/7] dmaengine: dw-edma: Add a channel transfer size operation

[Koichiro Den]

Expose the channel transfer size through a core operation for eDMA and
HDMA. LL progress handling will use it to check for a stopped channel
with no payload left to transfer.

PCI Express DM Controller Databook 6.10a-lca06, section 7.3.4,
"Detecting the End of the HDMA Transfer", documents this polling method:

  STATUS field of the HDMA Write/Read Channel Status register
  (HDMA_STATUS_OFF_[WR|RD]CH_i) is Stopped, and the HDMA Transfer Size
  register is 0x0.

The same condition applies to eDMA as well.

No functional change on its own.

Cc: stable@vger.kernel.org
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from v7:
  - No changes.

 drivers/dma/dw-edma/dw-edma-core.h    | 6 ++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c | 6 ++++++
 drivers/dma/dw-edma/dw-hdma-v0-core.c | 6 ++++++
 3 files changed, 18 insertions(+)

diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 76fd5a41a707..84a811bdcb6d 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -163,6 +163,7 @@ struct dw_edma_core_ops {
 	int (*ch_quiesce)(struct dw_edma_chan *chan);
 	u16 (*ch_count)(struct dw_edma *dw, enum dw_edma_dir dir);
 	enum dma_status (*ch_status)(struct dw_edma_chan *chan);
+	u32 (*ch_transfer_size)(struct dw_edma_chan *chan);
 	irqreturn_t (*handle_int)(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 				  dw_edma_handler_t handler);
 	void (*non_ll_start)(struct dw_edma_chan *chan, struct dw_edma_burst *child);
@@ -262,6 +263,11 @@ enum dma_status dw_edma_core_ch_status(struct dw_edma_chan *chan)
 	return chan->dw->core->ch_status(chan);
 }
 
+static inline u32 dw_edma_core_ch_transfer_size(struct dw_edma_chan *chan)
+{
+	return chan->dw->core->ch_transfer_size(chan);
+}
+
 static inline irqreturn_t
 dw_edma_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 			dw_edma_handler_t handler)
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index c053cb4c2844..fa8be85c7ac1 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -318,6 +318,11 @@ static enum dma_status dw_edma_v0_core_ch_status(struct dw_edma_chan *chan)
 		return DMA_ERROR;
 }
 
+static u32 dw_edma_v0_core_ch_transfer_size(struct dw_edma_chan *chan)
+{
+	return GET_CH_32(chan->dw, chan->dir, chan->id, transfer_size);
+}
+
 static void dw_edma_v0_core_clear_done_int(struct dw_edma_chan *chan)
 {
 	struct dw_edma *dw = chan->dw;
@@ -665,6 +670,7 @@ static const struct dw_edma_core_ops dw_edma_v0_core = {
 	.ch_quiesce = dw_edma_v0_core_ch_quiesce,
 	.ch_count = dw_edma_v0_core_ch_count,
 	.ch_status = dw_edma_v0_core_ch_status,
+	.ch_transfer_size = dw_edma_v0_core_ch_transfer_size,
 	.handle_int = dw_edma_v0_core_handle_int,
 	.ll_data = dw_edma_v0_core_ll_data,
 	.ll_link = dw_edma_v0_core_ll_link,
diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index d42cb4ad49a2..aa430219d0af 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -156,6 +156,11 @@ static enum dma_status dw_hdma_v0_core_ch_status(struct dw_edma_chan *chan)
 		return DMA_ERROR;
 }
 
+static u32 dw_hdma_v0_core_ch_transfer_size(struct dw_edma_chan *chan)
+{
+	return GET_CH_32(chan->dw, chan->dir, chan->id, transfer_size);
+}
+
 static void dw_hdma_v0_core_clear_done_int(struct dw_edma_chan *chan)
 {
 	struct dw_edma *dw = chan->dw;
@@ -425,6 +430,7 @@ static const struct dw_edma_core_ops dw_hdma_v0_core = {
 	.ch_quiesce = dw_hdma_v0_core_ch_quiesce,
 	.ch_count = dw_hdma_v0_core_ch_count,
 	.ch_status = dw_hdma_v0_core_ch_status,
+	.ch_transfer_size = dw_hdma_v0_core_ch_transfer_size,
 	.handle_int = dw_hdma_v0_core_handle_int,
 	.non_ll_start = dw_hdma_v0_core_non_ll_start,
 	.ll_data = dw_hdma_v0_core_ll_data,
-- 
2.54.0

[PATCH v8 2/7] dmaengine: dw-edma: Avoid resetting an LL on stale stopped status

[Koichiro Den]

CS can still report STOPPED briefly after a doorbell write, before the
DMA engine starts consuming the published LL entries. Finishing
termination relying solely on that value can reset the ring while a
transfer is about to start, leaving hardware and software ring positions
out of sync and stalling subsequent transfers.

Require LLP to reach the published tail, on top of the standard criteria
(ie. CS=STOPPED and zero transfer size) before
dw_edma_finish_termination(). Non-LL mode behavior is unchanged.

Cc: stable@vger.kernel.org
Fixes: db79c4e264ef ("dmaengine: dw-edma: Make DMA link list work as a circular buffer")
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from v7:
  - Resample LLP and stop status if stopped to catch progress made after
    the first LLP read (follow-up to Sashiko's review).

@Frank: Dropped your R-b due to the change; would appreciate another
look.

 drivers/dma/dw-edma/dw-edma-core.c | 45 ++++++++++++++++++++++++++++--
 1 file changed, 43 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index b08bb31657c9..0c8b9545a5ba 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -146,6 +146,44 @@ static bool dw_edma_ll_pending(struct dw_edma_chan *chan)
 	return chan->ll_head != chan->ll_done;
 }
 
+static bool dw_edma_ll_stopped(struct dw_edma_chan *chan)
+{
+	return dw_edma_core_ch_status(chan) == DMA_COMPLETE &&
+	       !dw_edma_core_ch_transfer_size(chan);
+}
+
+static int dw_edma_ll_read_idx(struct dw_edma_chan *chan, bool *stopped)
+{
+	int idx;
+
+	/* Sample LLP first in case the engine starts between the reads. */
+	idx = dw_edma_core_ll_cur_idx(chan);
+	*stopped = dw_edma_ll_stopped(chan);
+	if (*stopped) {
+		/*
+		 * Re-read once to catch the final LLP if the channel remains
+		 * stopped. If it restarts, later interrupts will report
+		 * further progress.
+		 */
+		idx = dw_edma_core_ll_cur_idx(chan);
+		*stopped = dw_edma_ll_stopped(chan);
+	}
+
+	return idx;
+}
+
+static bool dw_edma_ll_drained(struct dw_edma_chan *chan)
+{
+	bool stopped;
+	int idx;
+
+	idx = dw_edma_ll_read_idx(chan, &stopped);
+	if (idx == chan->ll_max)
+		idx = 0;
+
+	return stopped && idx >= 0 && (u32)idx == chan->ll_head;
+}
+
 static void dw_edma_core_ll_start(struct dw_edma_desc *desc)
 {
 	struct dw_edma_chan *chan = desc->chan;
@@ -455,7 +493,9 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
 		dw_edma_finish_termination(chan);
 	} else if (chan->status == EDMA_ST_IDLE) {
 		dw_edma_finish_termination(chan);
-	} else if (dw_edma_core_ch_status(chan) == DMA_COMPLETE) {
+	} else if (chan->non_ll ?
+		   dw_edma_core_ch_status(chan) == DMA_COMPLETE :
+		   dw_edma_ll_drained(chan)) {
 		/*
 		 * The channel is in a false BUSY state, probably didn't
 		 * receive or lost an interrupt
@@ -790,7 +830,8 @@ static void dw_edma_done_interrupt_locked(struct dw_edma_chan *chan)
 		if (!vd)
 			break;
 
-		dw_edma_finish_termination(chan);
+		if (chan->non_ll || dw_edma_ll_drained(chan))
+			dw_edma_finish_termination(chan);
 		break;
 
 	default:
-- 
2.54.0

[PATCH v8 3/7] dmaengine: dw-edma: Factor out descriptor completion

[Koichiro Den]

Completing a descriptor takes three steps: record the callback result,
unlink it from the issued list and hand it to virt-dma.

Factor them into dw_edma_complete_vdesc() to avoid sprinkling
vchan_cookie_complete() calls throughout the driver.

No functional change.

Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from v7:
  - Pick up Frank's Reviewed-by.

 drivers/dma/dw-edma/dw-edma-core.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 0c8b9545a5ba..346ec8ee631d 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -107,6 +107,15 @@ static void dw_hdma_set_callback_result(struct virt_dma_desc *vd,
 	res->residue = residue;
 }
 
+/* Must be called with vc.lock held. */
+static void dw_edma_complete_vdesc(struct virt_dma_desc *vd,
+				   enum dmaengine_tx_result result)
+{
+	dw_hdma_set_callback_result(vd, result);
+	list_del(&vd->node);
+	vchan_cookie_complete(vd);
+}
+
 static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
 {
 	u32 i;
@@ -809,9 +818,7 @@ static void dw_edma_done_interrupt_locked(struct dw_edma_chan *chan)
 
 		desc = vd2dw_edma_desc(vd);
 		if (desc->start_burst >= desc->nburst) {
-			dw_hdma_set_callback_result(vd, DMA_TRANS_NOERROR);
-			list_del(&vd->node);
-			vchan_cookie_complete(vd);
+			dw_edma_complete_vdesc(vd, DMA_TRANS_NOERROR);
 			if (!chan->non_ll)
 				chan->ll_done = chan->ll_head;
 		}
@@ -859,9 +866,7 @@ static void dw_edma_abort_interrupt(struct dw_edma_chan *chan)
 	if (vd && chan->request == EDMA_REQ_STOP) {
 		dw_edma_terminate_all_descs(chan);
 	} else if (vd) {
-		dw_hdma_set_callback_result(vd, DMA_TRANS_ABORTED);
-		list_del(&vd->node);
-		vchan_cookie_complete(vd);
+		dw_edma_complete_vdesc(vd, DMA_TRANS_ABORTED);
 	}
 	if (!chan->non_ll)
 		dw_edma_core_reset_ll(chan);
-- 
2.54.0

[PATCH v8 4/7] dmaengine: dw-edma: Reclaim LL entries from hardware progress

[Koichiro Den]

With dynamic appends, a DONE interrupt no longer means that all
published LL entries have completed. Prepare for this by using LLP to
advance ll_done and complete descriptors in issue order.

Reclaim completed entries and refill the ring as space becomes
available. Restart a stopped channel when published work remains. This
also allows descriptors larger than the ring to run in batches.

Hardware can keep running while the driver handles an interrupt, and
LLP does not always track payload completion. Use the following
exclusive completion boundaries, where idx is the raw LLP entry index
and prev(idx) is the previous data entry in the ring:

  State/event          Meaning of raw LLP               Boundary
  -------------------  -------------------------------  ---------
  eDMA/HDMA stopped    Next entry to process             idx

  eDMA running         Observed to advance before        prev(idx)
                       payload completion

  HDMA watermark       Inclusive LLE recycling limit     prev(idx)

A stopped sample requires CS=STOPPED and a zero transfer size.

While running, HDMA updates LLP only at watermark events. Its documented
recycling limit translates to an exclusive boundary of next(idx).
However, tests using that boundary for dmaengine completion reported
IOMMU faults on HDMA 6.30a SpacemiT K3. Use prev(idx) for both engines
while running to keep the handling simple and conservative. Entries held
back by this rule are eventually reclaimed on later progress or a
confirmed stop, where idx gives the exact boundary.

Handle termination and pause as follows:
- On termination, stop publishing new entries and drain those already
  published, restarting the channel if needed. Terminate descriptors
  without callbacks, as in the non-LL path.
- On pause, stop publishing new entries and enter the paused state once
  the channel stops. Keep unfinished work for resume. Reclaim delayed
  completions without restarting the channel.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from v7:
  - Read LLP before checking for a stop, and use the same sample for
    completion and termination (Sashiko).
  - Reclaim delayed completions while paused without restarting the
    channel (spotted during code inspection, not observed in hardware
    testing).

@Frank: Dropped your R-b due to the changes; would appreciate another
look.

 drivers/dma/dw-edma/dw-edma-core.c | 160 +++++++++++++++++++++++++----
 1 file changed, 140 insertions(+), 20 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 346ec8ee631d..1d13a25663ef 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -53,13 +53,6 @@ dw_edma_alloc_desc(struct dw_edma_chan *chan, size_t nburst)
 {
 	struct dw_edma_desc *desc;
 
-	/*
-	 * For now, a descriptor that does not fit would stall the channel
-	 * forever: reject it up front.
-	 */
-	if (!chan->non_ll && nburst > chan->ll_max - 1)
-		return NULL;
-
 	desc = kzalloc_flex(*desc, burst, nburst, GFP_NOWAIT);
 	if (unlikely(!desc))
 		return NULL;
@@ -181,18 +174,83 @@ static int dw_edma_ll_read_idx(struct dw_edma_chan *chan, bool *stopped)
 	return idx;
 }
 
+/* Return the first entry we cannot reclaim yet. */
+static int dw_edma_ll_boundary(struct dw_edma_chan *chan, int idx,
+			       bool stopped)
+{
+	if (idx < 0 || (u32)idx > chan->ll_max)
+		return -EINVAL;
+
+	/* After a stop, LLP points past the completed entries. */
+	if (stopped)
+		return idx == chan->ll_max ? 0 : idx;
+
+	/*
+	 * DMA_LLP can advance before payload completion. Keep the boundary
+	 * one entry behind it, wrapping at the start of the ring.
+	 */
+	return (idx + chan->ll_max - 1) % chan->ll_max;
+}
+
 static bool dw_edma_ll_drained(struct dw_edma_chan *chan)
 {
 	bool stopped;
 	int idx;
 
 	idx = dw_edma_ll_read_idx(chan, &stopped);
-	if (idx == chan->ll_max)
-		idx = 0;
+	idx = dw_edma_ll_boundary(chan, idx, true);
 
 	return stopped && idx >= 0 && (u32)idx == chan->ll_head;
 }
 
+static void dw_edma_ll_consume(struct dw_edma_chan *chan, int idx,
+			       bool stopped)
+{
+	struct virt_dma_desc *vd, *_vd;
+	u32 done, used;
+
+	lockdep_assert_held(&chan->vc.lock);
+
+	idx = dw_edma_ll_boundary(chan, idx, stopped);
+	if (idx < 0)
+		return;
+
+	done = dw_edma_core_get_ll_dist(chan, chan->ll_done, idx);
+	used = dw_edma_core_get_used_num(chan);
+
+	if (!done || done > used)
+		return;
+
+	/*
+	 * LLP may still point to the first pending entry. Subtracting one
+	 * can wrap the index and falsely mark all pending entries complete.
+	 * Reclaim them all only after the channel has stopped.
+	 */
+	if (!stopped && done == used)
+		return;
+
+	chan->ll_done = idx;
+
+	/* Match reclaimed entries to descriptors in issue order. */
+	list_for_each_entry_safe(vd, _vd, &chan->vc.desc_issued, node) {
+		struct dw_edma_desc *desc = vd2dw_edma_desc(vd);
+		u32 consumed = min_t(u32, done,
+				     desc->start_burst - desc->done_burst);
+
+		if (!consumed)
+			break;
+
+		desc->done_burst += consumed;
+		done -= consumed;
+		if (desc->done_burst != desc->nburst)
+			break;
+
+		dw_edma_complete_vdesc(vd, DMA_TRANS_NOERROR);
+	}
+
+	WARN_ON_ONCE(done);
+}
+
 static void dw_edma_core_ll_start(struct dw_edma_desc *desc)
 {
 	struct dw_edma_chan *chan = desc->chan;
@@ -221,7 +279,6 @@ static void dw_edma_core_ll_start(struct dw_edma_desc *desc)
 		}
 	}
 
-	desc->done_burst = desc->start_burst;
 	desc->start_burst = i;
 }
 
@@ -330,6 +387,61 @@ static void dw_edma_core_ch_maybe_doorbell(struct dw_edma_chan *chan)
 	dw_edma_core_ch_doorbell(chan);
 }
 
+/* Must be called with vc.lock held. */
+static void dw_edma_ll_advance(struct dw_edma_chan *chan, bool event)
+{
+	bool stopped;
+	int idx;
+
+	lockdep_assert_held(&chan->vc.lock);
+
+	idx = dw_edma_ll_read_idx(chan, &stopped);
+
+	/* Terminate without scheduling completion callbacks. */
+	if (chan->request == EDMA_REQ_STOP) {
+		if (!stopped)
+			return;
+
+		idx = dw_edma_ll_boundary(chan, idx, true);
+		if (idx >= 0 && (u32)idx == chan->ll_head)
+			dw_edma_finish_termination(chan);
+		else
+			/* A newly appended tail may have missed the last stop. */
+			dw_edma_core_ch_doorbell(chan);
+		return;
+	}
+
+	/*
+	 * HDMA updates LLP at watermark events while running. Use running
+	 * samples only when processing an LL interrupt.
+	 */
+	if (stopped || event)
+		dw_edma_ll_consume(chan, idx, stopped);
+
+	/* An interrupt may arrive after we have entered PAUSE. */
+	if (chan->status == EDMA_ST_PAUSE)
+		return;
+
+	if (chan->request == EDMA_REQ_PAUSE) {
+		if (!stopped)
+			return;
+
+		dw_edma_set_request(chan, EDMA_REQ_NONE);
+		chan->status = EDMA_ST_PAUSE;
+		return;
+	}
+
+	dw_edma_start_transfer(chan);
+	if (!dw_edma_ll_pending(chan)) {
+		chan->status = EDMA_ST_IDLE;
+		return;
+	}
+
+	chan->status = EDMA_ST_BUSY;
+	if (stopped)
+		dw_edma_core_ch_doorbell(chan);
+}
+
 static void dw_edma_device_caps(struct dma_chan *dchan,
 				struct dma_slave_caps *caps)
 {
@@ -514,6 +626,8 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
 		err = -EPERM;
 	} else {
 		dw_edma_set_request(chan, EDMA_REQ_STOP);
+		if (!chan->non_ll)
+			dw_edma_ll_advance(chan, false);
 	}
 	if (chan->status == EDMA_ST_IDLE)
 		dw_edma_set_request(chan, EDMA_REQ_NONE);
@@ -798,7 +912,7 @@ dw_edma_device_prep_interleaved_dma(struct dma_chan *dchan,
 	return dw_edma_device_transfer(&xfer, dw_edma_device_get_config(dchan, NULL));
 }
 
-/* Must be called with vc.lock held. */
+/* Non-LL completion; vc.lock must be held. */
 static void dw_edma_done_interrupt_locked(struct dw_edma_chan *chan)
 {
 	struct dw_edma_desc *desc;
@@ -817,11 +931,8 @@ static void dw_edma_done_interrupt_locked(struct dw_edma_chan *chan)
 			break;
 
 		desc = vd2dw_edma_desc(vd);
-		if (desc->start_burst >= desc->nburst) {
+		if (desc->start_burst >= desc->nburst)
 			dw_edma_complete_vdesc(vd, DMA_TRANS_NOERROR);
-			if (!chan->non_ll)
-				chan->ll_done = chan->ll_head;
-		}
 
 		if (chan->request == EDMA_REQ_PAUSE) {
 			dw_edma_set_request(chan, EDMA_REQ_NONE);
@@ -837,14 +948,12 @@ static void dw_edma_done_interrupt_locked(struct dw_edma_chan *chan)
 		if (!vd)
 			break;
 
-		if (chan->non_ll || dw_edma_ll_drained(chan))
-			dw_edma_finish_termination(chan);
+		dw_edma_finish_termination(chan);
 		break;
 
 	default:
 		break;
 	}
-	dw_edma_core_ch_maybe_doorbell(chan);
 }
 
 static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
@@ -856,6 +965,13 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
 
+static void dw_edma_ll_interrupt(struct dw_edma_chan *chan)
+{
+	guard(spinlock_irqsave)(&chan->vc.lock);
+
+	dw_edma_ll_advance(chan, true);
+}
+
 static void dw_edma_abort_interrupt(struct dw_edma_chan *chan)
 {
 	struct virt_dma_desc *vd;
@@ -884,8 +1000,12 @@ static void dw_edma_irq_work(struct work_struct *work)
 	do {
 		events = atomic_xchg(&chan->irq_pending, 0);
 
-		if (events & DW_EDMA_DEFERRED_DONE)
-			dw_edma_done_interrupt(chan);
+		if (events & DW_EDMA_DEFERRED_DONE) {
+			if (chan->non_ll)
+				dw_edma_done_interrupt(chan);
+			else
+				dw_edma_ll_interrupt(chan);
+		}
 		if (events & DW_EDMA_DEFERRED_ABORT)
 			dw_edma_abort_interrupt(chan);
 	} while (atomic_read(&chan->irq_pending));
-- 
2.54.0

[PATCH v8 5/7] dmaengine: dw-edma: Use HDMA watermark interrupts as LL progress

[Koichiro Den]

While HDMA is running, DMA_LLP is updated only at watermark events.
Enable watermark interrupts to track progress without waiting for the
channel to stop.

Set the watermark bits on entries that request an interrupt, configure
local or remote delivery, and pass events to the LL progress handler.
Unmask local watermark interrupts only for LL channels in local IRQ mode.

Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from v7:
  - Pick up Frank's Reviewed-by.

 drivers/dma/dw-edma/dw-hdma-v0-core.c | 62 ++++++++++++++++++++++-----
 drivers/dma/dw-edma/dw-hdma-v0-regs.h |  1 +
 2 files changed, 52 insertions(+), 11 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-hdma-v0-core.c b/drivers/dma/dw-edma/dw-hdma-v0-core.c
index aa430219d0af..0cccd0003a1b 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-hdma-v0-core.c
@@ -58,7 +58,8 @@ static u32 dw_hdma_v0_core_int_setup(struct dw_edma_chan *chan, u32 val)
 {
 	val &= ~(HDMA_V0_LOCAL_ABORT_INT_EN | HDMA_V0_REMOTE_ABORT_INT_EN |
 		 HDMA_V0_LOCAL_STOP_INT_EN | HDMA_V0_REMOTE_STOP_INT_EN |
-		 HDMA_V0_ABORT_INT_MASK | HDMA_V0_STOP_INT_MASK);
+		 HDMA_V0_ABORT_INT_MASK | HDMA_V0_WATERMARK_INT_MASK |
+		 HDMA_V0_STOP_INT_MASK);
 
 	/*
 	 * HDMA_INT_STATUS.STOP and .ABORT are latched only when LSIE and
@@ -68,6 +69,9 @@ static u32 dw_hdma_v0_core_int_setup(struct dw_edma_chan *chan, u32 val)
 	 */
 	val |= HDMA_V0_LOCAL_ABORT_INT_EN | HDMA_V0_LOCAL_STOP_INT_EN;
 
+	if (chan->non_ll || chan->irq_mode == DW_EDMA_CH_IRQ_REMOTE)
+		val |= HDMA_V0_WATERMARK_INT_MASK;
+
 	if (chan->irq_mode == DW_EDMA_CH_IRQ_REMOTE)
 		val |= HDMA_V0_REMOTE_ABORT_INT_EN |
 		       HDMA_V0_REMOTE_STOP_INT_EN |
@@ -81,10 +85,12 @@ static void dw_hdma_v0_core_ch_off(struct dw_edma *dw, enum dw_edma_dir dir,
 				   u16 id)
 {
 	SET_CH_32(dw, dir, id, int_setup,
-		  HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+		  HDMA_V0_STOP_INT_MASK | HDMA_V0_WATERMARK_INT_MASK |
+		  HDMA_V0_ABORT_INT_MASK);
 	SET_CH_32(dw, dir, id, ch_en, 0);
 	SET_CH_32(dw, dir, id, int_clear,
-		  HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+		  HDMA_V0_STOP_INT_MASK | HDMA_V0_WATERMARK_INT_MASK |
+		  HDMA_V0_ABORT_INT_MASK);
 }
 
 static void dw_hdma_v0_core_off(struct dw_edma *dw)
@@ -95,18 +101,26 @@ static void dw_hdma_v0_core_off(struct dw_edma *dw)
 	dir = EDMA_DIR_WRITE;
 	for (id = 0; id < dw->wr_ch_cnt; id++) {
 		SET_CH_32(dw, dir, id, int_setup,
-			  HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+			  HDMA_V0_STOP_INT_MASK |
+			  HDMA_V0_WATERMARK_INT_MASK |
+			  HDMA_V0_ABORT_INT_MASK);
 		SET_CH_32(dw, dir, id, int_clear,
-			  HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+			  HDMA_V0_STOP_INT_MASK |
+			  HDMA_V0_WATERMARK_INT_MASK |
+			  HDMA_V0_ABORT_INT_MASK);
 		SET_CH_32(dw, dir, id, ch_en, 0);
 	}
 
 	dir = EDMA_DIR_READ;
 	for (id = 0; id < dw->rd_ch_cnt; id++) {
 		SET_CH_32(dw, dir, id, int_setup,
-			  HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+			  HDMA_V0_STOP_INT_MASK |
+			  HDMA_V0_WATERMARK_INT_MASK |
+			  HDMA_V0_ABORT_INT_MASK);
 		SET_CH_32(dw, dir, id, int_clear,
-			  HDMA_V0_STOP_INT_MASK | HDMA_V0_ABORT_INT_MASK);
+			  HDMA_V0_STOP_INT_MASK |
+			  HDMA_V0_WATERMARK_INT_MASK |
+			  HDMA_V0_ABORT_INT_MASK);
 		SET_CH_32(dw, dir, id, ch_en, 0);
 	}
 }
@@ -168,6 +182,14 @@ static void dw_hdma_v0_core_clear_done_int(struct dw_edma_chan *chan)
 	SET_CH_32(dw, chan->dir, chan->id, int_clear, HDMA_V0_STOP_INT_MASK);
 }
 
+static void dw_hdma_v0_core_clear_watermark_int(struct dw_edma_chan *chan)
+{
+	struct dw_edma *dw = chan->dw;
+
+	SET_CH_32(dw, chan->dir, chan->id, int_clear,
+		  HDMA_V0_WATERMARK_INT_MASK);
+}
+
 static void dw_hdma_v0_core_clear_abort_int(struct dw_edma_chan *chan)
 {
 	struct dw_edma *dw = chan->dw;
@@ -208,17 +230,23 @@ dw_hdma_v0_core_handle_int(struct dw_edma_irq *dw_irq, enum dw_edma_dir dir,
 			continue;
 
 		val = dw_hdma_v0_core_status_int(chan);
-		if (FIELD_GET(HDMA_V0_STOP_INT_MASK, val)) {
+
+		if (FIELD_GET(HDMA_V0_STOP_INT_MASK, val))
 			dw_hdma_v0_core_clear_done_int(chan);
-			handler(chan, DW_EDMA_IRQ_STOP);
+		if (FIELD_GET(HDMA_V0_WATERMARK_INT_MASK, val))
+			dw_hdma_v0_core_clear_watermark_int(chan);
 
+		if (FIELD_GET(HDMA_V0_STOP_INT_MASK, val)) {
+			handler(chan, DW_EDMA_IRQ_STOP);
+			ret = IRQ_HANDLED;
+		} else if (FIELD_GET(HDMA_V0_WATERMARK_INT_MASK, val)) {
+			handler(chan, DW_EDMA_IRQ_PROGRESS);
 			ret = IRQ_HANDLED;
 		}
 
 		if (FIELD_GET(HDMA_V0_ABORT_INT_MASK, val)) {
 			dw_hdma_v0_core_clear_abort_int(chan);
 			handler(chan, DW_EDMA_IRQ_ABORT);
-
 			ret = IRQ_HANDLED;
 		}
 	}
@@ -275,7 +303,7 @@ static void dw_hdma_v0_core_ch_enable(struct dw_edma_chan *chan)
 
 	/* Enable engine */
 	SET_CH_32(dw, chan->dir, chan->id, ch_en, BIT(0));
-	/* Interrupt unmask - stop, abort */
+	/* Interrupt unmask - stop, watermark, abort */
 	tmp = GET_CH_32(dw, chan->dir, chan->id, int_setup);
 	tmp = dw_hdma_v0_core_int_setup(chan, tmp);
 	SET_CH_32(dw, chan->dir, chan->id, int_setup, tmp);
@@ -336,6 +364,11 @@ static void dw_hdma_v0_core_ch_config(struct dw_edma_chan *chan)
 	/* MSI done addr - low, high */
 	SET_CH_32(dw, chan->dir, chan->id, msi_stop.lsb, chan->msi.address_lo);
 	SET_CH_32(dw, chan->dir, chan->id, msi_stop.msb, chan->msi.address_hi);
+	/* MSI watermark addr - low, high */
+	SET_CH_32(dw, chan->dir, chan->id, msi_watermark.lsb,
+		  chan->msi.address_lo);
+	SET_CH_32(dw, chan->dir, chan->id, msi_watermark.msb,
+		  chan->msi.address_hi);
 	/* MSI abort addr - low, high */
 	SET_CH_32(dw, chan->dir, chan->id, msi_abort.lsb, chan->msi.address_lo);
 	SET_CH_32(dw, chan->dir, chan->id, msi_abort.msb, chan->msi.address_hi);
@@ -355,6 +388,13 @@ dw_hdma_v0_core_ll_data(struct dw_edma_chan *chan, struct dw_edma_burst *burst,
 	if (cb)
 		control |= DW_HDMA_V0_CB;
 
+	if (irq) {
+		control |= DW_HDMA_V0_LWIE;
+
+		if (chan->irq_mode == DW_EDMA_CH_IRQ_REMOTE)
+			control |= DW_HDMA_V0_RWIE;
+	}
+
 	dw_hdma_v0_write_ll_data(chan, idx, control, burst->sz, burst->sar,
 				 burst->dar);
 }
diff --git a/drivers/dma/dw-edma/dw-hdma-v0-regs.h b/drivers/dma/dw-edma/dw-hdma-v0-regs.h
index b88226cdd791..e240ffb3dea7 100644
--- a/drivers/dma/dw-edma/dw-hdma-v0-regs.h
+++ b/drivers/dma/dw-edma/dw-hdma-v0-regs.h
@@ -18,6 +18,7 @@
 #define HDMA_V0_LOCAL_STOP_INT_EN		BIT(4)
 #define HDMA_V0_REMOTE_STOP_INT_EN		BIT(3)
 #define HDMA_V0_ABORT_INT_MASK			BIT(2)
+#define HDMA_V0_WATERMARK_INT_MASK		BIT(1)
 #define HDMA_V0_STOP_INT_MASK			BIT(0)
 #define HDMA_V0_LINKLIST_EN			BIT(0)
 #define HDMA_V0_CONSUMER_CYCLE_STAT		BIT(1)
-- 
2.54.0

[PATCH v8 6/7] dmaengine: dw-edma: Dynamically append requests while running

[Koichiro Den]

From: Frank Li <Frank.Li@nxp.com>

Allow issue_pending() to fill free LL entries while the channel is busy,
then check progress and restart it if it has stopped. Publish requests
in descriptor order.

Multiple descriptors can now have entries in the LL ring. When resetting
the ring after an abort, mark the other descriptors with pending entries
as aborted too, so they cannot be replayed.

Since dw_edma_start_transfer() can now return zero with LL entries still
pending, keep such channels BUSY on resume.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
Co-developed-by: Koichiro Den <den@valinux.co.jp>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from v7:
  - No changes. The premature completion Sashiko reported on v7 is fixed in
    patches 2 and 4.
    https://lore.kernel.org/r/rtfjvblptmdez2znjgx7t4v5fumj6wabp2ft6zmnsm353nsves@vow63wcrnpol/

 drivers/dma/dw-edma/dw-edma-core.c | 65 ++++++++++++++++++++++--------
 1 file changed, 49 insertions(+), 16 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 1d13a25663ef..bcf4fca34591 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -111,8 +111,22 @@ static void dw_edma_complete_vdesc(struct virt_dma_desc *vd,
 
 static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
 {
+	struct virt_dma_desc *vd, *tmp;
 	u32 i;
 
+	/*
+	 * We cannot tell which pending entries completed before the reset.
+	 * Abort their descriptors to avoid repeating a transfer.
+	 */
+	list_for_each_entry_safe(vd, tmp, &chan->vc.desc_issued, node) {
+		struct dw_edma_desc *desc = vd2dw_edma_desc(vd);
+
+		if (desc->start_burst == desc->done_burst)
+			continue;
+
+		dw_edma_complete_vdesc(vd, DMA_TRANS_ABORTED);
+	}
+
 	chan->ll_head = 0;
 	chan->ll_done = 0;
 	/* Drop stale CB bits before reusing the circular LL ring. */
@@ -301,21 +315,34 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 {
 	struct dw_edma_desc *desc;
 	struct virt_dma_desc *vd;
+	int ret = 0;
 
-	vd = vchan_next_desc(&chan->vc);
-	if (!vd)
-		return 0;
+	if (chan->non_ll) {
+		vd = vchan_next_desc(&chan->vc);
+		if (!vd)
+			return 0;
 
-	desc = vd2dw_edma_desc(vd);
-	if (!desc)
-		return 0;
+		dw_edma_core_start(vd2dw_edma_desc(vd));
+		return 1;
+	}
 
-	if (!chan->non_ll && !chan->ll_valid)
-		dw_edma_core_reset_ll(chan);
+	list_for_each_entry(vd, &chan->vc.desc_issued, node) {
+		if (!dw_edma_core_get_free_num(chan))
+			break;
+
+		desc = vd2dw_edma_desc(vd);
+		/* Fully published descriptors may still be in flight. */
+		if (desc->start_burst == desc->nburst)
+			continue;
 
-	dw_edma_core_start(desc);
+		if (!chan->ll_valid)
+			dw_edma_core_reset_ll(chan);
 
-	return 1;
+		dw_edma_core_start(desc);
+		ret = 1;
+	}
+
+	return ret;
 }
 
 static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
@@ -593,7 +620,8 @@ static int dw_edma_device_resume(struct dma_chan *dchan)
 		err = -EPERM;
 	} else {
 		chan->status = EDMA_ST_BUSY;
-		if (!dw_edma_start_transfer(chan))
+		if (!dw_edma_start_transfer(chan) &&
+		    (chan->non_ll || !dw_edma_ll_pending(chan)))
 			chan->status = EDMA_ST_IDLE;
 		dw_edma_core_ch_maybe_doorbell(chan);
 	}
@@ -642,11 +670,16 @@ static void dw_edma_device_issue_pending(struct dma_chan *dchan)
 
 	spin_lock_irqsave(&chan->vc.lock, flags);
 	if (chan->configured && vchan_issue_pending(&chan->vc) &&
-	    chan->request == EDMA_REQ_NONE &&
-	    chan->status == EDMA_ST_IDLE) {
-		chan->status = EDMA_ST_BUSY;
-		dw_edma_start_transfer(chan);
-		dw_edma_core_ch_maybe_doorbell(chan);
+	    chan->request == EDMA_REQ_NONE) {
+		if (chan->status == EDMA_ST_IDLE) {
+			chan->status = EDMA_ST_BUSY;
+			dw_edma_start_transfer(chan);
+			dw_edma_core_ch_maybe_doorbell(chan);
+		} else if (!chan->non_ll &&
+			   chan->status == EDMA_ST_BUSY) {
+			dw_edma_start_transfer(chan);
+			dw_edma_ll_advance(chan, false);
+		}
 	}
 	spin_unlock_irqrestore(&chan->vc.lock, flags);
 }
-- 
2.54.0

[PATCH v8 7/7] dmaengine: dw-edma: Add trace support

[Koichiro Den]

From: Frank Li <Frank.Li@nxp.com>

Add tracepoints for LL publication, descriptor append and completion,
and doorbells. Record channel and ring indices to help diagnose stalls.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
Co-developed-by: Koichiro Den <den@valinux.co.jp>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes from v7:
  - No changes.

 drivers/dma/dw-edma/Makefile        |   2 +
 drivers/dma/dw-edma/dw-edma-core.c  |   9 +++
 drivers/dma/dw-edma/dw-edma-core.h  |   2 +
 drivers/dma/dw-edma/dw-edma-trace.c |   4 +
 drivers/dma/dw-edma/dw-edma-trace.h | 117 ++++++++++++++++++++++++++++
 5 files changed, 134 insertions(+)
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h

diff --git a/drivers/dma/dw-edma/Makefile b/drivers/dma/dw-edma/Makefile
index 83ab58f87760..5f395fa1ede4 100644
--- a/drivers/dma/dw-edma/Makefile
+++ b/drivers/dma/dw-edma/Makefile
@@ -1,8 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 
+CFLAGS_dw-edma-trace.o		:= -I$(src)
 obj-$(CONFIG_DW_EDMA)		+= dw-edma.o
 dw-edma-$(CONFIG_DEBUG_FS)	:= dw-edma-v0-debugfs.o	\
 				   dw-hdma-v0-debugfs.o
+dw-edma-$(CONFIG_TRACING)	+= dw-edma-trace.o
 dw-edma-objs			:= dw-edma-core.o	\
 				   dw-edma-v0-core.o	\
 				   dw-hdma-v0-core.o $(dw-edma-y)
diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index bcf4fca34591..94cdf7fb6370 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -259,6 +259,7 @@ static void dw_edma_ll_consume(struct dw_edma_chan *chan, int idx,
 		if (desc->done_burst != desc->nburst)
 			break;
 
+		trace_dw_edma_complete_desc(desc);
 		dw_edma_complete_vdesc(vd, DMA_TRANS_NOERROR);
 	}
 
@@ -285,6 +286,12 @@ static void dw_edma_core_ll_start(struct dw_edma_desc *desc)
 				     chan->ll_head, chan->cb,
 				     i == desc->nburst - 1 || free == 1);
 
+		trace_dw_edma_fill_ll(chan, chan->ll_head,
+				      desc->vd.tx.cookie,
+				      desc->burst[i].sar,
+				      desc->burst[i].dar, desc->burst[i].sz,
+				      chan->cb);
+
 		chan->ll_head++;
 
 		if (chan->ll_head == chan->ll_max) {
@@ -338,6 +345,7 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
 		if (!chan->ll_valid)
 			dw_edma_core_reset_ll(chan);
 
+		trace_dw_edma_append_desc(desc);
 		dw_edma_core_start(desc);
 		ret = 1;
 	}
@@ -401,6 +409,7 @@ static void dw_edma_core_ll_sync(struct dw_edma_chan *chan)
 static void dw_edma_core_ch_doorbell(struct dw_edma_chan *chan)
 {
 	dw_edma_core_ll_sync(chan);
+	trace_dw_edma_doorbell(chan);
 	dw_edma_core_do_ch_doorbell(chan);
 }
 
diff --git a/drivers/dma/dw-edma/dw-edma-core.h b/drivers/dma/dw-edma/dw-edma-core.h
index 84a811bdcb6d..769c0c08a711 100644
--- a/drivers/dma/dw-edma/dw-edma-core.h
+++ b/drivers/dma/dw-edma/dw-edma-core.h
@@ -154,6 +154,8 @@ struct dw_edma {
 	const struct dw_edma_core_ops	*core;
 };
 
+#include "dw-edma-trace.h"
+
 typedef void (*dw_edma_handler_t)(struct dw_edma_chan *chan,
 				  unsigned int events);
 
diff --git a/drivers/dma/dw-edma/dw-edma-trace.c b/drivers/dma/dw-edma/dw-edma-trace.c
new file mode 100644
index 000000000000..2620ad61a943
--- /dev/null
+++ b/drivers/dma/dw-edma/dw-edma-trace.c
@@ -0,0 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define CREATE_TRACE_POINTS
+#include "dw-edma-core.h"
diff --git a/drivers/dma/dw-edma/dw-edma-trace.h b/drivers/dma/dw-edma/dw-edma-trace.h
new file mode 100644
index 000000000000..0b199ab30b9a
--- /dev/null
+++ b/drivers/dma/dw-edma/dw-edma-trace.h
@@ -0,0 +1,117 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright 2023 NXP.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM dw_edma
+
+#if !defined(__LINUX_DW_EDMA_TRACE) || defined(TRACE_HEADER_MULTI_READ)
+#define __LINUX_DW_EDMA_TRACE
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+
+DECLARE_EVENT_CLASS(dw_edma_desc_info,
+	TP_PROTO(struct dw_edma_desc *desc),
+	TP_ARGS(desc),
+	TP_STRUCT__entry(
+		__field(size_t, nburst)
+		__field(size_t, start_burst)
+		__field(dma_cookie_t, cookie)
+		__field(u32, id)
+		__field(u8, dir)
+	),
+	TP_fast_assign(
+		__entry->nburst = desc->nburst;
+		__entry->start_burst = desc->start_burst;
+		__entry->id = desc->chan->id;
+		__entry->dir = desc->chan->dir;
+		__entry->cookie = desc->vd.tx.cookie;
+	),
+	TP_printk("chan %u%c desc %d nburst %zu start_burst %zu",
+		__entry->id,
+		__entry->dir ? 'R' : 'W',
+		__entry->cookie,
+		__entry->nburst,
+		__entry->start_burst)
+);
+
+DEFINE_EVENT(dw_edma_desc_info, dw_edma_append_desc,
+	TP_PROTO(struct dw_edma_desc *desc),
+	TP_ARGS(desc)
+);
+
+DEFINE_EVENT(dw_edma_desc_info, dw_edma_complete_desc,
+	TP_PROTO(struct dw_edma_desc *desc),
+	TP_ARGS(desc)
+);
+
+TRACE_EVENT(dw_edma_doorbell,
+	TP_PROTO(struct dw_edma_chan *chan),
+	TP_ARGS(chan),
+	TP_STRUCT__entry(
+		__field(u32, id)
+		__field(u8, dir)
+		__field(u32, done)
+		__field(u32, head)
+	),
+	TP_fast_assign(
+		__entry->id = chan->id;
+		__entry->dir = chan->dir;
+		__entry->done = chan->ll_done;
+		__entry->head = chan->ll_head;
+	),
+	TP_printk("chan %u%c done %u head %u",
+		__entry->id,
+		__entry->dir ? 'R' : 'W',
+		__entry->done,
+		__entry->head)
+);
+
+TRACE_EVENT(dw_edma_fill_ll,
+	TP_PROTO(struct dw_edma_chan *chan, u32 idx, dma_cookie_t cookie, u64 src,
+		 u64 dest, u32 sz, bool flag),
+	TP_ARGS(chan, idx, cookie, src, dest, sz, flag),
+	TP_STRUCT__entry(
+		__field(u32, idx)
+		__field(u64, src)
+		__field(u64, dest)
+		__field(u32, sz)
+		__field(u32, id)
+		__field(dma_cookie_t, cookie)
+		__field(bool, flag)
+		__field(u8, dir)
+	),
+	TP_fast_assign(
+		__entry->idx = idx;
+		__entry->src = src;
+		__entry->dest = dest;
+		__entry->sz = sz;
+		__entry->id = chan->id;
+		__entry->dir = chan->dir;
+		__entry->cookie = cookie;
+		__entry->flag = flag;
+	),
+	TP_printk("chan %u%c %d [%u] %c src: %08llx dest: %08llx sz: %04x",
+		__entry->id,
+		__entry->dir ? 'R' : 'W',
+		__entry->cookie,
+		__entry->idx,
+		__entry->flag ? 'C' : 'c',
+		__entry->src,
+		__entry->dest,
+		__entry->sz)
+);
+
+#endif
+
+/* This part must be outside the header guard. */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE dw-edma-trace
+
+#include <trace/define_trace.h>
-- 
2.54.0

Re: [PATCH v8 3/7] dmaengine: dw-edma: Factor out descriptor completion

[sashiko-bot]

> Completing a descriptor takes three steps: record the callback result,
> unlink it from the issued list and hand it to virt-dma.
> 
> Factor them into dw_edma_complete_vdesc() to avoid sprinkling
> vchan_cookie_complete() calls throughout the driver.
> 
> No functional change.
> 
> Suggested-by: Frank Li <Frank.Li@nxp.com>
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=3

Re: [PATCH v8 1/7] dmaengine: dw-edma: Add a channel transfer size operation

[sashiko-bot]

> Expose the channel transfer size through a core operation for eDMA and
> HDMA. LL progress handling will use it to check for a stopped channel
> with no payload left to transfer.
> 
> PCI Express DM Controller Databook 6.10a-lca06, section 7.3.4,
> "Detecting the End of the HDMA Transfer", documents this polling method:
> 
>   STATUS field of the HDMA Write/Read Channel Status register
>   (HDMA_STATUS_OFF_[WR|RD]CH_i) is Stopped, and the HDMA Transfer Size
>   register is 0x0.
> 
> The same condition applies to eDMA as well.
> 
> No functional change on its own.
> 
> Cc: stable@vger.kernel.org
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=1

Re: [PATCH v8 7/7] dmaengine: dw-edma: Add trace support

[sashiko-bot]

> From: Frank Li <Frank.Li@nxp.com>
> 
> Add tracepoints for LL publication, descriptor append and completion,
> and doorbells. Record channel and ring indices to help diagnose stalls.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> Co-developed-by: Koichiro Den <den@valinux.co.jp>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=7

Re: [PATCH v8 5/7] dmaengine: dw-edma: Use HDMA watermark interrupts as LL progress

[sashiko-bot]

> While HDMA is running, DMA_LLP is updated only at watermark events.
> Enable watermark interrupts to track progress without waiting for the
> channel to stop.
> 
> Set the watermark bits on entries that request an interrupt, configure
> local or remote delivery, and pass events to the LL progress handler.
> Unmask local watermark interrupts only for LL channels in local IRQ mode.
> 
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=5

Re: [PATCH v8 2/7] dmaengine: dw-edma: Avoid resetting an LL on stale stopped status

[sashiko-bot]

> CS can still report STOPPED briefly after a doorbell write, before the
> DMA engine starts consuming the published LL entries. Finishing
> termination relying solely on that value can reset the ring while a
> transfer is about to start, leaving hardware and software ring positions
> out of sync and stalling subsequent transfers.
> 
> Require LLP to reach the published tail, on top of the standard criteria
> (ie. CS=STOPPED and zero transfer size) before
> dw_edma_finish_termination(). Non-LL mode behavior is unchanged.
> 
> Cc: stable@vger.kernel.org
> Fixes: db79c4e264ef ("dmaengine: dw-edma: Make DMA link list work as a circular buffer")
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=2

Re: [PATCH v8 6/7] dmaengine: dw-edma: Dynamically append requests while running

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] dmaengine: dw-edma: stranded or aborted queued descriptors on
  LL reset
--

commit a60bb88e6010e30a59b85799859be8d9e37e8b49
Author: Frank Li <Frank.Li@nxp.com>

dmaengine: dw-edma: Dynamically append requests while running

This commit allows dw_edma_device_issue_pending() to append new linked-list
entries while a channel is running and aborts in-flight descriptors when
resetting the ring.

> diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> index 1d13a25663eff..bcf4fca345914 100644
> --- a/drivers/dma/dw-edma/dw-edma-core.c
> +++ b/drivers/dma/dw-edma/dw-edma-core.c
> @@ -111,8 +111,22 @@ static void dw_edma_complete_vdesc(struct virt_dma_desc *vd,
>  
>  static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
>  {
> +	struct virt_dma_desc *vd, *tmp;
>  	u32 i;
>  
> +	/*
> +	 * We cannot tell which pending entries completed before the reset.
> +	 * Abort their descriptors to avoid repeating a transfer.
> +	 */
> +	list_for_each_entry_safe(vd, tmp, &chan->vc.desc_issued, node) {
> +		struct dw_edma_desc *desc = vd2dw_edma_desc(vd);
> +
> +		if (desc->start_burst == desc->done_burst)
> +			continue;
> +
> +		dw_edma_complete_vdesc(vd, DMA_TRANS_ABORTED);
> +	}

[Severity: Medium]
Does this leave unstarted descriptors in chan->vc.desc_issued stranded on an
idle channel after a hardware abort?

When issued descriptors exceed the free LL ring capacity, dw_edma_start_transfer()
stops once dw_edma_core_get_free_num(chan) reaches zero, leaving trailing
descriptors in chan->vc.desc_issued with start_burst == 0 and done_burst == 0.
When dw_edma_abort_interrupt() runs, dw_edma_core_reset_ll() skips those
unstarted descriptors, and dw_edma_abort_interrupt() then sets chan->status to
EDMA_ST_IDLE without starting them:

drivers/dma/dw-edma/dw-edma-core.c:dw_edma_abort_interrupt() {
    ...
	if (!chan->non_ll)
		dw_edma_core_reset_ll(chan);
	dw_edma_set_request(chan, EDMA_REQ_NONE);
	chan->status = EDMA_ST_IDLE;
	spin_unlock_irqrestore(&chan->vc.lock, flags);
}

Because the client already called dma_async_issue_pending() for those
descriptors, they never execute or complete unless another transfer is issued
later.

Conversely, if both DW_EDMA_DEFERRED_DONE and DW_EDMA_DEFERRED_ABORT are
pending in dw_edma_irq_work(), dw_edma_ll_interrupt() runs first and calls
dw_edma_ll_advance() -> dw_edma_start_transfer(), which publishes those
waiting descriptors into the freed LL slots (setting start_burst > 0) right
before dw_edma_abort_interrupt() -> dw_edma_core_reset_ll() aborts them with
DMA_TRANS_ABORTED.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=6

Re: [PATCH v8 4/7] dmaengine: dw-edma: Reclaim LL entries from hardware progress

[sashiko-bot]

> With dynamic appends, a DONE interrupt no longer means that all
> published LL entries have completed. Prepare for this by using LLP to
> advance ll_done and complete descriptors in issue order.
> 
> Reclaim completed entries and refill the ring as space becomes
> available. Restart a stopped channel when published work remains. This
> also allows descriptors larger than the ring to run in batches.
> 
> Hardware can keep running while the driver handles an interrupt, and
> LLP does not always track payload completion. Use the following
> exclusive completion boundaries, where idx is the raw LLP entry index
> and prev(idx) is the previous data entry in the ring:
> 
>   State/event          Meaning of raw LLP               Boundary
>   -------------------  -------------------------------  ---------
> [ ... ]
> - On pause, stop publishing new entries and enter the paused state once
>   the channel stops. Keep unfinished work for resume. Reclaim delayed
>   completions without restarting the channel.
> 
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=4

Re: [PATCH v8 6/7] dmaengine: dw-edma: Dynamically append requests while running

[Koichiro Den]

On Sat, Nov 07, 2026 at 01:59:34PM +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential
> issue to consider:
> - [Medium] dmaengine: dw-edma: stranded or aborted queued descriptors on
>   LL reset
> --
> 
> commit a60bb88e6010e30a59b85799859be8d9e37e8b49
> Author: Frank Li <Frank.Li@nxp.com>
> 
> dmaengine: dw-edma: Dynamically append requests while running
> 
> This commit allows dw_edma_device_issue_pending() to append new linked-list
> entries while a channel is running and aborts in-flight descriptors when
> resetting the ring.
> 
> > diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
> > index 1d13a25663eff..bcf4fca345914 100644
> > --- a/drivers/dma/dw-edma/dw-edma-core.c
> > +++ b/drivers/dma/dw-edma/dw-edma-core.c
> > @@ -111,8 +111,22 @@ static void dw_edma_complete_vdesc(struct virt_dma_desc *vd,
> >  
> >  static void dw_edma_core_reset_ll(struct dw_edma_chan *chan)
> >  {
> > +	struct virt_dma_desc *vd, *tmp;
> >  	u32 i;
> >  
> > +	/*
> > +	 * We cannot tell which pending entries completed before the reset.
> > +	 * Abort their descriptors to avoid repeating a transfer.
> > +	 */
> > +	list_for_each_entry_safe(vd, tmp, &chan->vc.desc_issued, node) {
> > +		struct dw_edma_desc *desc = vd2dw_edma_desc(vd);
> > +
> > +		if (desc->start_burst == desc->done_burst)
> > +			continue;
> > +
> > +		dw_edma_complete_vdesc(vd, DMA_TRANS_ABORTED);
> > +	}
> 
> [Severity: Medium]
> Does this leave unstarted descriptors in chan->vc.desc_issued stranded on an
> idle channel after a hardware abort?

Thanks, Sashiko.

I replied to this point in the v7 thread:
https://lore.kernel.org/r/rtfjvblptmdez2znjgx7t4v5fumj6wabp2ft6zmnsm353nsves@vow63wcrnpol/

 "Yes, and this has been long-standing behavior since commit e63d79d1ffcd
  ("dmaengine: Add Synopsys eDMA IP core driver"). It's orthogonal to this series.
  I think this should be handled in a separate fix."

> 
> When issued descriptors exceed the free LL ring capacity, dw_edma_start_transfer()
> stops once dw_edma_core_get_free_num(chan) reaches zero, leaving trailing
> descriptors in chan->vc.desc_issued with start_burst == 0 and done_burst == 0.
> When dw_edma_abort_interrupt() runs, dw_edma_core_reset_ll() skips those
> unstarted descriptors, and dw_edma_abort_interrupt() then sets chan->status to
> EDMA_ST_IDLE without starting them:
> 
> drivers/dma/dw-edma/dw-edma-core.c:dw_edma_abort_interrupt() {
>     ...
> 	if (!chan->non_ll)
> 		dw_edma_core_reset_ll(chan);
> 	dw_edma_set_request(chan, EDMA_REQ_NONE);
> 	chan->status = EDMA_ST_IDLE;
> 	spin_unlock_irqrestore(&chan->vc.lock, flags);
> }
> 
> Because the client already called dma_async_issue_pending() for those
> descriptors, they never execute or complete unless another transfer is issued
> later.
> 
> Conversely, if both DW_EDMA_DEFERRED_DONE and DW_EDMA_DEFERRED_ABORT are
> pending in dw_edma_irq_work(), dw_edma_ll_interrupt() runs first and calls
> dw_edma_ll_advance() -> dw_edma_start_transfer(), which publishes those
> waiting descriptors into the freed LL slots (setting start_burst > 0) right
> before dw_edma_abort_interrupt() -> dw_edma_core_reset_ll() aborts them with
> DMA_TRANS_ABORTED.

Same here too (from the same reply):

 "Yes, but I consider this an acceptable trade-off for keeping the error path
  simple."

Best regards,
Koichiro Den

> 
> -- 
> Sashiko AI review · https://sashiko.dev/#/patchset/20261107133027.364237-1-den@valinux.co.jp?part=6