* [PATCH v7 1/8] dmaengine: dw-axi-dmac: keep cyclic mode in the descriptor
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:07 ` [PATCH v7 2/8] dmaengine: dw-axi-dmac: poll the correct enable bit during termination James Hilliard
` (6 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard, stable
dw_axi_dma_chan_prep_cyclic() sets chan->cyclic before allocating all the
hardware descriptors and leaves it set when preparation fails. The flag
also survives cyclic transfer errors and channel resource release. A
subsequent noncyclic transfer then takes the cyclic completion path,
leaving its cookie incomplete and its descriptor on the issued list.
Its zero period_len can also cause a division by zero.
Resetting the flag on failure or channel allocation is not sufficient:
preparing cyclic work while an ordinary transfer is active changes how
that transfer's completion interrupt is handled.
Store the mode in each descriptor and use the completed descriptor's
mode in the interrupt handler. Cyclic preparation sets only its own
descriptor's flag, while the zero-initialized noncyclic descriptors keep
it clear. Error recovery and channel reuse then use the next descriptor's
mode without relying on channel-wide cleanup.
Fixes: 1deb96c0fa58 ("dmaegine: dw-axi-dmac: Support device_prep_dma_cyclic()")
Closes: https://lore.kernel.org/r/sashiko-outbox-208213@kernel.org/
Cc: stable@vger.kernel.org
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 35 ++++++++++++--------------
drivers/dma/dw-axi-dmac/dw-axi-dmac.h | 2 +-
2 files changed, 17 insertions(+), 20 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index 7520c78f55cc..6dae63024721 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -801,7 +801,7 @@ dw_axi_dma_chan_prep_cyclic(struct dma_chan *dchan, dma_addr_t dma_addr,
chan->direction = direction;
desc->chan = chan;
- chan->cyclic = true;
+ desc->cyclic = true;
desc->length = 0;
desc->period_len = period_len;
@@ -1117,25 +1117,23 @@ static void axi_chan_block_xfer_complete(struct axi_dma_chan *chan)
goto out;
}
- if (chan->cyclic) {
- desc = vd_to_axi_desc(vd);
- if (desc) {
- llp = lo_hi_readq(chan->chan_regs + CH_LLP);
- for (i = 0; i < count; i++) {
- hw_desc = &desc->hw_desc[i];
- if (hw_desc->llp == llp) {
- axi_chan_irq_clear(chan, hw_desc->lli->status_lo);
- hw_desc->lli->ctl_hi |= cpu_to_le32(CH_CTL_H_LLI_VALID);
- desc->completed_blocks = i;
-
- if (((hw_desc->len * (i + 1)) % desc->period_len) == 0)
- vchan_cyclic_callback(vd);
- break;
- }
+ desc = vd_to_axi_desc(vd);
+ if (desc->cyclic) {
+ llp = lo_hi_readq(chan->chan_regs + CH_LLP);
+ for (i = 0; i < count; i++) {
+ hw_desc = &desc->hw_desc[i];
+ if (hw_desc->llp == llp) {
+ axi_chan_irq_clear(chan, hw_desc->lli->status_lo);
+ hw_desc->lli->ctl_hi |= cpu_to_le32(CH_CTL_H_LLI_VALID);
+ desc->completed_blocks = i;
+
+ if (((hw_desc->len * (i + 1)) % desc->period_len) == 0)
+ vchan_cyclic_callback(vd);
+ break;
}
-
- axi_chan_enable(chan);
}
+
+ axi_chan_enable(chan);
} else {
/* Remove the completed descriptor from issued list before completing */
list_del(&vd->node);
@@ -1204,7 +1202,6 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
vchan_get_all_descriptors(&chan->vc, &head);
- chan->cyclic = false;
spin_unlock_irqrestore(&chan->vc.lock, flags);
vchan_dma_desc_free_list(&chan->vc, &head);
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
index d8dd07efe60f..3c86e5d468ea 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
@@ -50,7 +50,6 @@ struct axi_dma_chan {
struct axi_dma_desc *desc;
struct dma_slave_config config;
enum dma_transfer_direction direction;
- bool cyclic;
/* these other elements are all protected by vc.lock */
bool is_paused;
};
@@ -105,6 +104,7 @@ struct axi_dma_desc {
u32 length;
u32 period_len;
u32 nr_hw_descs;
+ bool cyclic;
};
struct axi_dma_chan_config {
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 2/8] dmaengine: dw-axi-dmac: poll the correct enable bit during termination
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
2026-11-18 0:07 ` [PATCH v7 1/8] dmaengine: dw-axi-dmac: keep cyclic mode in the descriptor James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:07 ` [PATCH v7 3/8] dmaengine: dw-axi-dmac: retain descriptors when termination times out James Hilliard
` (5 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard, stable
For channels 16 through 31, the channel enable bits occupy bits 32 through
47 of the extended DMAC_CHEN register. dma_chan_terminate_all() only reads
the low word and tests BIT(chan->id), so it can conclude that an upper-bank
channel has stopped while its actual enable bit remains set. Descriptors
can then be freed while hardware still owns them.
Poll axi_chan_is_hw_enable(), which already handles both register banks
and the smaller channel layouts, instead of constructing another mask.
Fixes: 495e18b16e3d ("dmaengine: dw-axi-dmac: Add support DMAX_NUM_CHANNELS > 16")
Cc: stable@vger.kernel.org
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index 6dae63024721..405ab9a78955 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -28,6 +28,7 @@
#include <linux/property.h>
#include <linux/reset.h>
#include <linux/slab.h>
+#include <linux/time64.h>
#include <linux/types.h>
#include "dw-axi-dmac.h"
@@ -1179,17 +1180,17 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
static int dma_chan_terminate_all(struct dma_chan *dchan)
{
struct axi_dma_chan *chan = dchan_to_axi_dma_chan(dchan);
- u32 chan_active = BIT(chan->id) << DMAC_CHAN_EN_SHIFT;
unsigned long flags;
- u32 val;
+ bool enabled;
int ret;
LIST_HEAD(head);
axi_chan_disable(chan);
- ret = readl_poll_timeout_atomic(chan->chip->regs + DMAC_CHEN, val,
- !(val & chan_active), 1000, 50000);
- if (ret == -ETIMEDOUT)
+ ret = readx_poll_timeout_atomic(axi_chan_is_hw_enable, chan,
+ enabled, !enabled,
+ 1 * USEC_PER_MSEC, 50 * USEC_PER_MSEC);
+ if (ret)
dev_warn(dmaengine_chan_dev(dchan),
"%s failed to stop\n", axi_chan_name(chan));
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 3/8] dmaengine: dw-axi-dmac: retain descriptors when termination times out
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
2026-11-18 0:07 ` [PATCH v7 1/8] dmaengine: dw-axi-dmac: keep cyclic mode in the descriptor James Hilliard
2026-11-18 0:07 ` [PATCH v7 2/8] dmaengine: dw-axi-dmac: poll the correct enable bit during termination James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:07 ` [PATCH v7 4/8] dmaengine: dw-axi-dmac: use atomic polling for suspend acknowledgment James Hilliard
` (4 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard, stable
dma_chan_terminate_all() warns when the channel enable bit does not clear,
but then frees the descriptors and returns success. Its synchronize
callback only drains virtual-channel callbacks, so terminate_sync() can
also report success while hardware still accesses the transfer.
Return the polling error before detaching descriptors or changing the
peripheral routing. This keeps the transfer resources available while
the channel remains enabled and lets the caller retry termination without
treating the timeout as a successful stop.
Hold the channel lock across disabling, polling and descriptor detachment
so an interrupt handler cannot restart the channel between the stop check
and cleanup.
Fixes: 1deb96c0fa58 ("dmaegine: dw-axi-dmac: Support device_prep_dma_cyclic()")
Cc: stable@vger.kernel.org
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index 405ab9a78955..040c12d07db2 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -1185,22 +1185,25 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
int ret;
LIST_HEAD(head);
+ spin_lock_irqsave(&chan->vc.lock, flags);
+
axi_chan_disable(chan);
ret = readx_poll_timeout_atomic(axi_chan_is_hw_enable, chan,
enabled, !enabled,
1 * USEC_PER_MSEC, 50 * USEC_PER_MSEC);
- if (ret)
+ if (ret) {
+ spin_unlock_irqrestore(&chan->vc.lock, flags);
dev_warn(dmaengine_chan_dev(dchan),
"%s failed to stop\n", axi_chan_name(chan));
+ return ret;
+ }
if (chan->direction != DMA_MEM_TO_MEM)
dw_axi_dma_set_hw_channel(chan, false);
if (chan->direction == DMA_MEM_TO_DEV)
dw_axi_dma_set_byte_halfword(chan, false);
- spin_lock_irqsave(&chan->vc.lock, flags);
-
vchan_get_all_descriptors(&chan->vc, &head);
spin_unlock_irqrestore(&chan->vc.lock, flags);
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 4/8] dmaengine: dw-axi-dmac: use atomic polling for suspend acknowledgment
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
` (2 preceding siblings ...)
2026-11-18 0:07 ` [PATCH v7 3/8] dmaengine: dw-axi-dmac: retain descriptors when termination times out James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:07 ` [PATCH v7 5/8] dmaengine: dw-axi-dmac: preserve pending pause requests James Hilliard
` (3 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard, stable
The pause loop delays after its final status read, so an acknowledgment
that arrives during that delay is cleared without being observed and
the caller receives -EAGAIN.
Use readx_poll_timeout_atomic() to check the status after the final delay.
Keep the two-microsecond polling interval, 40-microsecond timeout and
-EAGAIN return value when suspension is not acknowledged.
Fixes: 1fe20f1b8454 ("dmaengine: Introduce DW AXI DMAC driver")
Cc: stable@vger.kernel.org
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index 040c12d07db2..1a53fbc951a9 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -1219,7 +1219,8 @@ static int dma_chan_pause(struct dma_chan *dchan)
{
struct axi_dma_chan *chan = dchan_to_axi_dma_chan(dchan);
unsigned long flags;
- unsigned int timeout = 20; /* timeout iterations */
+ u32 status;
+ int ret;
u64 val;
spin_lock_irqsave(&chan->vc.lock, flags);
@@ -1250,12 +1251,9 @@ static int dma_chan_pause(struct dma_chan *dchan)
}
}
- do {
- if (axi_chan_irq_read(chan) & DWAXIDMAC_IRQ_SUSPENDED)
- break;
-
- udelay(2);
- } while (--timeout);
+ ret = readx_poll_timeout_atomic(axi_chan_irq_read, chan,
+ status, status & DWAXIDMAC_IRQ_SUSPENDED,
+ 2, 40);
axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
@@ -1263,7 +1261,7 @@ static int dma_chan_pause(struct dma_chan *dchan)
spin_unlock_irqrestore(&chan->vc.lock, flags);
- return timeout ? 0 : -EAGAIN;
+ return ret ? -EAGAIN : 0;
}
/* Called in chan locked context */
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 5/8] dmaengine: dw-axi-dmac: preserve pending pause requests
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
` (3 preceding siblings ...)
2026-11-18 0:07 ` [PATCH v7 4/8] dmaengine: dw-axi-dmac: use atomic polling for suspend acknowledgment James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:07 ` [PATCH v7 6/8] dmaengine: dw-axi-dmac: retain descriptors until callbacks drain James Hilliard
` (2 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard, stable
dma_chan_pause() clears SUSPENDED even when polling times out. Hardware
can set that status after the last poll, and the unconditional clear then
discards an acknowledgment the caller never observed. Another channel's
interrupt can also consume the status because the shared IRQ handler
clears every channel's status before taking its lock.
Leave SUSPENDED to the pause and resume paths. Track the suspend request
separately from its acknowledgment so a timeout can be followed by either
a pause retry or a resume. Wait for acknowledgment before clearing the
hardware request, since hardware ignores an earlier clear. If that wait
also times out, return -EAGAIN with the request still pending.
A completed transfer is also quiescent, but its completion handler may
still need to advance a cyclic descriptor. Defer that restart while a
pause is pending, and resume it without reprogramming the first LLI.
Track the active descriptor until its completion handler runs so that
resume and issue_pending cannot restart a completed cyclic segment or
reissue a completed noncyclic descriptor while its IRQ is pending.
Start queued work after the active descriptor is retired, respecting the
pause request.
Clear stale suspension status at the start of a new descriptor rather
than at each cyclic restart. Reset the software state after successful
termination and when allocating a channel for a new client.
Store the lock-protected pause flags as bitfields in existing padding.
Fixes: 1fe20f1b8454 ("dmaengine: Introduce DW AXI DMAC driver")
Cc: stable@vger.kernel.org
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 98 ++++++++++++++++++++++----
drivers/dma/dw-axi-dmac/dw-axi-dmac.h | 8 ++-
2 files changed, 90 insertions(+), 16 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index 1a53fbc951a9..cd9a5b45d295 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -481,6 +481,8 @@ static void axi_chan_block_xfer_start(struct axi_dma_chan *chan,
irq_mask |= DWAXIDMAC_IRQ_SUSPENDED;
axi_chan_irq_set(chan, irq_mask);
+ axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
+ chan->desc = first;
axi_chan_enable(chan);
}
@@ -489,6 +491,9 @@ static void axi_chan_start_first_queued(struct axi_dma_chan *chan)
struct axi_dma_desc *desc;
struct virt_dma_desc *vd;
+ if (chan->desc || chan->pause_requested)
+ return;
+
vd = vchan_next_desc(&chan->vc);
if (!vd)
return;
@@ -543,6 +548,10 @@ static int dma_chan_alloc_chan_resources(struct dma_chan *dchan)
pm_runtime_put(chan->chip->dev);
return -ENOMEM;
}
+ chan->desc = NULL;
+ chan->pause_requested = false;
+ chan->is_paused = false;
+ chan->cyclic_restart = false;
dev_vdbg(dmaengine_chan_dev(dchan), "%s: allocating\n", axi_chan_name(chan));
return 0;
@@ -1087,6 +1096,8 @@ static noinline void axi_chan_handle_err(struct axi_dma_chan *chan, u32 status)
vchan_cookie_complete(vd);
/* Try to restart the controller */
+ chan->desc = NULL;
+ chan->cyclic_restart = false;
axi_chan_start_first_queued(chan);
out:
@@ -1134,11 +1145,18 @@ static void axi_chan_block_xfer_complete(struct axi_dma_chan *chan)
}
}
- axi_chan_enable(chan);
+ if (chan->pause_requested) {
+ chan->is_paused = true;
+ chan->cyclic_restart = true;
+ } else {
+ axi_chan_enable(chan);
+ }
} else {
/* Remove the completed descriptor from issued list before completing */
list_del(&vd->node);
vchan_cookie_complete(vd);
+ chan->desc = NULL;
+ axi_chan_start_first_queued(chan);
}
out:
@@ -1160,6 +1178,8 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
for (i = 0; i < dw->hdata->nr_channels; i++) {
chan = &dw->chan[i];
status = axi_chan_irq_read(chan);
+ /* Leave suspension status for the pause and resume paths. */
+ status &= ~DWAXIDMAC_IRQ_SUSPENDED;
axi_chan_irq_clear(chan, status);
dev_vdbg(chip->dev, "%s %u IRQ status: 0x%08x\n",
@@ -1206,6 +1226,11 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
vchan_get_all_descriptors(&chan->vc, &head);
+ axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
+ chan->desc = NULL;
+ chan->pause_requested = false;
+ chan->is_paused = false;
+ chan->cyclic_restart = false;
spin_unlock_irqrestore(&chan->vc.lock, flags);
vchan_dma_desc_free_list(&chan->vc, &head);
@@ -1215,16 +1240,49 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
return 0;
}
+static bool axi_chan_is_suspended(struct axi_dma_chan *chan)
+{
+ if (axi_chan_is_hw_enable(chan))
+ return axi_chan_irq_read(chan) & DWAXIDMAC_IRQ_SUSPENDED;
+
+ /* A completed transfer cannot acknowledge a new suspend request. */
+ axi_chan_disable(chan);
+ return true;
+}
+
+/* A timed-out request can still be acknowledged. */
+static int axi_chan_wait_for_suspend(struct axi_dma_chan *chan)
+ __must_hold(&chan->vc.lock)
+{
+ bool suspended;
+ int ret;
+
+ if (chan->is_paused)
+ return 0;
+
+ ret = readx_poll_timeout_atomic(axi_chan_is_suspended, chan,
+ suspended, suspended, 2, 40);
+ if (ret)
+ return -EAGAIN;
+
+ axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
+ chan->is_paused = true;
+ return 0;
+}
+
static int dma_chan_pause(struct dma_chan *dchan)
{
struct axi_dma_chan *chan = dchan_to_axi_dma_chan(dchan);
unsigned long flags;
- u32 status;
int ret;
u64 val;
spin_lock_irqsave(&chan->vc.lock, flags);
+ chan->pause_requested = true;
+ if (chan->is_paused || !axi_chan_is_hw_enable(chan))
+ goto wait;
+
if (chan->chip->dw->hdata->nr_channels >= DMAC_CHAN_16) {
val = axi_dma_ioread64(chan->chip, DMAC_CHSUSPREG);
if (chan->id >= DMAC_CHAN_16) {
@@ -1251,17 +1309,11 @@ static int dma_chan_pause(struct dma_chan *dchan)
}
}
- ret = readx_poll_timeout_atomic(axi_chan_irq_read, chan,
- status, status & DWAXIDMAC_IRQ_SUSPENDED,
- 2, 40);
-
- axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
-
- chan->is_paused = true;
-
+wait:
+ ret = axi_chan_wait_for_suspend(chan);
spin_unlock_irqrestore(&chan->vc.lock, flags);
- return ret ? -EAGAIN : 0;
+ return ret;
}
/* Called in chan locked context */
@@ -1295,6 +1347,8 @@ static inline void axi_chan_resume(struct axi_dma_chan *chan)
}
}
+ axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
+ chan->pause_requested = false;
chan->is_paused = false;
}
@@ -1302,15 +1356,31 @@ static int dma_chan_resume(struct dma_chan *dchan)
{
struct axi_dma_chan *chan = dchan_to_axi_dma_chan(dchan);
unsigned long flags;
+ int ret;
spin_lock_irqsave(&chan->vc.lock, flags);
- if (chan->is_paused)
- axi_chan_resume(chan);
+ ret = 0;
+ if (!chan->pause_requested)
+ goto out;
+
+ /* Hardware ignores resume until the suspend request is acknowledged. */
+ ret = axi_chan_wait_for_suspend(chan);
+ if (ret)
+ goto out;
+
+ axi_chan_resume(chan);
+ if (chan->cyclic_restart) {
+ chan->cyclic_restart = false;
+ axi_chan_enable(chan);
+ } else {
+ axi_chan_start_first_queued(chan);
+ }
+out:
spin_unlock_irqrestore(&chan->vc.lock, flags);
- return 0;
+ return ret;
}
static int axi_dma_suspend(struct axi_dma_chip *chip)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
index 3c86e5d468ea..b18c9880711a 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
@@ -42,6 +42,12 @@ struct axi_dma_chan {
void __iomem *chan_regs;
u8 id;
u8 hw_handshake_num;
+
+ /* Protected by vc.lock. */
+ bool pause_requested:1;
+ bool is_paused:1;
+ bool cyclic_restart:1;
+
atomic_t descs_allocated;
struct dma_pool *desc_pool;
@@ -50,8 +56,6 @@ struct axi_dma_chan {
struct axi_dma_desc *desc;
struct dma_slave_config config;
enum dma_transfer_direction direction;
- /* these other elements are all protected by vc.lock */
- bool is_paused;
};
struct dw_axi_dma {
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 6/8] dmaengine: dw-axi-dmac: retain descriptors until callbacks drain
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
` (4 preceding siblings ...)
2026-11-18 0:07 ` [PATCH v7 5/8] dmaengine: dw-axi-dmac: preserve pending pause requests James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:07 ` [PATCH v7 7/8] dmaengine: dw-axi-dmac: serialize interrupt status with channel reuse James Hilliard
2026-11-18 0:07 ` [PATCH v7 8/8] dmaengine: dw-axi-dmac: guard shared IRQs across the controller lifetime James Hilliard
7 siblings, 0 replies; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard,
Wang YuWei, stable
Termination frees every descriptor while a pending period callback can
still refer to one through vc->cyclic. An executing callback has already
cleared that pointer, but still receives a pointer to the descriptor's
tx_result and can use it after termination frees the descriptor.
An error interrupt can also move the cyclic descriptor to desc_completed
before termination. Looking only at the head of desc_issued therefore
does not cover every pending cyclic callback.
Use vchan_terminate_all() after stopping the channel to clear the pending
alias and move the collected descriptors to desc_terminated. The existing
synchronize callback drains the tasklet before reclaiming that list.
Descriptors already detached onto the tasklet's private completed list
remain owned by the tasklet. Repeated termination preserves the retired
list, including when called from a completion callback.
Fixes: 1deb96c0fa58 ("dmaegine: dw-axi-dmac: Support device_prep_dma_cyclic()")
Reported-by: Wang YuWei <1973615295@qq.com>
Link: https://lore.kernel.org/r/tencent_C00D995A9F70206DFD1330DCB1222EEBCB07@qq.com/
Reported-by: Mitch Barnett <mitch@accuratesound.ca>
Cc: Wang YuWei <1973615295@qq.com>
Cc: stable@vger.kernel.org
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index cd9a5b45d295..97098367535f 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -1203,7 +1203,6 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
unsigned long flags;
bool enabled;
int ret;
- LIST_HEAD(head);
spin_lock_irqsave(&chan->vc.lock, flags);
@@ -1224,7 +1223,7 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
if (chan->direction == DMA_MEM_TO_DEV)
dw_axi_dma_set_byte_halfword(chan, false);
- vchan_get_all_descriptors(&chan->vc, &head);
+ vchan_terminate_all(&chan->vc);
axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
chan->desc = NULL;
@@ -1233,8 +1232,6 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
chan->cyclic_restart = false;
spin_unlock_irqrestore(&chan->vc.lock, flags);
- vchan_dma_desc_free_list(&chan->vc, &head);
-
dev_vdbg(dmaengine_chan_dev(dchan), "terminated: %s\n", axi_chan_name(chan));
return 0;
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 7/8] dmaengine: dw-axi-dmac: serialize interrupt status with channel reuse
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
` (5 preceding siblings ...)
2026-11-18 0:07 ` [PATCH v7 6/8] dmaengine: dw-axi-dmac: retain descriptors until callbacks drain James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:07 ` [PATCH v7 8/8] dmaengine: dw-axi-dmac: guard shared IRQs across the controller lifetime James Hilliard
7 siblings, 0 replies; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard, stable
The interrupt handler reads and acknowledges channel status before taking
the channel lock. Termination can finish and a new transfer can start
before the handler takes that lock. The saved completion or error status
then applies to the new descriptor, stopping it and completing its cookie
before its DMA transfer has finished.
Termination also leaves completion and error status pending. A subsequent
interrupt can consume that status after the channel has been reused, even
if no handler was running when the old transfer was stopped.
Read and acknowledge status under the channel lock and keep it held while
processing the event. Move locking out of the completion and error helpers
into their sole caller, and clear all channel status after a successful
stop. Status collection, descriptor ownership and channel reuse then share
the same serialization.
Initialize the virtual channels before requesting the shared IRQs so every
lock is ready before the interrupt handler can run.
Fixes: 1fe20f1b8454 ("dmaengine: Introduce DW AXI DMAC driver")
Cc: stable@vger.kernel.org
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 31 ++++++++++++--------------
1 file changed, 14 insertions(+), 17 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index 97098367535f..0116c44d3bef 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -20,6 +20,7 @@
#include <linux/iopoll.h>
#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/kernel.h>
+#include <linux/lockdep.h>
#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_dma.h>
@@ -1071,9 +1072,8 @@ static void axi_chan_list_dump_lli(struct axi_dma_chan *chan,
static noinline void axi_chan_handle_err(struct axi_dma_chan *chan, u32 status)
{
struct virt_dma_desc *vd;
- unsigned long flags;
- spin_lock_irqsave(&chan->vc.lock, flags);
+ lockdep_assert_held(&chan->vc.lock);
axi_chan_disable(chan);
@@ -1082,7 +1082,7 @@ static noinline void axi_chan_handle_err(struct axi_dma_chan *chan, u32 status)
if (!vd) {
dev_err(vchan_chan_dev(&chan->vc), "BUG: %s, IRQ with no descriptors\n",
axi_chan_name(chan));
- goto out;
+ return;
}
/* Remove the completed descriptor from issued list */
list_del(&vd->node);
@@ -1099,9 +1099,6 @@ static noinline void axi_chan_handle_err(struct axi_dma_chan *chan, u32 status)
chan->desc = NULL;
chan->cyclic_restart = false;
axi_chan_start_first_queued(chan);
-
-out:
- spin_unlock_irqrestore(&chan->vc.lock, flags);
}
static void axi_chan_block_xfer_complete(struct axi_dma_chan *chan)
@@ -1110,11 +1107,11 @@ static void axi_chan_block_xfer_complete(struct axi_dma_chan *chan)
struct axi_dma_hw_desc *hw_desc;
struct axi_dma_desc *desc;
struct virt_dma_desc *vd;
- unsigned long flags;
u64 llp;
int i;
- spin_lock_irqsave(&chan->vc.lock, flags);
+ lockdep_assert_held(&chan->vc.lock);
+
if (unlikely(axi_chan_is_hw_enable(chan))) {
dev_err(vchan_chan_dev(&chan->vc), "BUG: %s caught DWAXIDMAC_IRQ_DMA_TRF, but channel not idle!\n",
axi_chan_name(chan));
@@ -1126,7 +1123,7 @@ static void axi_chan_block_xfer_complete(struct axi_dma_chan *chan)
if (!vd) {
dev_err(vchan_chan_dev(&chan->vc), "BUG: %s, IRQ with no descriptors\n",
axi_chan_name(chan));
- goto out;
+ return;
}
desc = vd_to_axi_desc(vd);
@@ -1158,9 +1155,6 @@ static void axi_chan_block_xfer_complete(struct axi_dma_chan *chan)
chan->desc = NULL;
axi_chan_start_first_queued(chan);
}
-
-out:
- spin_unlock_irqrestore(&chan->vc.lock, flags);
}
static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
@@ -1168,6 +1162,7 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
struct axi_dma_chip *chip = dev_id;
struct dw_axi_dma *dw = chip->dw;
struct axi_dma_chan *chan;
+ unsigned long flags;
u32 status, i;
@@ -1177,6 +1172,7 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
/* Poll, clear and process every channel interrupt status */
for (i = 0; i < dw->hdata->nr_channels; i++) {
chan = &dw->chan[i];
+ spin_lock_irqsave(&chan->vc.lock, flags);
status = axi_chan_irq_read(chan);
/* Leave suspension status for the pause and resume paths. */
status &= ~DWAXIDMAC_IRQ_SUSPENDED;
@@ -1189,6 +1185,7 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
axi_chan_handle_err(chan, status);
else if (status & DWAXIDMAC_IRQ_DMA_TRF)
axi_chan_block_xfer_complete(chan);
+ spin_unlock_irqrestore(&chan->vc.lock, flags);
}
/* Re-enable interrupts */
@@ -1225,7 +1222,7 @@ static int dma_chan_terminate_all(struct dma_chan *dchan)
vchan_terminate_all(&chan->vc);
- axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_SUSPENDED);
+ axi_chan_irq_clear(chan, DWAXIDMAC_IRQ_ALL);
chan->desc = NULL;
chan->pause_requested = false;
chan->is_paused = false;
@@ -1599,10 +1596,6 @@ static int dw_probe(struct platform_device *pdev)
if (!dw->chan)
return -ENOMEM;
- ret = axi_req_irqs(pdev, chip);
- if (ret)
- return ret;
-
INIT_LIST_HEAD(&dw->dma.channels);
for (i = 0; i < hdata->nr_channels; i++) {
struct axi_dma_chan *chan = &dw->chan[i];
@@ -1616,6 +1609,10 @@ static int dw_probe(struct platform_device *pdev)
vchan_init(&chan->vc, &dw->dma);
}
+ ret = axi_req_irqs(pdev, chip);
+ if (ret)
+ return ret;
+
/* Set capabilities */
dma_cap_set(DMA_MEMCPY, dw->dma.cap_mask);
dma_cap_set(DMA_SLAVE, dw->dma.cap_mask);
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 8/8] dmaengine: dw-axi-dmac: guard shared IRQs across the controller lifetime
2026-11-18 0:07 [PATCH v7 0/8] dmaengine: dw-axi-dmac: fix termination and pause acknowledgment James Hilliard
` (6 preceding siblings ...)
2026-11-18 0:07 ` [PATCH v7 7/8] dmaengine: dw-axi-dmac: serialize interrupt status with channel reuse James Hilliard
@ 2026-11-18 0:07 ` James Hilliard
2026-11-18 0:32 ` sashiko-bot
7 siblings, 1 reply; 10+ messages in thread
From: James Hilliard @ 2026-11-18 0:07 UTC (permalink / raw)
To: Eugeniy Paltsev, Vinod Koul, Frank Li, Sergey Khimich,
Andy Shevchenko, Sia Jee Heng
Cc: Mitch Barnett, dmaengine, linux-kernel, James Hilliard
Requesting a shared interrupt allows the handler to run before probe has
enabled the clocks or registered the DMA channel devices. Initializing
the virtual channels first protects their locks, but does not make the
register accesses or channel-device diagnostics safe. The same register
access problem exists when another device raises a shared interrupt while
the controller is runtime-suspended.
Publish IRQ readiness after hardware initialization and DMA registration.
Once published, take a runtime PM reference only if the controller is
already active. This keeps the clocks enabled through the handler without
waking an idle controller for another device's interrupt. Drain handlers
before disabling clocks, including during forced system suspend, which
does not wait for their runtime PM references.
Keep the initial power reference until registration completes and mark
the manually resumed device active before enabling runtime PM. Balance
the clocks and reference on probe failure. Resume through runtime PM on
removal so an active controller does not acquire an extra clock reference.
Withdraw IRQ readiness and mask the controller before releasing its
shared IRQs. A handler already in flight can re-enable the source on
exit, so mask it again after draining all IRQs, while the clocks are
still enabled.
Fixes: 1fe20f1b8454 ("dmaengine: Introduce DW AXI DMAC driver")
Closes: https://lore.kernel.org/r/sashiko-outbox-216269@kernel.org/
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c | 77 ++++++++++++++++++++------
drivers/dma/dw-axi-dmac/dw-axi-dmac.h | 1 +
2 files changed, 60 insertions(+), 18 deletions(-)
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
index 0116c44d3bef..b1d753036744 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
@@ -276,6 +276,7 @@ static void axi_dma_hw_init(struct axi_dma_chip *chip)
for (i = 0; i < chip->dw->hdata->nr_channels; i++) {
axi_chan_irq_disable(&chip->dw->chan[i], DWAXIDMAC_IRQ_ALL);
axi_chan_disable(&chip->dw->chan[i]);
+ axi_chan_irq_clear(&chip->dw->chan[i], DWAXIDMAC_IRQ_ALL);
}
ret = dma_set_mask_and_coherent(chip->dev, DMA_BIT_MASK(64));
if (ret)
@@ -1166,6 +1167,14 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
u32 status, i;
+ /* Pair with publication after channel registration in dw_probe(). */
+ if (!smp_load_acquire(&chip->irq_ready))
+ return IRQ_NONE;
+
+ /* Shared interrupts can arrive while the controller is suspended. */
+ if (IS_ENABLED(CONFIG_PM) && pm_runtime_get_if_active(chip->dev) <= 0)
+ return IRQ_NONE;
+
/* Disable DMAC interrupts. We'll enable them after processing channels */
axi_dma_irq_disable(chip);
@@ -1191,6 +1200,9 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
/* Re-enable interrupts */
axi_dma_irq_enable(chip);
+ if (IS_ENABLED(CONFIG_PM))
+ pm_runtime_put(chip->dev);
+
return IRQ_HANDLED;
}
@@ -1377,12 +1389,28 @@ static int dma_chan_resume(struct dma_chan *dchan)
return ret;
}
-static int axi_dma_suspend(struct axi_dma_chip *chip)
+static void axi_dma_quiesce_irqs(struct axi_dma_chip *chip)
{
axi_dma_irq_disable(chip);
- axi_dma_disable(chip);
+ for (unsigned int i = 0; i < DMAC_MAX_CHANNELS; i++)
+ if (chip->irq[i] > 0)
+ synchronize_irq(chip->irq[i]);
+
+ /* A handler already in progress may have re-enabled interrupts. */
+ axi_dma_irq_disable(chip);
+}
+static void axi_dma_power_off(struct axi_dma_chip *chip)
+{
+ axi_dma_disable(chip);
clk_bulk_disable_unprepare(ARRAY_SIZE(chip->clks), chip->clks);
+}
+
+static int axi_dma_suspend(struct axi_dma_chip *chip)
+{
+ /* System sleep can suspend despite an IRQ handler's runtime PM reference. */
+ axi_dma_quiesce_irqs(chip);
+ axi_dma_power_off(chip);
return 0;
}
@@ -1651,8 +1679,6 @@ static int dw_probe(struct platform_device *pdev)
dma_set_max_seg_size(&pdev->dev, MAX_BLOCK_SIZE);
platform_set_drvdata(pdev, chip);
- pm_runtime_enable(chip->dev);
-
/*
* We can't just call pm_runtime_get here instead of
* pm_runtime_get_noresume + axi_dma_resume because we need
@@ -1661,16 +1687,23 @@ static int dw_probe(struct platform_device *pdev)
pm_runtime_get_noresume(chip->dev);
ret = axi_dma_resume(chip);
if (ret < 0)
- goto err_pm_disable;
+ goto err_pm_put;
axi_dma_hw_init(chip);
- pm_runtime_put(chip->dev);
+ ret = pm_runtime_set_active(chip->dev);
+ if (ret)
+ goto err_suspend;
+ pm_runtime_enable(chip->dev);
ret = dmaenginem_async_device_register(&dw->dma);
if (ret)
goto err_pm_disable;
+ /* Publish initialized hardware and channel devices to the IRQ handler. */
+ smp_store_release(&chip->irq_ready, true);
+ pm_runtime_put(chip->dev);
+
/* Register with OF helpers for DMA lookups */
ret = of_dma_controller_register(pdev->dev.of_node,
dw_axi_dma_of_xlate, dw);
@@ -1685,6 +1718,11 @@ static int dw_probe(struct platform_device *pdev)
err_pm_disable:
pm_runtime_disable(chip->dev);
+err_suspend:
+ axi_dma_suspend(chip);
+ pm_runtime_set_suspended(chip->dev);
+err_pm_put:
+ pm_runtime_put_noidle(chip->dev);
return ret;
}
@@ -1697,21 +1735,24 @@ static void dw_remove(struct platform_device *pdev)
u32 i;
int ret;
- /* Enable clk before accessing to registers */
- ret = clk_bulk_prepare_enable(ARRAY_SIZE(chip->clks), chip->clks);
- if (ret)
- dev_warn(chip->dev, "failed to enable clocks before remove: %d\n",
- ret);
- axi_dma_irq_disable(chip);
- for (i = 0; i < dw->hdata->nr_channels; i++) {
- axi_chan_disable(&chip->dw->chan[i]);
- axi_chan_irq_disable(&chip->dw->chan[i], DWAXIDMAC_IRQ_ALL);
+ WRITE_ONCE(chip->irq_ready, false);
+ ret = pm_runtime_resume_and_get(chip->dev);
+ pm_runtime_disable(chip->dev);
+ if (ret < 0) {
+ dev_warn(chip->dev, "failed to resume before remove: %d\n", ret);
+ goto free_irqs;
}
- axi_dma_disable(chip);
- pm_runtime_disable(chip->dev);
- axi_dma_suspend(chip);
+ axi_dma_quiesce_irqs(chip);
+ for (i = 0; i < dw->hdata->nr_channels; i++) {
+ axi_chan_disable(&dw->chan[i]);
+ axi_chan_irq_disable(&dw->chan[i], DWAXIDMAC_IRQ_ALL);
+ }
+ axi_dma_power_off(chip);
+ pm_runtime_put_noidle(chip->dev);
+ pm_runtime_set_suspended(chip->dev);
+free_irqs:
for (i = 0; i < DMAC_MAX_CHANNELS; i++)
if (chip->irq[i] > 0)
devm_free_irq(chip->dev, chip->irq[i], chip);
diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
index b18c9880711a..926d14b23615 100644
--- a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
+++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h
@@ -70,6 +70,7 @@ struct dw_axi_dma {
struct axi_dma_chip {
struct device *dev;
int irq[DMAC_MAX_CHANNELS];
+ bool irq_ready;
void __iomem *regs;
void __iomem *apb_regs;
struct clk_bulk_data clks[2];
--
2.53.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v7 8/8] dmaengine: dw-axi-dmac: guard shared IRQs across the controller lifetime
2026-11-18 0:07 ` [PATCH v7 8/8] dmaengine: dw-axi-dmac: guard shared IRQs across the controller lifetime James Hilliard
@ 2026-11-18 0:32 ` sashiko-bot
0 siblings, 0 replies; 10+ messages in thread
From: sashiko-bot @ 2026-11-18 0:32 UTC (permalink / raw)
To: James Hilliard; +Cc: dmaengine, vkoul, Frank.Li
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] dmaengine: dw-axi-dmac: unhandled IRQ storm in dw_probe()
before irq_ready
- [Medium] dmaengine: dw-axi-dmac: unhandled IRQ in dw_remove() before
masking DMAC
- [Medium] dmaengine: dw-axi-dmac: dw_axi_dma_interrupt() drops IRQs in
PM transitions
--
commit 7c162a89d8b1aefab137f65050f65004c3cd340c
Author: James Hilliard <james.hilliard1@gmail.com>
dmaengine: dw-axi-dmac: guard shared IRQs across the controller lifetime
This commit gates the shared interrupt handler on an irq_ready flag and a
runtime PM active check, while updating probe, suspend, and remove paths
to drain and mask interrupts around power state transitions.
> diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
> index 0116c44d3bef1..b1d7530367448 100644
> --- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
> +++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c
[ ... ]
> @@ -1166,6 +1167,14 @@ static irqreturn_t dw_axi_dma_interrupt(int irq, void *dev_id)
>
> u32 status, i;
>
> + /* Pair with publication after channel registration in dw_probe(). */
> + if (!smp_load_acquire(&chip->irq_ready))
> + return IRQ_NONE;
> +
> + /* Shared interrupts can arrive while the controller is suspended. */
> + if (IS_ENABLED(CONFIG_PM) && pm_runtime_get_if_active(chip->dev) <= 0)
> + return IRQ_NONE;
[Severity: Medium]
Can pm_runtime_get_if_active() drop valid controller interrupts during PM
transitions?
pm_runtime_get_if_active() returns 0 unless runtime_status is RPM_ACTIVE,
and returns -EINVAL when runtime PM is disabled (disable_depth > 0).
However, controller interrupts (INT_EN_MASK in DMAC_CFG) are enabled in
hardware during transition windows where those conditions do not hold:
- In rpm_resume(), runtime_status is still RPM_RESUMING when
axi_dma_runtime_resume() -> axi_dma_resume() calls axi_dma_irq_enable().
- In rpm_suspend(), runtime_status is already RPM_SUSPENDING before
axi_dma_runtime_suspend() -> axi_dma_quiesce_irqs() masks INT_EN_MASK.
- In pm_runtime_force_suspend() and pm_runtime_force_resume(),
disable_depth is non-zero while hardware interrupts are still enabled
before suspend or already re-enabled during resume.
If the controller asserts its level-triggered interrupt line during any of
these windows, dw_axi_dma_interrupt() returns IRQ_NONE without masking
DMAC_CFG or clearing CH_INTSTATUS, causing an unhandled IRQ storm that can
disable the shared IRQ line.
[ ... ]
> @@ -1661,16 +1687,23 @@ static int dw_probe(struct platform_device *pdev)
> pm_runtime_get_noresume(chip->dev);
> ret = axi_dma_resume(chip);
> if (ret < 0)
> - goto err_pm_disable;
> + goto err_pm_put;
>
> axi_dma_hw_init(chip);
>
> - pm_runtime_put(chip->dev);
> + ret = pm_runtime_set_active(chip->dev);
> + if (ret)
> + goto err_suspend;
> + pm_runtime_enable(chip->dev);
>
> ret = dmaenginem_async_device_register(&dw->dma);
> if (ret)
> goto err_pm_disable;
>
> + /* Publish initialized hardware and channel devices to the IRQ handler. */
> + smp_store_release(&chip->irq_ready, true);
> + pm_runtime_put(chip->dev);
[Severity: High]
Can dw_probe() trigger an unhandled interrupt storm before chip->irq_ready
is set to true?
axi_req_irqs() has already registered dw_axi_dma_interrupt() earlier in
dw_probe(), and there are two windows where hardware interrupts are enabled
while chip->irq_ready is still false:
1. axi_dma_resume() enables global DMAC interrupts via
axi_dma_irq_enable() before axi_dma_hw_init() masks and clears the
per-channel interrupts. On platforms without a reset line, or after a
kexec or prior unbind (since dw_remove() does not clear CH_INTSTATUS),
any stale pending channel interrupt will immediately assert the IRQ
line while chip->irq_ready is false.
2. dmaenginem_async_device_register() publishes the DMA_MEMCPY channels to
in-kernel consumers before smp_store_release(&chip->irq_ready, true)
runs, so a client on another CPU can allocate a channel and complete a
transfer before chip->irq_ready becomes true.
In either window, dw_axi_dma_interrupt() returns IRQ_NONE without masking
or clearing the level-triggered interrupt, causing a hardirq storm until
note_interrupt() permanently disables the shared IRQ line.
[ ... ]
> @@ -1697,21 +1735,24 @@ static void dw_remove(struct platform_device *pdev)
> u32 i;
> int ret;
>
> - /* Enable clk before accessing to registers */
> - ret = clk_bulk_prepare_enable(ARRAY_SIZE(chip->clks), chip->clks);
> - if (ret)
> - dev_warn(chip->dev, "failed to enable clocks before remove: %d\n",
> - ret);
> - axi_dma_irq_disable(chip);
> - for (i = 0; i < dw->hdata->nr_channels; i++) {
> - axi_chan_disable(&chip->dw->chan[i]);
> - axi_chan_irq_disable(&chip->dw->chan[i], DWAXIDMAC_IRQ_ALL);
> + WRITE_ONCE(chip->irq_ready, false);
> + ret = pm_runtime_resume_and_get(chip->dev);
> + pm_runtime_disable(chip->dev);
> + if (ret < 0) {
> + dev_warn(chip->dev, "failed to resume before remove: %d\n", ret);
> + goto free_irqs;
> }
> - axi_dma_disable(chip);
>
> - pm_runtime_disable(chip->dev);
> - axi_dma_suspend(chip);
> + axi_dma_quiesce_irqs(chip);
[Severity: Medium]
Does clearing chip->irq_ready and disabling runtime PM before
axi_dma_quiesce_irqs() leave hardware interrupts unmasked while
dw_axi_dma_interrupt() refuses to handle them?
In dw_remove(), WRITE_ONCE(chip->irq_ready, false) and
pm_runtime_disable() both run before axi_dma_quiesce_irqs() masks
INT_EN_MASK in DMAC_CFG. Furthermore, if the device was runtime-suspended,
pm_runtime_resume_and_get() calls axi_dma_resume() -> axi_dma_irq_enable()
and unmasks INT_EN_MASK while chip->irq_ready is already false. Any
level-triggered DMAC interrupt firing in this window causes
dw_axi_dma_interrupt() to return IRQ_NONE without masking or clearing the
source, which can trigger a spurious IRQ storm and disable the shared IRQ
line.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117-submit-dw-axi-dma-stop-pause-v1-v7-0-d5f2309afd72@gmail.com?part=8
^ permalink raw reply [flat|nested] 10+ messages in thread