[PATCH v4 0/3] dmaengine: dw-edma: Prepare channels for remote use

[PATCH v4 0/3] dmaengine: dw-edma: Prepare channels for remote use

[Koichiro Den]

Hi,

This small series contains standalone refactoring and new infrastructure
for dmaengine and dw-edma.

It prepares PCIe EPC-local DMA channels for remote use. The upcoming
vNTB-embedded DMA support [1] will be the first user and depend on this
series.

[1] https://lore.kernel.org/r/20260903082327.2345602-1-den@valinux.co.jp/

Best regards,
Koichiro
---
Changes in v4:
  - Drop unnecessary READ_ONCE()/WRITE_ONCE() for irq_mode. (Frank)
  - Allow repeated dmaengine_slave_config() calls on idle channels when
    the IRQ mode is unchanged. (Sashiko)
  - Extend patch 2 to support native HDMA.
  - Simplify IRQ mode handling, assuming the channel has no pending
    interrupt status when its mode changes. Treat racing reads by shared
    IRQ handlers and same-value stores on release as harmless.
  - No changes in patches 1 and 3.

Changes in v3:
  - Rework patch 2 to use a common channel configuration, retain generic
    dma_slave_config fields, and close the shared-IRQ routing race.
    (Frank, Sashiko)
  - No code changes in patches 1 and 3.

Changes in v2:
  - Split and rework vNTB v1 patches 1, 4, and 5 into this prerequisite
    series.
  - Fold in the relevant PCI DMA EPF v7 review.

v3: https://lore.kernel.org/r/20260903064533.2269557-1-den@valinux.co.jp/
v2: https://lore.kernel.org/r/20260828163611.2691264-1-den@valinux.co.jp/
v1: https://lore.kernel.org/r/20260312165005.1148676-1-den@valinux.co.jp/


Koichiro Den (3):
  dmaengine: Allow drivers to assign static channel IDs
  dmaengine: dw-edma: Configure remote interrupt routing
  dmaengine: dw-edma: Account for the MSI vector offset

 drivers/dma/dmaengine.c            |  13 ++-
 drivers/dma/dw-edma/dw-edma-core.c | 161 +++++++++++++++++++++--------
 include/linux/dma/edma.h           |  21 ++++
 include/linux/dmaengine.h          |  20 ++++
 4 files changed, 169 insertions(+), 46 deletions(-)


base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
-- 
2.51.0

[PATCH v4 1/3] dmaengine: Allow drivers to assign static channel IDs

[Koichiro Den]

The dmaengine core assigns channel IDs in registration order. If a driver
skips a hardware channel, chan_id can differ from the hardware numbering
and a client cannot reliably correlate a requested channel with hardware
resources.

Let a driver request an exact channel ID before device registration.
Reserve static IDs through the existing IDA so they remain unique, while
retaining automatic IDA allocation as the default.

For example, idma32 uses chan_id to select DMA_CTL_CH() and
DMA_XBAR_SEL(), so it relies on ascending registration order to match
chan_id with the hardware channel number.

Use direction-flattened IDs for dw-edma channels. Unlike the
direction-local hardware channel number, these IDs are unique within the
DMA device.

Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v4:
  - No changes.

 drivers/dma/dmaengine.c            | 13 ++++++++-----
 drivers/dma/dw-edma/dw-edma-core.c |  1 +
 include/linux/dmaengine.h          | 20 ++++++++++++++++++++
 3 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index 6ffd8bd82154..cc64a4679e6f 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -1078,6 +1078,7 @@ static int __dma_async_device_channel_register(struct dma_device *device,
 					       struct dma_chan *chan,
 					       const char *name)
 {
+	unsigned int id;
 	int rc;
 
 	chan->local = alloc_percpu(typeof(*chan->local));
@@ -1089,11 +1090,13 @@ static int __dma_async_device_channel_register(struct dma_device *device,
 		goto err_free_local;
 	}
 
-	/*
-	 * When the chan_id is a negative value, we are dynamically adding
-	 * the channel. Otherwise we are static enumerating.
-	 */
-	chan->chan_id = ida_alloc(&device->chan_ida, GFP_KERNEL);
+	if (chan->chan_id & DMA_CHAN_ID_STATIC) {
+		id = chan->chan_id & ~DMA_CHAN_ID_STATIC;
+		chan->chan_id = ida_alloc_range(&device->chan_ida, id, id,
+						GFP_KERNEL);
+	} else {
+		chan->chan_id = ida_alloc(&device->chan_ida, GFP_KERNEL);
+	}
 	if (chan->chan_id < 0) {
 		pr_err("%s: unable to alloc ida for chan: %d\n",
 		       __func__, chan->chan_id);
diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index 03b2c2188351..a678c70a78fe 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -988,6 +988,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)
 					&dw->chip->dt_region_rd[chan->id];
 
 		vchan_init(&chan->vc, dma);
+		dmaengine_set_static_chan_id(&chan->vc.chan, i);
 
 		dw_edma_core_ch_config(chan);
 	}
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index fe33a20abc61..f669b79d7731 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -369,6 +369,26 @@ struct dma_chan {
 	void *private;
 };
 
+#define DMA_CHAN_ID_STATIC	BIT(30)
+
+/**
+ * dmaengine_set_static_chan_id - request an exact DMA engine channel ID
+ * @chan: DMA channel
+ * @id: channel ID, unique within the DMA device
+ *
+ * Drivers may call this after initializing @chan and before registering its
+ * DMA device. The dmaengine core reserves @id from the device IDA instead of
+ * assigning the next available ID.
+ */
+static inline void dmaengine_set_static_chan_id(struct dma_chan *chan,
+						unsigned int id)
+{
+	if (WARN_ON_ONCE(id >= DMA_CHAN_ID_STATIC))
+		return;
+
+	chan->chan_id = DMA_CHAN_ID_STATIC | id;
+}
+
 /**
  * struct dma_chan_dev - relate sysfs device node to backing channel device
  * @chan: driver channel device
-- 
2.51.0

[PATCH v4 2/3] dmaengine: dw-edma: Configure remote interrupt routing

[Koichiro Den]

An endpoint function can reserve an endpoint-local channel while the RC
programs it through an exposed register window. Such a channel must route
interrupts remotely and ignore them on the endpoint.

Use dma_slave_config to set per-channel interrupt routing on idle channels
of a local eDMA or HDMA instance. Releasing a remote-routed channel
quiesces the hardware and drains its local IRQ before restoring default
routing.

The eDMA quiesce may stop a complete direction. The caller must own every
channel in that direction and stop remote programming first.

Suggested-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v4:
  - Drop unnecessary READ_ONCE()/WRITE_ONCE() for irq_mode. (Frank)
  - Allow repeated dmaengine_slave_config() calls on idle channels
    when the IRQ mode is unchanged. (Sashiko)
  - Simplify IRQ mode handling, assuming the channel has no pending
    interrupt status when its mode changes. Treat racing reads by
    shared IRQ handlers and same-value stores on release as harmless.
  - Support native HDMA.

 drivers/dma/dw-edma/dw-edma-core.c | 136 ++++++++++++++++++++++-------
 include/linux/dma/edma.h           |  21 +++++
 2 files changed, 125 insertions(+), 32 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index a678c70a78fe..c978da30bac5 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -177,48 +177,76 @@ dw_edma_get_default_irq_mode(struct dw_edma_chan *chan)
 						  DW_EDMA_CH_IRQ_REMOTE;
 }
 
+static int dw_edma_device_config_irq_mode(struct dw_edma_chan *chan,
+					  enum dw_edma_ch_irq_mode mode)
+{
+	if (!(chan->dw->chip->flags & DW_EDMA_CHIP_LOCAL) ||
+	    (mode != DW_EDMA_CH_IRQ_LOCAL && mode != DW_EDMA_CH_IRQ_REMOTE))
+		return -EINVAL;
+
+	guard(spinlock_irqsave)(&chan->vc.lock);
+
+	if (chan->status != EDMA_ST_IDLE || chan->request != EDMA_REQ_NONE)
+		return -EBUSY;
+
+	/* IRQ routing cannot change after the initial configuration. */
+	if (chan->irq_mode == mode)
+		return 0;
+
+	if (chan->configured)
+		return -EBUSY;
+
+	chan->irq_mode = mode;
+
+	return 0;
+}
+
 static int dw_edma_device_config(struct dma_chan *dchan,
 				 struct dma_slave_config *config)
 {
+	const struct dw_edma_chan_config *dw_config = config->peripheral_config;
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
-	bool cfg_non_ll;
-	int non_ll = 0;
-
-	chan->non_ll = false;
-	if (chan->dw->chip->mf == EDMA_MF_HDMA_NATIVE) {
-		if (config->peripheral_config &&
-		    config->peripheral_size != sizeof(int)) {
-			dev_err(dchan->device->dev,
-				"config param peripheral size mismatch\n");
+	bool non_ll = false;
+	u32 flags = 0;
+	int ret;
+
+	if (dw_config) {
+		if (config->peripheral_size != sizeof(*dw_config) ||
+		    dw_config->flags & ~(DW_EDMA_CH_CONFIG_NON_LL |
+					 DW_EDMA_CH_CONFIG_IRQ_MODE))
 			return -EINVAL;
-		}
+		flags = dw_config->flags;
+	}
 
-		/*
-		 * When there is no valid LLP base address available then the
-		 * default DMA ops will use the non-LL mode.
-		 *
-		 * Cases where LL mode is enabled and client wants to use the
-		 * non-LL mode then also client can do so via providing the
-		 * peripheral_config param.
-		 */
-		cfg_non_ll = chan->dw->chip->cfg_non_ll;
-		if (config->peripheral_config) {
-			non_ll = *(int *)config->peripheral_config;
+	/*
+	 * When there is no valid LLP base address available then the
+	 * default DMA ops will use the non-LL mode.
+	 *
+	 * When LL mode is the default, clients can request non-LL mode
+	 * through DW_EDMA_CH_CONFIG_NON_LL.
+	 */
+	non_ll = chan->dw->chip->mf == EDMA_MF_HDMA_NATIVE &&
+		 chan->dw->chip->cfg_non_ll;
 
-			if (cfg_non_ll && !non_ll) {
-				dev_err(dchan->device->dev, "invalid configuration\n");
-				return -EINVAL;
-			}
+	if (flags & DW_EDMA_CH_CONFIG_NON_LL) {
+		if (chan->dw->chip->mf != EDMA_MF_HDMA_NATIVE)
+			return -EINVAL;
+
+		if (chan->dw->chip->cfg_non_ll && !dw_config->non_ll) {
+			dev_err(dchan->device->dev, "invalid configuration\n");
+			return -EINVAL;
 		}
 
-		if (cfg_non_ll || non_ll)
-			chan->non_ll = true;
-	} else if (config->peripheral_config) {
-		dev_err(dchan->device->dev,
-			"peripheral config param applicable only for HDMA\n");
-		return -EINVAL;
+		non_ll = dw_config->non_ll;
+	}
+
+	if (flags & DW_EDMA_CH_CONFIG_IRQ_MODE) {
+		ret = dw_edma_device_config_irq_mode(chan, dw_config->irq_mode);
+		if (ret)
+			return ret;
 	}
 
+	chan->non_ll = non_ll;
 	memcpy(&chan->config, config, sizeof(*config));
 	chan->configured = true;
 
@@ -890,11 +918,53 @@ static void dw_edma_wait_termination(struct dma_chan *dchan)
 		 "timeout waiting for channel termination\n");
 }
 
+static void dw_edma_synchronize_chan_irq(struct dw_edma_chan *chan)
+{
+	struct dw_edma *dw = chan->dw;
+	unsigned long *mask;
+	int i;
+
+	/*
+	 * A shared handler may retain this channel's status across quiesce.
+	 * With nr_irqs == 1, it scans both directions even if routing and
+	 * delegation are direction-wide. Drain it before allowing a routing change.
+	 */
+	for (i = 0; i < dw->nr_irqs; i++) {
+		mask = chan->dir == EDMA_DIR_WRITE ? dw->irq[i].wr_mask :
+						     dw->irq[i].rd_mask;
+		if (!test_bit(chan->id, mask))
+			continue;
+
+		synchronize_irq(dw->chip->ops->irq_vector(dw->chip->dev, i));
+		return;
+	}
+}
+
 static void dw_edma_device_synchronize(struct dma_chan *dchan)
 {
 	struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+	bool remote;
+
+	/*
+	 * irq_mode is fixed after initial configuration. The free path
+	 * restores it only after synchronization.
+	 */
+	remote = chan->dw->chip->flags & DW_EDMA_CHIP_LOCAL &&
+		 chan->irq_mode == DW_EDMA_CH_IRQ_REMOTE;
+
+	/*
+	 * Peer-driven transfers bypass local descriptor tracking, so quiesce
+	 * the hardware explicitly.
+	 */
+	if (remote && dw_edma_core_ch_quiesce(chan))
+		dev_warn(chan->dw->chip->dev,
+			 "failed to quiesce remote-routed %s channel %u\n",
+			 chan->dir == EDMA_DIR_WRITE ? "write" : "read",
+			 chan->id);
 
 	dw_edma_wait_termination(dchan);
+	if (remote)
+		dw_edma_synchronize_chan_irq(chan);
 	cancel_work_sync(&chan->irq_work);
 	atomic_set(&chan->irq_pending, 0);
 	vchan_synchronize(&chan->vc);
@@ -907,8 +977,10 @@ static void dw_edma_free_chan_resources(struct dma_chan *dchan)
 	dw_edma_device_terminate_all(dchan);
 	dw_edma_device_synchronize(dchan);
 
-	scoped_guard(spinlock_irqsave, &chan->vc.lock)
+	scoped_guard(spinlock_irqsave, &chan->vc.lock) {
 		chan->configured = false;
+		chan->irq_mode = dw_edma_get_default_irq_mode(chan);
+	}
 
 	vchan_free_chan_resources(&chan->vc);
 }
diff --git a/include/linux/dma/edma.h b/include/linux/dma/edma.h
index 3c8e2ef9dee0..43831fa57357 100644
--- a/include/linux/dma/edma.h
+++ b/include/linux/dma/edma.h
@@ -101,6 +101,27 @@ enum dw_edma_ch_irq_mode {
 	DW_EDMA_CH_IRQ_REMOTE,
 };
 
+#define DW_EDMA_CH_CONFIG_NON_LL	BIT(0)
+#define DW_EDMA_CH_CONFIG_IRQ_MODE	BIT(1)
+
+/**
+ * struct dw_edma_chan_config - dw-edma channel configuration
+ * @flags: fields selected by DW_EDMA_CH_CONFIG_*
+ * @non_ll: use HDMA non-linked-list mode
+ * @irq_mode: interrupt routing mode
+ *
+ * Pass this structure through dma_slave_config.peripheral_config. Before
+ * synchronizing a remote-routed channel, the client must stop remote
+ * programming and own every channel affected by the hardware quiesce: the
+ * entire direction for eDMA-compatible layouts, or the individual channel for
+ * native HDMA.
+ */
+struct dw_edma_chan_config {
+	u32 flags;
+	bool non_ll;
+	enum dw_edma_ch_irq_mode irq_mode;
+};
+
 /**
  * struct dw_edma_chip - representation of DesignWare eDMA controller hardware
  * @dev:		 struct device of the eDMA controller
-- 
2.51.0

[PATCH v4 3/3] dmaengine: dw-edma: Account for the MSI vector offset

[Koichiro Den]

get_cached_msi_msg() returns the base message shared by a multi-MSI
descriptor. dw-edma currently derives per-channel data from its local IRQ
index and does not adjust a common IRQ at all. Both assume eDMA starts at
the descriptor's first vector.

That is not true when eDMA receives a tail subset. Compose each message
from the IRQ offset relative to the descriptor base in both paths.

While at it, avoid reading PCI MSI attributes from descriptors owned by
non-PCI devices.

Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v4:
  - No changes.

 drivers/dma/dw-edma/dw-edma-core.c | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index c978da30bac5..1e6fbe0922a9 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -7,6 +7,7 @@
  */
 
 #include <linux/module.h>
+#include <linux/pci.h>
 #include <linux/delay.h>
 #include <linux/device.h>
 #include <linux/kernel.h>
@@ -1106,12 +1107,23 @@ static inline void dw_edma_dec_irq_alloc(int *nr_irqs, u32 *alloc, u16 cnt)
 	}
 }
 
+static void dw_edma_compose_msi(int irq, struct msi_msg *msi)
+{
+	struct msi_desc *desc = irq_get_msi_desc(irq);
+
+	if (!desc)
+		return;
+
+	get_cached_msi_msg(irq, msi);
+	if (dev_is_pci(desc->dev) && !desc->pci.msi_attrib.is_msix)
+		msi->data += irq - desc->irq;
+}
+
 static int dw_edma_irq_request(struct dw_edma *dw,
 			       u32 *wr_alloc, u32 *rd_alloc)
 {
 	struct dw_edma_chip *chip = dw->chip;
 	struct device *dev = dw->chip->dev;
-	struct msi_desc *msi_desc;
 	int i, err = 0;
 	u32 ch_cnt;
 	int irq;
@@ -1136,8 +1148,7 @@ static int dw_edma_irq_request(struct dw_edma *dw,
 			return err;
 		}
 
-		if (irq_get_msi_desc(irq))
-			get_cached_msi_msg(irq, &dw->irq[0].msi);
+		dw_edma_compose_msi(irq, &dw->irq[0].msi);
 
 		dw->nr_irqs = 1;
 	} else {
@@ -1160,12 +1171,7 @@ static int dw_edma_irq_request(struct dw_edma *dw,
 					  &dw->irq[i]);
 			if (err)
 				goto err_irq_free;
-			msi_desc = irq_get_msi_desc(irq);
-			if (msi_desc) {
-				get_cached_msi_msg(irq, &dw->irq[i].msi);
-				if (!msi_desc->pci.msi_attrib.is_msix)
-					dw->irq[i].msi.data = dw->irq[0].msi.data + i;
-			}
+			dw_edma_compose_msi(irq, &dw->irq[i].msi);
 		}
 
 		dw->nr_irqs = i;
-- 
2.51.0

Re: [PATCH v4 1/3] dmaengine: Allow drivers to assign static channel IDs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but there is unprotected list manipulation and channel count increments when dynamically registering DMA channels.
- [High] This problem wasn't introduced by this patch, but the exported function `dma_async_device_channel_register` calls `dma_channel_rebalance` without holding the required `dma_list_mutex`.
--

commit c9b886d4c7abff6716428d3defc1d41c973eb40b
Author: Koichiro Den <den@valinux.co.jp>

dmaengine: Allow drivers to assign static channel IDs

The dmaengine core assigns channel IDs in registration order. If a driver
skips a hardware channel, chan_id can differ from the hardware numbering
and a client cannot reliably correlate a requested channel with hardware
resources.

> diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
> index 6ffd8bd82154a..cc64a4679e6fa 100644
> --- a/drivers/dma/dmaengine.c
> +++ b/drivers/dma/dmaengine.c
> @@ -1078,6 +1078,7 @@ static int __dma_async_device_channel_register(struct dma_device *device,
>  					       struct dma_chan *chan,
>  					       const char *name)
>  {
> +	unsigned int id;
>  	int rc;
>  
>  	chan->local = alloc_percpu(typeof(*chan->local));

[ ... ]

> -	/*
> -	 * When the chan_id is a negative value, we are dynamically adding
> -	 * the channel. Otherwise we are static enumerating.
> -	 */
> -	chan->chan_id = ida_alloc(&device->chan_ida, GFP_KERNEL);
> +	if (chan->chan_id & DMA_CHAN_ID_STATIC) {
> +		id = chan->chan_id & ~DMA_CHAN_ID_STATIC;
> +		chan->chan_id = ida_alloc_range(&device->chan_ida, id, id,
> +						GFP_KERNEL);
> +	} else {
> +		chan->chan_id = ida_alloc(&device->chan_ida, GFP_KERNEL);
> +	}
>  	if (chan->chan_id < 0) {
>  		pr_err("%s: unable to alloc ida for chan: %d\n",
>  		       __func__, chan->chan_id);

[Severity: High]
This problem wasn't introduced by this patch, but there is unprotected list
manipulation and channel count increments when dynamically registering DMA
channels.

When dynamically registering DMA channels, the idxd driver calls
list_add_tail(&chan->device_node, &dma->channels) in
drivers/dma/idxd/dma.c:idxd_register_dma_channel() and then
dma_async_device_channel_register() calls
drivers/dma/dmaengine.c:__dma_async_device_channel_register() which does:

	chan->client_count = 0;
	device->chancnt++;

	return 0;

Since idxd probes workqueues (WQs) as separate devices, these can be probed
concurrently. Without proper locking protecting the dma_device's channel list
and count, concurrent registrations can result in list corruption and race
conditions, potentially leading to kernel panics or data corruption.

Could we add proper locking to protect the channel list and count during
dynamic channel registration?

[Severity: High]
This problem wasn't introduced by this patch, but the exported function
dma_async_device_channel_register() calls dma_channel_rebalance() without
holding the required dma_list_mutex.

In drivers/dma/dmaengine.c:dma_async_device_channel_register():

int dma_async_device_channel_register(struct dma_device *device,
				      struct dma_chan *chan)
{
	int rc;

	rc = __dma_async_device_channel_register(device, chan, NULL);
	if (rc < 0)
		return rc;

	dma_channel_rebalance();
	return 0;
}

The dma_channel_rebalance() function iterates over the global dma_device_list
using list_for_each_entry(). Because dma_list_mutex is a static unexported
mutex in drivers/dma/dmaengine.c, callers of
dma_async_device_channel_register() cannot hold it.

This leads to an unprotected concurrent traversal of the global DMA device list,
which can race with other drivers registering or unregistering DMA devices.
Could this result in use-after-free or list corruption crashes?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260912174023.1227397-1-den@valinux.co.jp?part=1