[PATCH v3 0/7] PCI: endpoint: Remote DMA support via vNTB

[PATCH v3 0/7] PCI: endpoint: Remote DMA support via vNTB

[Koichiro Den]

Hi,

This series lets an RC use endpoint-local DesignWare eDMA read channels
through vNTB. It keeps the export within pci-epf-vntb and ntb_hw_epf
rather than defining a generic remote-DMA ABI or using a separate PCI
DMA function.

On the endpoint, pci-epf-vntb reserves the complete eDMA read direction
and exposes those channels as a group. HDMA is not yet supported,
keeping the series minimal.

The DMA feature is opt-in through the pci-epf-vntb configfs attribute
'dma_bar', which is -1 by default. Without it, no DMA extension is
exported. pci-epf-ntb is unchanged.

Base and dependency
===================

This series is based on next-20260827 (3d83758432b5) plus:

  [PATCH v2 0/3] dmaengine: dw-edma: Prepare channels for remote use
  https://lore.kernel.org/r/20260828163611.2691264-1-den@valinux.co.jp/

The prerequisite is for the dmaengine tree and is not part of
pci/endpoint. linux-next provides a common buildable base for both
series. Once the prerequisite lands, this series can be rebased onto
pci/endpoint.

Best regards,
Koichiro
---
Changes in v3:
  - Rework DMA BAR setup around pci_epf_assign_bar_space() and submaps.
    Allocate backing only for uncovered BAR ranges. This handles the BAR
    backing alignment issue in pci-epf-vntb instead of adding a generic
    pci_epf_alloc_space() workaround. (Sashiko)
    https://lore.kernel.org/r/nvgnu3enktslxuiv7kdapcotoaxlfqhivar4jbf2tqvrk4otz5@exapzemrq37q/
  - Set up a shared DMA/MW BAR only once, preserve MW translations, and
    simplify error paths. (Sashiko)
  - Make dma_bar the explicit opt-in and drop use_dma and automatic BAR
    selection. Split BAR sharing into its own patch, then document the
    final interface at the end of the series instead of updating the
    documentation twice.
  - Validate peer-provided DW eDMA region offsets before MMIO mapping.
    (Sashiko)
  - Include several other minor cleanups and refactorings.

Changes in v2:
  - Rewrite most of v1 and move the dw-edma preparation into the
    companion series. Drop the generic pci-ep-dma API, auxiliary-bus
    child/frontend, test ntb client, and preparatory MW-state
    refactoring.
    See individual patch trailers for provenance.
  - Replace v1's post-registration channel-ID cache with driver-supplied
    static IDs. (Frank)
    https://lore.kernel.org/r/abMcgQOHDD55Yv0e@lizhi-Precision-Tower-5810/
  - Let pci-epf-vntb reserve channels with dma_request_channel() and
    configure interrupt routing with dmaengine_slave_config(). This
    removes the PCI EPC DMA delegation API. (Frank)
    https://lore.kernel.org/r/ao2nHoCwfTEEiFSr@SMW015318/
  - Drop v1's dma_offset and dma_num_chans attributes. use_dma now enables
    export of the complete eDMA read direction as a group.

v2: https://lore.kernel.org/r/20260828170932.2735807-1-den@valinux.co.jp/
v1: https://lore.kernel.org/r/20260312165005.1148676-1-den@valinux.co.jp/

Related PCI DMA EPF v7:
  https://lore.kernel.org/r/20260813063757.3131865-1-den@valinux.co.jp/


Koichiro Den (7):
  PCI: endpoint: Add DMA auxiliary resource metadata
  PCI: dwc: Expose endpoint DMA resources
  PCI: endpoint: pci-epf-vntb: Move epf_ntb_is_bar_used() up
  PCI: endpoint: pci-epf-vntb: Export endpoint DMA channels
  PCI: endpoint: pci-epf-vntb: Allow DMA and MW to share a BAR
  NTB: ntb_hw_epf: Discover vNTB-embedded DMA
  Documentation: PCI: endpoint: Document vNTB DMA export

 Documentation/PCI/endpoint/pci-vntb-howto.rst |  24 +-
 drivers/ntb/hw/epf/ntb_hw_epf.c               | 362 ++++++++-
 .../pci/controller/dwc/pcie-designware-ep.c   | 107 ++-
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 716 ++++++++++++++++--
 include/linux/pci-epc.h                       |  34 +-
 5 files changed, 1166 insertions(+), 77 deletions(-)

-- 
2.51.0

[PATCH v3 1/7] PCI: endpoint: Add DMA auxiliary resource metadata

[Koichiro Den]

Endpoint functions that export controller-local DMA channels need the
controller register and descriptor-memory layout.

Add DMA control and descriptor-memory resource types. The control resource
carries the register layout and channel counts. Each descriptor region is
associated with the static channel ID assigned by the local DMA driver.

Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v3:
  - No changes.

 include/linux/pci-epc.h | 34 +++++++++++++++++++++++++++++++---
 1 file changed, 31 insertions(+), 3 deletions(-)

diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
index f247cf9bcf1a..e0348e7bc534 100644
--- a/include/linux/pci-epc.h
+++ b/include/linux/pci-epc.h
@@ -65,6 +65,8 @@ struct pci_epc_map {
  * enum pci_epc_aux_resource_type - auxiliary resource type identifiers
  * @PCI_EPC_AUX_DOORBELL_MMIO: Doorbell MMIO, that might be outside the DMA
  *                             controller register window
+ * @PCI_EPC_AUX_DMA_CTRL_MMIO: DMA controller MMIO register window
+ * @PCI_EPC_AUX_DMA_DESC_MEM: DMA descriptor memory
  *
  * EPC backends may expose auxiliary blocks (e.g. DMA engines) by mapping their
  * register windows and descriptor memories into BAR space. This enum
@@ -72,13 +74,25 @@ struct pci_epc_map {
  */
 enum pci_epc_aux_resource_type {
 	PCI_EPC_AUX_DOORBELL_MMIO,
+	PCI_EPC_AUX_DMA_CTRL_MMIO,
+	PCI_EPC_AUX_DMA_DESC_MEM,
 };
 
 /**
- * struct pci_epc_aux_resource - a physical auxiliary resource that may be
- *                               exposed for peer use
+ * enum pci_epc_aux_dma_reg_layout - DMA controller register layout
+ * @PCI_EPC_AUX_DMA_REG_LAYOUT_UNKNOWN: unknown or uninitialized layout
+ * @PCI_EPC_AUX_DMA_REG_LAYOUT_DW_EDMA: Synopsys DesignWare eDMA/HDMA layout
+ */
+enum pci_epc_aux_dma_reg_layout {
+	PCI_EPC_AUX_DMA_REG_LAYOUT_UNKNOWN = 0,
+	PCI_EPC_AUX_DMA_REG_LAYOUT_DW_EDMA,
+};
+
+/**
+ * struct pci_epc_aux_resource - an auxiliary resource that may be exposed for
+ *                               peer use
  * @type:       resource type, see enum pci_epc_aux_resource_type
- * @phys_addr:  physical base address of the resource
+ * @phys_addr:  CPU physical base address of an MMIO resource
  * @size:       size of the resource in bytes
  * @bar:        BAR number where this resource is already exposed to the RC
  *              (NO_BAR if not)
@@ -99,6 +113,20 @@ struct pci_epc_aux_resource {
 			int irq; /* IRQ number for the doorbell handler */
 			u32 data; /* write value to ring the doorbell */
 		} db_mmio;
+
+		/* PCI_EPC_AUX_DMA_CTRL_MMIO */
+		struct {
+			enum pci_epc_aux_dma_reg_layout reg_layout;
+			u32 reg_layout_data;
+			u16 ep_to_rc_ch_cnt;
+			u16 rc_to_ep_ch_cnt;
+		} dma_ctrl;
+
+		/* PCI_EPC_AUX_DMA_DESC_MEM */
+		struct {
+			dma_addr_t dma_addr; /* Endpoint-local DMA address */
+			u16 chan_id;
+		} dma_desc;
 	} u;
 };
 
-- 
2.51.0

[PATCH v3 2/7] PCI: dwc: Expose endpoint DMA resources

[Koichiro Den]

Expose the endpoint-integrated eDMA register window and linked-list memory
through the EPC auxiliary resource API. Endpoint functions can then choose
which channels to export and map the required windows.

Associate each linked-list region with the direction-flattened static
channel ID used by the local dw-edma device. Report these resources only
after that device is registered and only for channels with linked-list
memory.

When the register window is already part of a reserved BAR, report its BAR
and offset. Otherwise report its CPU physical address.

While at it, harden the existing doorbell resource query. Report it
only after the local dw-edma device is registered, and reject VF
queries because DWC cannot assign DMA/HDMA registers to VFs.

Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v3:
  - No changes.

 .../pci/controller/dwc/pcie-designware-ep.c   | 107 ++++++++++++++++--
 1 file changed, 99 insertions(+), 8 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index de8ee3db4360..c5e07edb52b1 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -858,6 +858,15 @@ dw_pcie_ep_find_bar_rsvd_region(struct dw_pcie_ep *ep,
 	return NULL;
 }
 
+static int dw_pcie_ep_check_edma_vfunc(u8 vfunc_no)
+{
+	/*
+	 * The DWC endpoint databook says it is not possible to assign the
+	 * DMA/HDMA registers to any Virtual Function.
+	 */
+	return vfunc_no ? -EOPNOTSUPP : 0;
+}
+
 static int
 dw_pcie_ep_get_aux_resources_count(struct pci_epc *epc, u8 func_no,
 				   u8 vfunc_no)
@@ -865,14 +874,30 @@ dw_pcie_ep_get_aux_resources_count(struct pci_epc *epc, u8 func_no,
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct dw_edma_chip *edma = &pci->edma;
+	u16 ll_wr_cnt, ll_rd_cnt;
+	int count = 0;
+	int ret;
 
 	if (!pci->edma_reg_size)
 		return 0;
 
-	if (edma->db_offset == ~0)
+	ret = dw_pcie_ep_check_edma_vfunc(vfunc_no);
+	if (ret)
+		return ret;
+
+	if (!edma->dw)
 		return 0;
 
-	return 1;
+	ll_wr_cnt = edma->ll_wr_cnt;
+	ll_rd_cnt = edma->ll_rd_cnt;
+
+	if (!edma->cfg_non_ll)
+		count += 1 + ll_wr_cnt + ll_rd_cnt;
+
+	if (edma->db_offset != ~0)
+		count++;
+
+	return count;
 }
 
 static int
@@ -885,14 +910,32 @@ dw_pcie_ep_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	const struct pci_epc_bar_rsvd_region *rsvd;
 	struct dw_edma_chip *edma = &pci->edma;
 	enum pci_barno dma_ctrl_bar = NO_BAR;
-	resource_size_t db_offset = edma->db_offset;
+	resource_size_t db_offset;
 	resource_size_t dma_ctrl_bar_offset = 0;
 	resource_size_t dma_reg_size;
-	int count;
+	u16 ll_wr_cnt, ll_rd_cnt;
+	bool has_ll;
+	unsigned int i;
+	int count, ret;
 
-	count = dw_pcie_ep_get_aux_resources_count(epc, func_no, vfunc_no);
-	if (count < 0)
-		return count;
+	if (!pci->edma_reg_size)
+		return 0;
+
+	ret = dw_pcie_ep_check_edma_vfunc(vfunc_no);
+	if (ret)
+		return ret;
+
+	if (!edma->dw)
+		return 0;
+
+	ll_wr_cnt = edma->ll_wr_cnt;
+	ll_rd_cnt = edma->ll_rd_cnt;
+	db_offset = edma->db_offset;
+	has_ll = !edma->cfg_non_ll;
+
+	count = db_offset != ~0;
+	if (has_ll)
+		count += 1 + ll_wr_cnt + ll_rd_cnt;
 
 	if (num_resources < count)
 		return -ENOSPC;
@@ -909,6 +952,54 @@ dw_pcie_ep_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	if (rsvd && rsvd->size < dma_reg_size)
 		dma_reg_size = rsvd->size;
 
+	count = 0;
+	if (has_ll) {
+		resources[count++] = (struct pci_epc_aux_resource) {
+			.type = PCI_EPC_AUX_DMA_CTRL_MMIO,
+			.phys_addr = pci->edma_reg_phys,
+			.size = dma_reg_size,
+			.bar = dma_ctrl_bar,
+			.bar_offset = dma_ctrl_bar_offset,
+			.u.dma_ctrl = {
+				.reg_layout = PCI_EPC_AUX_DMA_REG_LAYOUT_DW_EDMA,
+				.reg_layout_data = edma->mf,
+				.ep_to_rc_ch_cnt = ll_wr_cnt,
+				.rc_to_ep_ch_cnt = ll_rd_cnt,
+			},
+		};
+
+		for (i = 0; i < ll_wr_cnt; i++) {
+			struct dw_edma_region *ll = &edma->ll_region_wr[i];
+
+			resources[count++] = (struct pci_epc_aux_resource) {
+				.type = PCI_EPC_AUX_DMA_DESC_MEM,
+				.size = ll->sz,
+				.bar = NO_BAR,
+				.u.dma_desc = {
+					.dma_addr = ll->paddr,
+					.chan_id = i,
+				},
+			};
+		}
+
+		for (i = 0; i < ll_rd_cnt; i++) {
+			struct dw_edma_region *ll = &edma->ll_region_rd[i];
+
+			resources[count++] = (struct pci_epc_aux_resource) {
+				.type = PCI_EPC_AUX_DMA_DESC_MEM,
+				.size = ll->sz,
+				.bar = NO_BAR,
+				.u.dma_desc = {
+					.dma_addr = ll->paddr,
+					.chan_id = ll_wr_cnt + i,
+				},
+			};
+		}
+	}
+
+	if (db_offset == ~0)
+		return 0;
+
 	/*
 	 * For interrupt-emulation doorbells, report a standalone resource
 	 * instead of bundling it into the DMA controller MMIO resource.
@@ -917,7 +1008,7 @@ dw_pcie_ep_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 				  sizeof(u32), dma_reg_size))
 		return -EINVAL;
 
-	resources[0] = (struct pci_epc_aux_resource) {
+	resources[count] = (struct pci_epc_aux_resource) {
 		.type = PCI_EPC_AUX_DOORBELL_MMIO,
 		.phys_addr = pci->edma_reg_phys + db_offset,
 		.size = sizeof(u32),
-- 
2.51.0

[PATCH v3 3/7] PCI: endpoint: pci-epf-vntb: Move epf_ntb_is_bar_used() up

[Koichiro Den]

Move epf_ntb_is_bar_used() up so later BAR selection code can call it
without a forward declaration. No functional change.

Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v3:
  - No changes.

 drivers/pci/endpoint/functions/pci-epf-vntb.c | 40 +++++++++----------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index c3caec927d74..d12d134ce553 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -199,6 +199,26 @@ static int epf_ntb_link_up(struct epf_ntb *ntb, bool link_up)
 	return 0;
 }
 
+/**
+ * epf_ntb_is_bar_used() - Check if a bar is used in the ntb configuration
+ * @ntb: NTB device that facilitates communication between HOST and VHOST
+ * @barno: Checked bar number
+ *
+ * Returns: true if used, false if free.
+ */
+static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
+				enum pci_barno barno)
+{
+	int i;
+
+	for (i = 0; i < VNTB_BAR_NUM; i++) {
+		if (ntb->epf_ntb_bar[i] == barno)
+			return true;
+	}
+
+	return false;
+}
+
 /**
  * epf_ntb_configure_mw() - Configure the Outbound Address Space for VHOST
  *   to access the memory window of HOST
@@ -830,26 +850,6 @@ static void epf_ntb_mw_bar_clear(struct epf_ntb *ntb, int num_mws)
 	}
 }
 
-/**
- * epf_ntb_is_bar_used() - Check if a bar is used in the ntb configuration
- * @ntb: NTB device that facilitates communication between HOST and VHOST
- * @barno: Checked bar number
- *
- * Returns: true if used, false if free.
- */
-static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
-				enum pci_barno barno)
-{
-	int i;
-
-	for (i = 0; i < VNTB_BAR_NUM; i++) {
-		if (ntb->epf_ntb_bar[i] == barno)
-			return true;
-	}
-
-	return false;
-}
-
 /**
  * epf_ntb_find_bar() - Assign BAR number when no configuration is provided
  * @ntb: NTB device that facilitates communication between HOST and VHOST
-- 
2.51.0

[PATCH v3 4/7] PCI: endpoint: pci-epf-vntb: Export endpoint DMA channels

[Koichiro Den]

An RC may use endpoint-local DMA read channels to transfer data directly
to an endpoint DMA address once both sides agree to use them. Quiescing
an unrolled eDMA channel disables its whole direction, so reserve the
complete read direction and route its interrupts to the RC when dma_bar
is configured.

Describe the controller and per-channel descriptor memory in a private
control-region extension.

Add a dma_bar configfs attribute. An explicit BAR selection enables DMA
export; leaving it unassigned keeps the feature disabled.

Keep resources already assigned to a BAR in place, and map the rest
through the selected dma_bar.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v3:
  - Use dma_bar as the opt-in and drop use_dma and automatic BAR selection.
  - Split DMA/MW BAR sharing and the documentation into separate patches.
  - Avoid mixing cleanup helpers with goto-based error paths. (Sashiko)
  - Use pci_epf_assign_bar_space() and back only uncovered BAR ranges.
    (Sashiko)

 drivers/pci/endpoint/functions/pci-epf-vntb.c | 575 +++++++++++++++++-
 1 file changed, 567 insertions(+), 8 deletions(-)

diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index d12d134ce553..1842493c33b0 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -39,8 +39,12 @@
 #include <linux/atomic.h>
 #include <linux/bitops.h>
 #include <linux/delay.h>
+#include <linux/dma/edma.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
 #include <linux/io.h>
 #include <linux/module.h>
+#include <linux/overflow.h>
 #include <linux/slab.h>
 
 #include <linux/pci-ep-msi.h>
@@ -56,6 +60,8 @@ static struct workqueue_struct *kpcintb_workqueue;
 #define COMMAND_TEARDOWN_MW		4
 #define COMMAND_LINK_UP			5
 #define COMMAND_LINK_DOWN		6
+#define COMMAND_CONFIGURE_DMA		7
+#define COMMAND_TEARDOWN_DMA		8
 
 #define COMMAND_STATUS_OK		1
 #define COMMAND_STATUS_ERROR		2
@@ -69,6 +75,10 @@ static struct workqueue_struct *kpcintb_workqueue;
 #define MSIX_ENABLE			BIT(16)
 #define MAX_MW				4
 
+#define EPF_NTB_DMA_MAGIC		0x414d444e /* "NDMA": NTB DMA */
+#define EPF_NTB_DMA_REVISION		1
+#define EPF_NTB_DMA_TYPE_DW_EDMA	1
+
 /* Limit per-work execution to avoid monopolizing kworker on doorbell storms. */
 #define VNTB_PEER_DB_WORK_BUDGET	5
 
@@ -79,6 +89,7 @@ enum epf_ntb_bar {
 	BAR_MW2,
 	BAR_MW3,
 	BAR_MW4,
+	BAR_DMA,
 	VNTB_BAR_NUM,
 };
 
@@ -91,6 +102,30 @@ enum epf_irq_slot {
 #define MIN_DB_COUNT			(EPF_IRQ_DB_START + 1)
 #define MAX_DB_COUNT			32
 
+/* Private wire extension consumed by ntb_hw_epf. */
+struct epf_ntb_dma_region_ctrl {
+	u32 bar;
+	u32 offset;
+	u32 size;
+} __packed;
+
+struct epf_ntb_dma_chan_ctrl {
+	struct epf_ntb_dma_region_ctrl desc;
+	u32 desc_addr_lo;
+	u32 desc_addr_hi;
+} __packed;
+
+struct epf_ntb_dma_ctrl {
+	u32 magic;
+	u16 revision;
+	u16 length;
+	u32 type;
+	/* BAR range occupied by resources without a fixed BAR assignment. */
+	struct epf_ntb_dma_region_ctrl submap;
+	struct epf_ntb_dma_region_ctrl reg;
+	struct epf_ntb_dma_chan_ctrl chan[EDMA_MAX_RD_CH];
+} __packed;
+
 /*
  * +--------------------------------------------------+ Base
  * |                                                  |
@@ -129,8 +164,21 @@ struct epf_ntb_ctrl {
 	u32 db_entry_size;
 	u32 db_data[MAX_DB_COUNT];
 	u32 db_offset[MAX_DB_COUNT];
+	struct epf_ntb_dma_ctrl dma;
 } __packed;
 
+struct epf_ntb_dma {
+	struct epf_ntb_dma_ctrl ctrl;
+	struct dma_chan *dchan[EDMA_MAX_RD_CH];
+	void *bar_scratch;
+	dma_addr_t bar_scratch_phys;
+	size_t bar_scratch_size;
+	struct pci_epf_bar_submap submap[EDMA_MAX_RD_CH + 2];
+	struct pci_epf_bar_submap *reg_submap;
+	unsigned int num_submap;
+	u16 rd_ch_cnt;
+};
+
 struct epf_ntb {
 	struct ntb_dev ntb;
 	struct pci_epf *epf;
@@ -159,6 +207,7 @@ struct epf_ntb {
 	enum pci_barno epf_ntb_bar[VNTB_BAR_NUM];
 
 	struct epf_ntb_ctrl *reg;
+	struct epf_ntb_dma *dma;
 
 	u32 *epf_db;
 
@@ -211,7 +260,8 @@ static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
 {
 	int i;
 
-	for (i = 0; i < VNTB_BAR_NUM; i++) {
+	/* BAR_DMA is checked separately because it may share an MW BAR. */
+	for (i = 0; i < BAR_DMA; i++) {
 		if (ntb->epf_ntb_bar[i] == barno)
 			return true;
 	}
@@ -219,6 +269,404 @@ static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
 	return false;
 }
 
+static int epf_ntb_dma_validate_bar(struct epf_ntb *ntb,
+				    const struct pci_epc_features *features)
+{
+	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
+
+	if (epf_ntb_is_bar_used(ntb, barno) ||
+	    pci_epc_get_next_free_bar(features, barno) != barno)
+		return -EINVAL;
+
+	return 0;
+}
+
+struct epf_ntb_dma_filter {
+	struct device *dev;
+	int chan_id;
+};
+
+static bool epf_ntb_dma_filter(struct dma_chan *chan, void *data)
+{
+	struct epf_ntb_dma_filter *filter = data;
+
+	return chan->device->dev == filter->dev &&
+	       chan->chan_id == filter->chan_id;
+}
+
+static int epf_ntb_dma_add_region(struct epf_ntb_dma *dma,
+				  const struct pci_epc_aux_resource *resource,
+				  dma_addr_t target_addr,
+				  enum pci_barno barno, size_t align, u32 *next,
+				  struct epf_ntb_dma_region_ctrl *region)
+{
+	struct pci_epf_bar_submap *submap;
+	resource_size_t delta, map_size, size;
+	dma_addr_t base;
+
+	if (!resource->size || resource->size > U32_MAX)
+		return -EINVAL;
+
+	region->size = resource->size;
+	if (resource->bar != NO_BAR) {
+		if (resource->bar < BAR_0 || resource->bar > BAR_5 ||
+		    resource->bar_offset > U32_MAX)
+			return -EINVAL;
+
+		region->bar = resource->bar;
+		region->offset = resource->bar_offset;
+		return 0;
+	}
+	submap = &dma->submap[dma->num_submap];
+
+	/*
+	 * Meet the EPC alignment requirement by mapping an aligned superset
+	 * and advertising the resource after any leading padding.
+	 */
+	base = ALIGN_DOWN(target_addr, align);
+	delta = target_addr - base;
+	if (check_add_overflow(delta, resource->size, &size))
+		return -EOVERFLOW;
+	map_size = ALIGN(size, align);
+	if (map_size < size || map_size > U32_MAX - *next)
+		return -EOVERFLOW;
+
+	submap->phys_addr = base;
+	submap->size = map_size;
+	region->bar = barno;
+	region->offset = *next + delta;
+	*next += map_size;
+	dma->num_submap++;
+
+	return 0;
+}
+
+/* DW eDMA */
+
+static int epf_ntb_dw_edma_claim(struct device *dev, int chan_id,
+				 struct dma_chan **dchan)
+{
+	enum dw_edma_ch_irq_mode mode = DW_EDMA_CH_IRQ_REMOTE;
+	struct epf_ntb_dma_filter filter = {
+		.dev = dev,
+		.chan_id = chan_id,
+	};
+	struct dma_slave_config config = {
+		.peripheral_config = &mode,
+		.peripheral_size = sizeof(mode),
+	};
+	dma_cap_mask_t mask;
+	struct dma_chan *chan;
+	int ret;
+
+	dma_cap_zero(mask);
+	dma_cap_set(DMA_SLAVE, mask);
+	chan = dma_request_channel(mask, epf_ntb_dma_filter, &filter);
+	if (!chan)
+		return -EBUSY;
+
+	ret = dmaengine_slave_config(chan, &config);
+	if (ret) {
+		dma_release_channel(chan);
+		return ret;
+	}
+
+	*dchan = chan;
+
+	return 0;
+}
+
+static void epf_ntb_dw_edma_release_channels(struct epf_ntb *ntb,
+					     struct epf_ntb_dma *dma,
+					     bool quiesce)
+{
+	unsigned int i;
+	int ret;
+
+	if (quiesce) {
+		/*
+		 * RC programming has stopped and this EPF owns the complete read
+		 * direction, so one termination quiesces the direction.
+		 */
+		ret = dmaengine_terminate_sync(dma->dchan[0]);
+		if (ret)
+			dev_warn(&ntb->epf->dev,
+				 "failed to terminate remote DMA: %d\n", ret);
+	}
+
+	for (i = 0; i < dma->rd_ch_cnt; i++) {
+		if (!dma->dchan[i])
+			continue;
+
+		dma_release_channel(dma->dchan[i]);
+	}
+}
+
+static const struct pci_epc_aux_resource *
+epf_ntb_dw_edma_find_desc(const struct pci_epc_aux_resource *resources,
+			  unsigned int count, u16 chan_id)
+{
+	unsigned int i;
+
+	for (i = 0; i < count; i++)
+		if (resources[i].type == PCI_EPC_AUX_DMA_DESC_MEM &&
+		    resources[i].u.dma_desc.chan_id == chan_id)
+			return &resources[i];
+
+	return NULL;
+}
+
+static int
+epf_ntb_dw_edma_collect(struct epf_ntb *ntb,
+			struct epf_ntb_dma *dma,
+			const struct pci_epc_aux_resource *ctrl,
+			const struct pci_epc_aux_resource *resources,
+			unsigned int count)
+{
+	const struct pci_epc_features *features;
+	const struct pci_epc_aux_resource *desc[EDMA_MAX_RD_CH];
+	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
+	struct device *dma_dev;
+	bool needs_submap;
+	unsigned int i;
+	size_t align;
+	u32 next = 0;
+	int ret;
+
+	if (ctrl->u.dma_ctrl.reg_layout_data != EDMA_MF_EDMA_UNROLL)
+		return -EOPNOTSUPP;
+	if (ctrl->u.dma_ctrl.ep_to_rc_ch_cnt > EDMA_MAX_WR_CH ||
+	    !ctrl->u.dma_ctrl.rc_to_ep_ch_cnt ||
+	    ctrl->u.dma_ctrl.rc_to_ep_ch_cnt > EDMA_MAX_RD_CH)
+		return -EINVAL;
+
+	dma->rd_ch_cnt = ctrl->u.dma_ctrl.rc_to_ep_ch_cnt;
+
+	features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no,
+					ntb->epf->vfunc_no);
+	if (!features)
+		return -EOPNOTSUPP;
+
+	align = features->align ?: 1;
+	if (!is_power_of_2(align))
+		return -EINVAL;
+
+	needs_submap = ctrl->bar == NO_BAR;
+	/* DW eDMA static IDs place read channels after all write channels. */
+	for (i = 0; i < dma->rd_ch_cnt; i++) {
+		u16 chan_id = ctrl->u.dma_ctrl.ep_to_rc_ch_cnt + i;
+
+		desc[i] = epf_ntb_dw_edma_find_desc(resources, count, chan_id);
+		if (!desc[i])
+			return -EINVAL;
+		needs_submap |= desc[i]->bar == NO_BAR;
+	}
+	if (needs_submap) {
+		if (!features->subrange_mapping ||
+		    !features->dynamic_inbound_mapping)
+			return -EOPNOTSUPP;
+		ret = epf_ntb_dma_validate_bar(ntb, features);
+		if (ret)
+			return ret;
+	}
+
+	dma->ctrl.magic = EPF_NTB_DMA_MAGIC;
+	dma->ctrl.revision = EPF_NTB_DMA_REVISION;
+	dma->ctrl.type = EPF_NTB_DMA_TYPE_DW_EDMA;
+	dma->ctrl.submap.bar = U32_MAX;
+	dma->ctrl.length = offsetof(struct epf_ntb_dma_ctrl,
+				    chan[dma->rd_ch_cnt]);
+
+	if (ctrl->bar == NO_BAR)
+		dma->reg_submap = &dma->submap[dma->num_submap];
+	ret = epf_ntb_dma_add_region(dma, ctrl, ctrl->phys_addr,
+				     barno, align, &next, &dma->ctrl.reg);
+	if (ret)
+		return ret;
+	for (i = 0; i < dma->rd_ch_cnt; i++) {
+		struct epf_ntb_dma_chan_ctrl *chan = &dma->ctrl.chan[i];
+		dma_addr_t dma_addr = desc[i]->u.dma_desc.dma_addr;
+
+		ret = epf_ntb_dma_add_region(dma, desc[i], dma_addr, barno,
+					     align, &next,
+					     &chan->desc);
+		if (ret)
+			return ret;
+		chan->desc_addr_lo = lower_32_bits(dma_addr);
+		chan->desc_addr_hi = upper_32_bits(dma_addr);
+	}
+	if (dma->num_submap) {
+		dma->ctrl.submap.bar = barno;
+		dma->ctrl.submap.size = next;
+	}
+
+	dma_dev = ntb->epf->epc->dev.parent;
+	for (i = 0; i < dma->rd_ch_cnt; i++) {
+		u16 chan_id = ctrl->u.dma_ctrl.ep_to_rc_ch_cnt + i;
+
+		ret = epf_ntb_dw_edma_claim(dma_dev, chan_id, &dma->dchan[i]);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/* Common endpoint DMA */
+
+static void epf_ntb_dma_release_channels(struct epf_ntb *ntb,
+					 struct epf_ntb_dma *dma,
+					 bool quiesce)
+{
+	switch (dma->ctrl.type) {
+	case EPF_NTB_DMA_TYPE_DW_EDMA:
+		epf_ntb_dw_edma_release_channels(ntb, dma, quiesce);
+		break;
+	}
+}
+
+static int epf_ntb_dma_collect(struct epf_ntb *ntb)
+{
+	const struct pci_epc_aux_resource *ctrl = NULL;
+	struct device *dma_dev;
+	dma_addr_t dma_addr;
+	unsigned int i;
+	int count, ret;
+
+	if (ntb->epf_ntb_bar[BAR_DMA] == NO_BAR)
+		return 0;
+
+	count = pci_epc_get_aux_resources_count(ntb->epf->epc,
+						ntb->epf->func_no,
+						ntb->epf->vfunc_no);
+	if (count <= 0)
+		return count ?: -ENODEV;
+
+	struct pci_epc_aux_resource *resources __free(kfree) =
+		kcalloc(count, sizeof(*resources), GFP_KERNEL);
+	if (!resources)
+		return -ENOMEM;
+
+	ret = pci_epc_get_aux_resources(ntb->epf->epc, ntb->epf->func_no,
+					ntb->epf->vfunc_no, resources, count);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < count; i++) {
+		if (resources[i].type != PCI_EPC_AUX_DMA_CTRL_MMIO)
+			continue;
+		if (ctrl)
+			return -EINVAL;
+		ctrl = &resources[i];
+	}
+	if (!ctrl)
+		return -ENODEV;
+
+	struct epf_ntb_dma *dma __free(kfree) =
+		kzalloc(sizeof(*dma), GFP_KERNEL);
+	if (!dma)
+		return -ENOMEM;
+
+	switch (ctrl->u.dma_ctrl.reg_layout) {
+	case PCI_EPC_AUX_DMA_REG_LAYOUT_DW_EDMA:
+		ret = epf_ntb_dw_edma_collect(ntb, dma, ctrl, resources, count);
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+	if (ret) {
+		epf_ntb_dma_release_channels(ntb, dma, false);
+		return ret;
+	}
+
+	/*
+	 * CTRL_MMIO carries a CPU physical address, while DMA_DESC_MEM already
+	 * carries an endpoint DMA address. Convert only the former.
+	 */
+	if (dma->reg_submap) {
+		dma_dev = ntb->epf->epc->dev.parent;
+		dma_addr = dma_map_resource(dma_dev,
+					    dma->reg_submap->phys_addr,
+					    dma->reg_submap->size,
+					    DMA_BIDIRECTIONAL, 0);
+		if (dma_mapping_error(dma_dev, dma_addr)) {
+			epf_ntb_dma_release_channels(ntb, dma, false);
+			return -EIO;
+		}
+		dma->reg_submap->phys_addr = dma_addr;
+	}
+
+	ntb->dma = no_free_ptr(dma);
+
+	return 0;
+}
+
+static void epf_ntb_dma_release(struct epf_ntb *ntb, bool quiesce)
+{
+	struct epf_ntb_dma *dma = ntb->dma;
+	struct device *dev;
+
+	if (!dma)
+		return;
+
+	epf_ntb_dma_release_channels(ntb, dma, quiesce);
+	dev = ntb->epf->epc->dev.parent;
+	if (dma->reg_submap)
+		dma_unmap_resource(dev, dma->reg_submap->phys_addr,
+				   dma->reg_submap->size, DMA_BIDIRECTIONAL, 0);
+	if (dma->bar_scratch)
+		dma_free_coherent(dev, dma->bar_scratch_size,
+				  dma->bar_scratch, dma->bar_scratch_phys);
+	kfree(dma);
+	ntb->dma = NULL;
+}
+
+static int epf_ntb_dma_set_bar(struct epf_ntb *ntb, bool active)
+{
+	struct pci_epf_bar_submap *old_submap;
+	struct epf_ntb_dma *dma = ntb->dma;
+	struct pci_epf_bar *bar;
+	unsigned int old_num_submap;
+	int restore, ret;
+
+	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
+	old_submap = bar->submap;
+	old_num_submap = bar->num_submap;
+	bar->submap = active ? dma->submap : NULL;
+	bar->num_submap = active ? dma->num_submap : 0;
+
+	ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
+			      ntb->epf->vfunc_no, bar);
+	if (!ret)
+		return 0;
+
+	/* A failed dynamic update may have already removed the old mapping. */
+	bar->submap = old_submap;
+	bar->num_submap = old_num_submap;
+	restore = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
+				  ntb->epf->vfunc_no, bar);
+	if (restore)
+		dev_warn(&ntb->epf->dev,
+			 "failed to restore DMA BAR mapping: %d\n", restore);
+
+	return ret;
+}
+
+static int epf_ntb_dma_set_active(struct epf_ntb *ntb, bool active)
+{
+	struct epf_ntb_dma *dma = ntb->dma;
+	struct pci_epf_bar *bar;
+
+	if (!dma || !dma->num_submap)
+		return 0;
+
+	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
+	if (active == !!bar->num_submap)
+		return 0;
+
+	return epf_ntb_dma_set_bar(ntb, active);
+}
+
 /**
  * epf_ntb_configure_mw() - Configure the Outbound Address Space for VHOST
  *   to access the memory window of HOST
@@ -339,6 +787,13 @@ static void epf_ntb_cmd_handler(struct work_struct *work)
 		epf_ntb_teardown_mw(ntb, argument);
 		ctrl->command_status = COMMAND_STATUS_OK;
 		break;
+	case COMMAND_CONFIGURE_DMA:
+	case COMMAND_TEARDOWN_DMA:
+		ret = epf_ntb_dma_set_active(ntb,
+					     command == COMMAND_CONFIGURE_DMA);
+		ctrl->command_status = ret ? COMMAND_STATUS_ERROR :
+					     COMMAND_STATUS_OK;
+		break;
 	case COMMAND_LINK_UP:
 		ntb->linkup = true;
 		ret = epf_ntb_link_up(ntb, true);
@@ -459,9 +914,8 @@ static void epf_ntb_config_spad_bar_free(struct epf_ntb *ntb)
  *   region
  * @ntb: NTB device that facilitates communication between HOST and VHOST
  *
- * Allocate the Local Memory mentioned in the above diagram. The size of
- * CONFIG REGION is sizeof(struct epf_ntb_ctrl) and size of SCRATCHPAD REGION
- * is obtained from "spad-count" configfs entry.
+ * Allocate the control and scratchpad regions, omitting the optional DMA
+ * extension when no channels are exported.
  *
  * Returns: Zero for success, or an error code in case of failure
  */
@@ -481,7 +935,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
 	barno = ntb->epf_ntb_bar[BAR_CONFIG];
 	spad_count = ntb->spad_count;
 
-	ctrl_size = ALIGN(sizeof(struct epf_ntb_ctrl), sizeof(u32));
+	ctrl_size = ntb->dma ? sizeof(struct epf_ntb_ctrl) :
+			       offsetof(struct epf_ntb_ctrl, dma);
+	ctrl_size = ALIGN(ctrl_size, sizeof(u32));
 	spad_size = 2 * spad_count * sizeof(u32);
 
 	base = pci_epf_alloc_space(epf, ctrl_size + spad_size,
@@ -507,6 +963,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
 		ntb->reg->db_offset[i] = 0;
 	}
 
+	if (ntb->dma)
+		ctrl->dma = ntb->dma->ctrl;
+
 	return 0;
 }
 
@@ -738,6 +1197,83 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
 
 static void epf_ntb_mw_bar_clear(struct epf_ntb *ntb, int num_mws);
 
+static int epf_ntb_dma_bar_init(struct epf_ntb *ntb)
+{
+	const struct pci_epc_features *features;
+	struct epf_ntb_dma *dma = ntb->dma;
+	struct device *dev = ntb->epf->epc->dev.parent;
+	struct pci_epf_bar *bar;
+	enum pci_barno barno;
+	size_t backing_size;
+	u32 mapped_size;
+	int ret;
+
+	features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no,
+					ntb->epf->vfunc_no);
+	if (!features)
+		return -EOPNOTSUPP;
+
+	barno = ntb->epf_ntb_bar[BAR_DMA];
+	mapped_size = dma->ctrl.submap.size;
+	/*
+	 * Submaps cannot be installed until the host assigns the BAR address.
+	 * Use address 0 for the temporary BAR Match Mode mapping, as is done
+	 * for regular vNTB MW BARs.
+	 */
+	ret = pci_epf_assign_bar_space(ntb->epf, mapped_size, barno, features,
+				       PRIMARY_INTERFACE, 0);
+	if (ret)
+		return ret;
+
+	bar = &ntb->epf->bar[barno];
+	if (bar->size > U32_MAX)
+		return -EOVERFLOW;
+
+	backing_size = bar->size - mapped_size;
+	if (backing_size) {
+		/* Back the BAR tail added by the power-of-two size rounding. */
+		dma->bar_scratch = dma_alloc_coherent(dev, backing_size,
+						      &dma->bar_scratch_phys,
+						      GFP_KERNEL);
+		if (!dma->bar_scratch)
+			return -ENOMEM;
+		dma->bar_scratch_size = backing_size;
+		if (!IS_ALIGNED(dma->bar_scratch_phys, features->align ?: 1))
+			return -EINVAL;
+
+		dma->submap[dma->num_submap++] = (struct pci_epf_bar_submap) {
+			.phys_addr = dma->bar_scratch_phys,
+			.size = backing_size,
+		};
+	}
+
+	return pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
+			       ntb->epf->vfunc_no, bar);
+}
+
+static void epf_ntb_dma_bar_clear(struct epf_ntb *ntb)
+{
+	struct epf_ntb_dma *dma = ntb->dma;
+	struct pci_epf_bar *bar;
+	enum pci_barno barno;
+
+	if (!dma || !dma->num_submap)
+		return;
+
+	barno = ntb->epf_ntb_bar[BAR_DMA];
+	bar = &ntb->epf->bar[barno];
+	pci_epc_clear_bar(ntb->epf->epc, ntb->epf->func_no,
+			  ntb->epf->vfunc_no, bar);
+	bar->submap = NULL;
+	bar->num_submap = 0;
+	bar->phys_addr = 0;
+	bar->addr = NULL;
+	bar->size = 0;
+	bar->mem_size = 0;
+	bar->barno = 0;
+	bar->flags = 0;
+}
+
 /**
  * epf_ntb_db_bar_clear() - Clear doorbell BAR and free memory
  *   allocated in peer's outbound address space
@@ -877,7 +1413,8 @@ static int epf_ntb_find_bar(struct epf_ntb *ntb,
 		 * Verify if the BAR found is not already assigned
 		 * through the provided configuration
 		 */
-		if (!epf_ntb_is_bar_used(ntb, barno))
+		if (ntb->epf_ntb_bar[BAR_DMA] != barno &&
+		    !epf_ntb_is_bar_used(ntb, barno))
 			ntb->epf_ntb_bar[bar] = barno;
 
 		barno += 1;
@@ -976,11 +1513,19 @@ static int epf_ntb_epc_init(struct epf_ntb *ntb)
 		goto err_mw_bar_init;
 	}
 
+	if (ntb->dma && ntb->dma->num_submap) {
+		ret = epf_ntb_dma_bar_init(ntb);
+		if (ret) {
+			dev_err(dev, "DMA BAR init failed\n");
+			goto err_dma_bar_init;
+		}
+	}
+
 	if (vfunc_no <= 1) {
 		ret = pci_epc_write_header(epc, func_no, vfunc_no, epf->header);
 		if (ret) {
 			dev_err(dev, "Configuration header write failed\n");
-			goto err_write_header;
+			goto err_dma_bar_init;
 		}
 	}
 
@@ -992,7 +1537,8 @@ static int epf_ntb_epc_init(struct epf_ntb *ntb)
 
 	return 0;
 
-err_write_header:
+err_dma_bar_init:
+	epf_ntb_dma_bar_clear(ntb);
 	epf_ntb_mw_bar_clear(ntb, ntb->num_mws);
 err_mw_bar_init:
 	epf_ntb_db_bar_clear(ntb);
@@ -1015,6 +1561,7 @@ static void epf_ntb_epc_cleanup(struct epf_ntb *ntb)
 	disable_delayed_work_sync(&ntb->cmd_handler);
 	disable_work_sync(&ntb->peer_db_work);
 	atomic64_set(&ntb->peer_db_pending, 0);
+	epf_ntb_dma_bar_clear(ntb);
 	epf_ntb_mw_bar_clear(ntb, ntb->num_mws);
 	epf_ntb_db_bar_clear(ntb);
 	epf_ntb_config_sspad_bar_clear(ntb);
@@ -1221,10 +1768,13 @@ EPF_NTB_BAR_R(mw3_bar, BAR_MW3)
 EPF_NTB_BAR_W(mw3_bar, BAR_MW3)
 EPF_NTB_BAR_R(mw4_bar, BAR_MW4)
 EPF_NTB_BAR_W(mw4_bar, BAR_MW4)
+EPF_NTB_BAR_R(dma_bar, BAR_DMA)
+EPF_NTB_BAR_W(dma_bar, BAR_DMA)
 
 CONFIGFS_ATTR(epf_ntb_, spad_count);
 CONFIGFS_ATTR(epf_ntb_, db_count);
 CONFIGFS_ATTR(epf_ntb_, num_mws);
+CONFIGFS_ATTR(epf_ntb_, dma_bar);
 CONFIGFS_ATTR(epf_ntb_, mw1);
 CONFIGFS_ATTR(epf_ntb_, mw2);
 CONFIGFS_ATTR(epf_ntb_, mw3);
@@ -1243,6 +1793,7 @@ static struct configfs_attribute *epf_ntb_attrs[] = {
 	&epf_ntb_attr_spad_count,
 	&epf_ntb_attr_db_count,
 	&epf_ntb_attr_num_mws,
+	&epf_ntb_attr_dma_bar,
 	&epf_ntb_attr_mw1,
 	&epf_ntb_attr_mw2,
 	&epf_ntb_attr_mw3,
@@ -1743,6 +2294,12 @@ static int epf_ntb_bind(struct pci_epf *epf)
 		return ret;
 	}
 
+	ret = epf_ntb_dma_collect(ntb);
+	if (ret) {
+		dev_err(dev, "Failed to prepare NTB DMA export\n");
+		return ret;
+	}
+
 	ret = epf_ntb_config_spad_bar_alloc(ntb);
 	if (ret) {
 		dev_err(dev, "Failed to allocate BAR memory\n");
@@ -1779,6 +2336,7 @@ static int epf_ntb_bind(struct pci_epf *epf)
 	epf_ntb_epc_cleanup(ntb);
 err_bar_alloc:
 	epf_ntb_config_spad_bar_free(ntb);
+	epf_ntb_dma_release(ntb, false);
 
 	return ret;
 }
@@ -1795,6 +2353,7 @@ static void epf_ntb_unbind(struct pci_epf *epf)
 
 	epf_ntb_epc_cleanup(ntb);
 	epf_ntb_config_spad_bar_free(ntb);
+	epf_ntb_dma_release(ntb, true);
 
 	pci_unregister_driver(&vntb_pci_driver);
 }
-- 
2.51.0

[PATCH v3 5/7] PCI: endpoint: pci-epf-vntb: Allow DMA and MW to share a BAR

[Koichiro Den]

Some endpoint configurations have no spare BAR for DMA resources. Allow
dma_bar to select an MW BAR and place the DMA ranges after that MW.

BAR sharing requires ntb_hw_epf to parse the DMA extension, since older
versions treat the whole BAR as an MW.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v3:
  - New patch split from the DMA export patch.
  - Set up a shared DMA BAR only once per bind. (Sashiko)
  - Preserve DMA submaps when updating a shared MW translation. (Sashiko)

 drivers/pci/endpoint/functions/pci-epf-vntb.c | 133 +++++++++++++-----
 1 file changed, 94 insertions(+), 39 deletions(-)

diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 1842493c33b0..aa659ec62d93 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -44,6 +44,7 @@
 #include <linux/dmaengine.h>
 #include <linux/io.h>
 #include <linux/module.h>
+#include <linux/mutex.h>
 #include <linux/overflow.h>
 #include <linux/slab.h>
 
@@ -168,12 +169,13 @@ struct epf_ntb_ctrl {
 } __packed;
 
 struct epf_ntb_dma {
+	struct mutex lock; /* Serialize submap toggles and MW translation updates */
 	struct epf_ntb_dma_ctrl ctrl;
 	struct dma_chan *dchan[EDMA_MAX_RD_CH];
 	void *bar_scratch;
 	dma_addr_t bar_scratch_phys;
 	size_t bar_scratch_size;
-	struct pci_epf_bar_submap submap[EDMA_MAX_RD_CH + 2];
+	struct pci_epf_bar_submap submap[EDMA_MAX_RD_CH + 3];
 	struct pci_epf_bar_submap *reg_submap;
 	unsigned int num_submap;
 	u16 rd_ch_cnt;
@@ -269,18 +271,25 @@ static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
 	return false;
 }
 
-static int epf_ntb_dma_validate_bar(struct epf_ntb *ntb,
-				    const struct pci_epc_features *features)
+static u64 epf_ntb_dma_bar_offset(struct epf_ntb *ntb,
+				  enum pci_barno barno)
 {
-	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
+	unsigned int i;
 
-	if (epf_ntb_is_bar_used(ntb, barno) ||
-	    pci_epc_get_next_free_bar(features, barno) != barno)
-		return -EINVAL;
+	for (i = 0; i < ntb->num_mws; i++)
+		if (ntb->epf_ntb_bar[BAR_MW1 + i] == barno)
+			return ntb->mws_size[i];
 
 	return 0;
 }
 
+static bool epf_ntb_dma_shares_bar(struct epf_ntb *ntb,
+				   enum pci_barno barno)
+{
+	return ntb->dma && ntb->dma->num_submap &&
+	       ntb->epf_ntb_bar[BAR_DMA] == barno;
+}
+
 struct epf_ntb_dma_filter {
 	struct device *dev;
 	int chan_id;
@@ -428,6 +437,7 @@ epf_ntb_dw_edma_collect(struct epf_ntb *ntb,
 	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
 	struct device *dma_dev;
 	bool needs_submap;
+	u64 offset;
 	unsigned int i;
 	size_t align;
 	u32 next = 0;
@@ -465,9 +475,24 @@ epf_ntb_dw_edma_collect(struct epf_ntb *ntb,
 		if (!features->subrange_mapping ||
 		    !features->dynamic_inbound_mapping)
 			return -EOPNOTSUPP;
-		ret = epf_ntb_dma_validate_bar(ntb, features);
-		if (ret)
-			return ret;
+
+		offset = epf_ntb_dma_bar_offset(ntb, barno);
+		if (!offset &&
+		    (epf_ntb_is_bar_used(ntb, barno) ||
+		     pci_epc_get_next_free_bar(features, barno) != barno))
+			return -EINVAL;
+		if (offset > U32_MAX)
+			return -EOVERFLOW;
+		dma->ctrl.submap.offset = offset;
+		next = offset;
+		if (next) {
+			/*
+			 * submap[0] covers the fixed-size MW prefix. Leave its target
+			 * at zero until the MW translation is installed.
+			 */
+			dma->submap[0].size = next;
+			dma->num_submap = 1;
+		}
 	}
 
 	dma->ctrl.magic = EPF_NTB_DMA_MAGIC;
@@ -497,7 +522,7 @@ epf_ntb_dw_edma_collect(struct epf_ntb *ntb,
 	}
 	if (dma->num_submap) {
 		dma->ctrl.submap.bar = barno;
-		dma->ctrl.submap.size = next;
+		dma->ctrl.submap.size = next - dma->ctrl.submap.offset;
 	}
 
 	dma_dev = ntb->epf->epc->dev.parent;
@@ -567,6 +592,8 @@ static int epf_ntb_dma_collect(struct epf_ntb *ntb)
 	if (!dma)
 		return -ENOMEM;
 
+	mutex_init(&dma->lock);
+
 	switch (ctrl->u.dma_ctrl.reg_layout) {
 	case PCI_EPC_AUX_DMA_REG_LAYOUT_DW_EDMA:
 		ret = epf_ntb_dw_edma_collect(ntb, dma, ctrl, resources, count);
@@ -621,17 +648,25 @@ static void epf_ntb_dma_release(struct epf_ntb *ntb, bool quiesce)
 	ntb->dma = NULL;
 }
 
-static int epf_ntb_dma_set_bar(struct epf_ntb *ntb, bool active)
+static int epf_ntb_dma_set_bar_locked(struct epf_ntb *ntb, bool active,
+				      const dma_addr_t *mw_addr)
 {
 	struct pci_epf_bar_submap *old_submap;
 	struct epf_ntb_dma *dma = ntb->dma;
 	struct pci_epf_bar *bar;
 	unsigned int old_num_submap;
+	dma_addr_t old_mw_addr;
 	int restore, ret;
 
+	lockdep_assert_held(&dma->lock);
+
 	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
 	old_submap = bar->submap;
 	old_num_submap = bar->num_submap;
+	if (mw_addr) {
+		old_mw_addr = dma->submap[0].phys_addr;
+		dma->submap[0].phys_addr = *mw_addr;
+	}
 	bar->submap = active ? dma->submap : NULL;
 	bar->num_submap = active ? dma->num_submap : 0;
 
@@ -641,13 +676,15 @@ static int epf_ntb_dma_set_bar(struct epf_ntb *ntb, bool active)
 		return 0;
 
 	/* A failed dynamic update may have already removed the old mapping. */
+	if (mw_addr)
+		dma->submap[0].phys_addr = old_mw_addr;
 	bar->submap = old_submap;
 	bar->num_submap = old_num_submap;
 	restore = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
 				  ntb->epf->vfunc_no, bar);
 	if (restore)
 		dev_warn(&ntb->epf->dev,
-			 "failed to restore DMA BAR mapping: %d\n", restore);
+			 "failed to restore DMA/MW BAR mapping: %d\n", restore);
 
 	return ret;
 }
@@ -660,11 +697,13 @@ static int epf_ntb_dma_set_active(struct epf_ntb *ntb, bool active)
 	if (!dma || !dma->num_submap)
 		return 0;
 
+	guard(mutex)(&dma->lock);
+
 	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
 	if (active == !!bar->num_submap)
 		return 0;
 
-	return epf_ntb_dma_set_bar(ntb, active);
+	return epf_ntb_dma_set_bar_locked(ntb, active, NULL);
 }
 
 /**
@@ -1214,7 +1253,7 @@ static int epf_ntb_dma_bar_init(struct epf_ntb *ntb)
 		return -EOPNOTSUPP;
 
 	barno = ntb->epf_ntb_bar[BAR_DMA];
-	mapped_size = dma->ctrl.submap.size;
+	mapped_size = dma->ctrl.submap.offset + dma->ctrl.submap.size;
 	/*
 	 * Submaps cannot be installed until the host assigns the BAR address.
 	 * Use address 0 for the temporary BAR Match Mode mapping, as is done
@@ -1312,6 +1351,7 @@ static void epf_ntb_db_bar_clear(struct epf_ntb *ntb)
  */
 static int epf_ntb_mw_bar_init(struct epf_ntb *ntb)
 {
+	bool shared;
 	int ret = 0;
 	int i;
 	u64 size;
@@ -1321,22 +1361,25 @@ static int epf_ntb_mw_bar_init(struct epf_ntb *ntb)
 	for (i = 0; i < ntb->num_mws; i++) {
 		size = ntb->mws_size[i];
 		barno = ntb->epf_ntb_bar[BAR_MW1 + i];
-
-		ntb->epf->bar[barno].barno = barno;
-		ntb->epf->bar[barno].size = size;
-		ntb->epf->bar[barno].addr = NULL;
-		ntb->epf->bar[barno].phys_addr = 0;
-		ntb->epf->bar[barno].flags |= upper_32_bits(size) ?
-				PCI_BASE_ADDRESS_MEM_TYPE_64 :
-				PCI_BASE_ADDRESS_MEM_TYPE_32;
-
-		ret = pci_epc_set_bar(ntb->epf->epc,
-				      ntb->epf->func_no,
-				      ntb->epf->vfunc_no,
-				      &ntb->epf->bar[barno]);
-		if (ret) {
-			dev_err(dev, "MW set failed\n");
-			goto err_alloc_mem;
+		shared = epf_ntb_dma_shares_bar(ntb, barno);
+
+		if (!shared) {
+			ntb->epf->bar[barno].barno = barno;
+			ntb->epf->bar[barno].size = size;
+			ntb->epf->bar[barno].addr = NULL;
+			ntb->epf->bar[barno].phys_addr = 0;
+			ntb->epf->bar[barno].flags |= upper_32_bits(size) ?
+					PCI_BASE_ADDRESS_MEM_TYPE_64 :
+					PCI_BASE_ADDRESS_MEM_TYPE_32;
+
+			ret = pci_epc_set_bar(ntb->epf->epc,
+					      ntb->epf->func_no,
+					      ntb->epf->vfunc_no,
+					      &ntb->epf->bar[barno]);
+			if (ret) {
+				dev_err(dev, "MW set failed\n");
+				goto err_alloc_mem;
+			}
 		}
 
 		/* Allocate EPC outbound memory windows to vpci vntb device */
@@ -1353,10 +1396,11 @@ static int epf_ntb_mw_bar_init(struct epf_ntb *ntb)
 	return ret;
 
 err_set_bar:
-	pci_epc_clear_bar(ntb->epf->epc,
-			  ntb->epf->func_no,
-			  ntb->epf->vfunc_no,
-			  &ntb->epf->bar[barno]);
+	if (!shared)
+		pci_epc_clear_bar(ntb->epf->epc,
+				  ntb->epf->func_no,
+				  ntb->epf->vfunc_no,
+				  &ntb->epf->bar[barno]);
 err_alloc_mem:
 	epf_ntb_mw_bar_clear(ntb, i);
 	return ret;
@@ -1374,10 +1418,11 @@ static void epf_ntb_mw_bar_clear(struct epf_ntb *ntb, int num_mws)
 
 	for (i = 0; i < num_mws; i++) {
 		barno = ntb->epf_ntb_bar[BAR_MW1 + i];
-		pci_epc_clear_bar(ntb->epf->epc,
-				  ntb->epf->func_no,
-				  ntb->epf->vfunc_no,
-				  &ntb->epf->bar[barno]);
+		if (!epf_ntb_dma_shares_bar(ntb, barno))
+			pci_epc_clear_bar(ntb->epf->epc,
+					  ntb->epf->func_no,
+					  ntb->epf->vfunc_no,
+					  &ntb->epf->bar[barno]);
 
 		pci_epc_mem_free_addr(ntb->epf->epc,
 				      ntb->vpci_mw_phy[i],
@@ -1974,6 +2019,16 @@ static int vntb_epf_mw_set_trans(struct ntb_dev *ndev, int pidx, int idx,
 	dev = &ntb->ntb.dev;
 	barno = ntb->epf_ntb_bar[BAR_MW1 + idx];
 	epf_bar = &ntb->epf->bar[barno];
+	if (epf_ntb_dma_shares_bar(ntb, barno)) {
+		/* DMA submaps follow this MW, so its extent cannot be changed. */
+		if (size != ntb->mws_size[idx])
+			return -EINVAL;
+
+		guard(mutex)(&ntb->dma->lock);
+
+		return epf_ntb_dma_set_bar_locked(ntb, true, &addr);
+	}
+
 	epf_bar->phys_addr = addr;
 	epf_bar->barno = barno;
 	epf_bar->size = size;
-- 
2.51.0

[PATCH v3 6/7] NTB: ntb_hw_epf: Discover vNTB-embedded DMA

[Koichiro Den]

A vNTB PCI function can now expose endpoint-local DMA channels to its
host. Decode the extension and register the advertised channels before
the NTB device, so clients of that NTB device can use the DMA
functionality in the same PCI function. Endpoints without the extension
keep the existing path.

Only DW eDMA is supported for now. Dispatch on the advertised DMA type
and keep type-specific setup in local helpers. Support for another
implementation can add a new case and its own helpers.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v3:
  - Validate DW eDMA register and linked-list offsets before mapping them.
    (Sashiko)

 drivers/ntb/hw/epf/ntb_hw_epf.c | 362 ++++++++++++++++++++++++++++++--
 1 file changed, 350 insertions(+), 12 deletions(-)

diff --git a/drivers/ntb/hw/epf/ntb_hw_epf.c b/drivers/ntb/hw/epf/ntb_hw_epf.c
index c47607d4f8a7..16cde97d4bde 100644
--- a/drivers/ntb/hw/epf/ntb_hw_epf.c
+++ b/drivers/ntb/hw/epf/ntb_hw_epf.c
@@ -8,6 +8,7 @@
 
 #include <linux/atomic.h>
 #include <linux/delay.h>
+#include <linux/dma/edma.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/slab.h>
@@ -20,6 +21,8 @@
 #define CMD_TEARDOWN_MW		4
 #define CMD_LINK_UP		5
 #define CMD_LINK_DOWN		6
+#define CMD_CONFIGURE_DMA	7
+#define CMD_TEARDOWN_DMA	8
 
 #define NTB_EPF_ARGUMENT	0x4
 #define MSIX_ENABLE		BIT(16)
@@ -44,6 +47,32 @@
 #define NTB_EPF_DB_DATA(n)	(0x34 + (n) * 4)
 #define NTB_EPF_DB_OFFSET(n)	(0xB4 + (n) * 4)
 
+/* Private DMA wire extension produced by pci-epf-vntb. */
+#define NTB_EPF_DMA_BASE	0x134
+#define NTB_EPF_DMA_MAGIC	(NTB_EPF_DMA_BASE + 0x00)
+#define NTB_EPF_DMA_REV_LEN	(NTB_EPF_DMA_BASE + 0x04)
+#define NTB_EPF_DMA_TYPE	(NTB_EPF_DMA_BASE + 0x08)
+#define NTB_EPF_DMA_REGION_BAR(base)	((base) + 0x00)
+#define NTB_EPF_DMA_REGION_OFFSET(base)	((base) + 0x04)
+#define NTB_EPF_DMA_REGION_SIZE(base)	((base) + 0x08)
+#define NTB_EPF_DMA_REGION_SIZEOF	0x0C
+#define NTB_EPF_DMA_SUBMAP_BASE	(NTB_EPF_DMA_BASE + 0x0C)
+#define NTB_EPF_DMA_COMMON_SIZE	0x18
+#define NTB_EPF_DMA_REG_BASE	(NTB_EPF_DMA_BASE + NTB_EPF_DMA_COMMON_SIZE)
+#define NTB_EPF_DMA_CHAN_BASE	(NTB_EPF_DMA_REG_BASE + \
+					 NTB_EPF_DMA_REGION_SIZEOF)
+#define NTB_EPF_DMA_CHAN_SIZE	0x14
+#define NTB_EPF_DMA_CHAN_DESC_BASE(n)	(NTB_EPF_DMA_CHAN_BASE + \
+					 (n) * NTB_EPF_DMA_CHAN_SIZE + 0x00)
+#define NTB_EPF_DMA_CHAN_DESC_ADDR_LO(n)	(NTB_EPF_DMA_CHAN_BASE + \
+					 (n) * NTB_EPF_DMA_CHAN_SIZE + 0x0C)
+#define NTB_EPF_DMA_CHAN_DESC_ADDR_HI(n)	(NTB_EPF_DMA_CHAN_BASE + \
+					 (n) * NTB_EPF_DMA_CHAN_SIZE + 0x10)
+
+#define NTB_EPF_DMA_MAGIC_VALUE	0x414d444e /* "NDMA": NTB DMA */
+#define NTB_EPF_DMA_REVISION	1
+#define NTB_EPF_DMA_TYPE_DW_EDMA	1
+
 /*
  * Legacy doorbell slot layout when paired with pci-epf-*ntb:
  *
@@ -97,6 +126,25 @@ struct ntb_epf_irq_ctx {
 	unsigned int irq_no;
 };
 
+struct ntb_epf_dma_region {
+	unsigned int bar;
+	resource_size_t offset;
+	resource_size_t size;
+	void __iomem *vaddr;
+};
+
+struct ntb_epf_dw_edma {
+	struct dw_edma_chip chip;
+	struct ntb_epf_dma_region reg;
+	struct ntb_epf_dma_region ll[EDMA_MAX_RD_CH];
+};
+
+struct ntb_epf_dma {
+	struct ntb_epf_dma_region submap;
+	unsigned int nr_irqs;
+	u32 type;
+};
+
 struct ntb_epf_dev {
 	struct ntb_dev ntb;
 	struct device *dev;
@@ -108,6 +156,8 @@ struct ntb_epf_dev {
 	unsigned int mw_count;
 	unsigned int spad_count;
 	unsigned int db_count;
+	struct ntb_epf_dma dma;
+	struct ntb_epf_dw_edma dw_edma;
 
 	void __iomem *ctrl_reg;
 	void __iomem *db_reg;
@@ -162,6 +212,245 @@ static int ntb_epf_send_command(struct ntb_epf_dev *ndev, u32 command,
 	return ret;
 }
 
+static bool ntb_epf_dma_region_parse(struct ntb_epf_dev *ndev, u32 base,
+				     bool optional,
+				     struct ntb_epf_dma_region *region)
+{
+	resource_size_t bar_len;
+	u32 bar, offset, size;
+
+	bar = readl(ndev->ctrl_reg + NTB_EPF_DMA_REGION_BAR(base));
+	offset = readl(ndev->ctrl_reg + NTB_EPF_DMA_REGION_OFFSET(base));
+	size = readl(ndev->ctrl_reg + NTB_EPF_DMA_REGION_SIZE(base));
+	region->bar = bar;
+	region->offset = offset;
+	region->size = size;
+	if (!size)
+		return optional && bar == U32_MAX && !offset;
+	if (bar > BAR_5)
+		return false;
+
+	bar_len = pci_resource_len(ndev->ntb.pdev, bar);
+	if (offset > bar_len || size > bar_len - offset)
+		return false;
+
+	return true;
+}
+
+/* DW eDMA */
+
+static int ntb_epf_dw_edma_parse(struct ntb_epf_dev *ndev, u32 length)
+{
+	struct ntb_epf_dw_edma *edma = &ndev->dw_edma;
+	struct dw_edma_chip *chip = &edma->chip;
+	u32 count, chan_offset;
+	unsigned int i;
+
+	chan_offset = NTB_EPF_DMA_CHAN_BASE - NTB_EPF_DMA_BASE;
+	if (length < chan_offset + NTB_EPF_DMA_CHAN_SIZE ||
+	    (length - chan_offset) % NTB_EPF_DMA_CHAN_SIZE)
+		return -EINVAL;
+
+	count = (length - chan_offset) / NTB_EPF_DMA_CHAN_SIZE;
+	if (count > EDMA_MAX_RD_CH)
+		return -EINVAL;
+
+	if (!ntb_epf_dma_region_parse(ndev, NTB_EPF_DMA_REG_BASE, false,
+				      &edma->reg) ||
+	    !IS_ALIGNED(edma->reg.offset, sizeof(u32)))
+		return -EINVAL;
+
+	chip->mf = EDMA_MF_EDMA_UNROLL;
+	chip->nr_irqs = count;
+	chip->ll_rd_cnt = count;
+	chip->func_no = PCI_FUNC(ndev->ntb.pdev->devfn);
+
+	for (i = 0; i < count; i++) {
+		u64 paddr;
+
+		if (!ntb_epf_dma_region_parse(ndev,
+					      NTB_EPF_DMA_CHAN_DESC_BASE(i),
+					      false, &edma->ll[i]) ||
+		    !IS_ALIGNED(edma->ll[i].offset, sizeof(u32)))
+			return -EINVAL;
+
+		paddr = readl(ndev->ctrl_reg +
+			      NTB_EPF_DMA_CHAN_DESC_ADDR_LO(i));
+		paddr |= (u64)readl(ndev->ctrl_reg +
+				    NTB_EPF_DMA_CHAN_DESC_ADDR_HI(i)) << 32;
+		if (paddr == U64_MAX)
+			return -EINVAL;
+
+		chip->ll_region_rd[i].sz = edma->ll[i].size;
+		chip->ll_region_rd[i].paddr = paddr;
+	}
+	ndev->dma.nr_irqs = count;
+
+	return 0;
+}
+
+static int ntb_epf_dw_edma_irq_vector(struct device *dev, unsigned int nr)
+{
+	struct ntb_epf_dev *ndev = dev_get_drvdata(dev);
+
+	/* DMA vectors follow the NTB link and doorbell vector block. */
+	return pci_irq_vector(ndev->ntb.pdev, ndev->db_count + 1 + nr);
+}
+
+static const struct dw_edma_plat_ops ntb_epf_dw_edma_ops = {
+	.irq_vector = ntb_epf_dw_edma_irq_vector,
+};
+
+static int ntb_epf_dw_edma_map_region(struct pci_dev *pdev,
+				      struct ntb_epf_dma_region *region)
+{
+	region->vaddr = pci_iomap_range(pdev, region->bar, region->offset,
+					region->size);
+	return region->vaddr ? 0 : -ENOMEM;
+}
+
+static void ntb_epf_dw_edma_unmap_regions(struct ntb_epf_dev *ndev)
+{
+	struct ntb_epf_dw_edma *edma = &ndev->dw_edma;
+	struct pci_dev *pdev = ndev->ntb.pdev;
+	unsigned int i;
+
+	for (i = 0; i < edma->chip.ll_rd_cnt; i++) {
+		if (!edma->ll[i].vaddr)
+			continue;
+		pci_iounmap(pdev, edma->ll[i].vaddr);
+	}
+	if (edma->reg.vaddr)
+		pci_iounmap(pdev, edma->reg.vaddr);
+}
+
+static int ntb_epf_dw_edma_init(struct ntb_epf_dev *ndev)
+{
+	struct ntb_epf_dw_edma *edma = &ndev->dw_edma;
+	struct dw_edma_chip *chip = &edma->chip;
+	struct pci_dev *pdev = ndev->ntb.pdev;
+	unsigned int i;
+	int ret;
+
+	/* Install the endpoint BAR submaps before mapping the advertised regions. */
+	ret = ntb_epf_send_command(ndev, CMD_CONFIGURE_DMA, 0);
+	if (ret)
+		return ret;
+
+	ret = ntb_epf_dw_edma_map_region(pdev, &edma->reg);
+	if (ret)
+		goto err_teardown;
+	for (i = 0; i < chip->ll_rd_cnt; i++) {
+		ret = ntb_epf_dw_edma_map_region(pdev, &edma->ll[i]);
+		if (ret)
+			goto err_iounmap;
+		chip->ll_region_rd[i].vaddr.io = edma->ll[i].vaddr;
+	}
+
+	chip->dev = ndev->dev;
+	chip->ops = &ntb_epf_dw_edma_ops;
+	chip->flags = DW_EDMA_CHIP_PARTIAL;
+	chip->reg_base = edma->reg.vaddr;
+
+	ret = dw_edma_probe(chip);
+	if (ret)
+		goto err_iounmap;
+
+	return 0;
+
+err_iounmap:
+	ntb_epf_dw_edma_unmap_regions(ndev);
+err_teardown:
+	ntb_epf_send_command(ndev, CMD_TEARDOWN_DMA, 0);
+	return ret;
+}
+
+static void ntb_epf_dw_edma_deinit(struct ntb_epf_dev *ndev)
+{
+	struct ntb_epf_dw_edma *edma = &ndev->dw_edma;
+	int ret;
+
+	dw_edma_remove(&edma->chip);
+	ntb_epf_dw_edma_unmap_regions(ndev);
+
+	ret = ntb_epf_send_command(ndev, CMD_TEARDOWN_DMA, 0);
+	if (ret)
+		dev_warn(ndev->dev, "Failed to teardown endpoint DMA\n");
+}
+
+/* Common endpoint DMA */
+
+static int ntb_epf_dma_parse(struct ntb_epf_dev *ndev)
+{
+	u32 magic, rev_len, length, type;
+	u32 spad_off;
+	int ret;
+
+	/*
+	 * Legacy endpoints place scratchpads where this extension would begin.
+	 * Once the extension is present, malformed metadata is fatal.
+	 */
+	spad_off = readl(ndev->ctrl_reg + NTB_EPF_SPAD_OFFSET);
+	if (spad_off < NTB_EPF_DMA_BASE + NTB_EPF_DMA_COMMON_SIZE)
+		return 0;
+
+	magic = readl(ndev->ctrl_reg + NTB_EPF_DMA_MAGIC);
+	if (!magic)
+		return 0;
+	if (magic == U32_MAX)
+		return -EIO;
+	if (magic != NTB_EPF_DMA_MAGIC_VALUE)
+		return -EINVAL;
+
+	rev_len = readl(ndev->ctrl_reg + NTB_EPF_DMA_REV_LEN);
+	length = upper_16_bits(rev_len);
+	if (lower_16_bits(rev_len) != NTB_EPF_DMA_REVISION ||
+	    length < NTB_EPF_DMA_COMMON_SIZE ||
+	    spad_off < NTB_EPF_DMA_BASE + length)
+		return -EINVAL;
+	if (!ntb_epf_dma_region_parse(ndev, NTB_EPF_DMA_SUBMAP_BASE, true,
+				      &ndev->dma.submap))
+		return -EINVAL;
+
+	type = readl(ndev->ctrl_reg + NTB_EPF_DMA_TYPE);
+	if (type == U32_MAX)
+		return -EIO;
+
+	switch (type) {
+	case NTB_EPF_DMA_TYPE_DW_EDMA:
+		ret = ntb_epf_dw_edma_parse(ndev, length);
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+
+	if (!ret)
+		ndev->dma.type = type;
+
+	return ret;
+}
+
+static int ntb_epf_dma_init(struct ntb_epf_dev *ndev)
+{
+	switch (ndev->dma.type) {
+	case 0:
+		return 0;
+	case NTB_EPF_DMA_TYPE_DW_EDMA:
+		return ntb_epf_dw_edma_init(ndev);
+	default:
+		return -EOPNOTSUPP;
+	}
+}
+
+static void ntb_epf_dma_deinit(struct ntb_epf_dev *ndev)
+{
+	switch (ndev->dma.type) {
+	case NTB_EPF_DMA_TYPE_DW_EDMA:
+		ntb_epf_dw_edma_deinit(ndev);
+		break;
+	}
+}
+
 static int ntb_epf_mw_to_bar(struct ntb_epf_dev *ndev, int idx)
 {
 	struct device *dev = ndev->dev;
@@ -174,6 +463,25 @@ static int ntb_epf_mw_to_bar(struct ntb_epf_dev *ndev, int idx)
 	return ndev->barno_map[BAR_MW1 + idx];
 }
 
+static resource_size_t ntb_epf_mw_offset(struct ntb_epf_dev *ndev, int idx)
+{
+	return !idx ? readl(ndev->ctrl_reg + NTB_EPF_MW1_OFFSET) : 0;
+}
+
+static resource_size_t ntb_epf_mw_size(struct ntb_epf_dev *ndev, int idx,
+				       int bar)
+{
+	resource_size_t offset, end;
+
+	offset = ntb_epf_mw_offset(ndev, idx);
+	end = pci_resource_len(ndev->ntb.pdev, bar);
+	/* The DMA submap offset marks the end of an MW sharing this BAR. */
+	if (ndev->dma.submap.size && ndev->dma.submap.bar == bar)
+		end = min(end, ndev->dma.submap.offset);
+
+	return offset < end ? end - offset : 0;
+}
+
 static int ntb_epf_mw_count(struct ntb_dev *ntb, int pidx)
 {
 	struct ntb_epf_dev *ndev = ntb_ndev(ntb);
@@ -212,7 +520,7 @@ static int ntb_epf_mw_get_align(struct ntb_dev *ntb, int pidx, int idx,
 		*size_align = 1;
 
 	if (size_max)
-		*size_max = pci_resource_len(ndev->ntb.pdev, bar);
+		*size_max = ntb_epf_mw_size(ndev, idx, bar);
 
 	return 0;
 }
@@ -373,15 +681,19 @@ static int ntb_epf_init_isr(struct ntb_epf_dev *ndev, int msi_min, int msi_max)
 {
 	struct pci_dev *pdev = ndev->ntb.pdev;
 	struct device *dev = ndev->dev;
+	unsigned int dma_irqs = ndev->dma.nr_irqs;
+	unsigned int ntb_irqs;
 	u32 argument = MSIX_ENABLE;
 	int irq;
 	int ret;
 	int i;
 
-	irq = pci_alloc_irq_vectors(pdev, msi_min, msi_max, PCI_IRQ_MSIX);
+	irq = pci_alloc_irq_vectors(pdev, msi_min + dma_irqs,
+				    msi_max + dma_irqs, PCI_IRQ_MSIX);
 	if (irq < 0) {
 		dev_dbg(dev, "Failed to get MSIX interrupts\n");
-		irq = pci_alloc_irq_vectors(pdev, msi_min, msi_max,
+		irq = pci_alloc_irq_vectors(pdev, msi_min + dma_irqs,
+					    msi_max + dma_irqs,
 					    PCI_IRQ_MSI);
 		if (irq < 0) {
 			dev_err(dev, "Failed to get MSI interrupts\n");
@@ -390,8 +702,9 @@ static int ntb_epf_init_isr(struct ntb_epf_dev *ndev, int msi_min, int msi_max)
 		argument &= ~MSIX_ENABLE;
 	}
 
-	ndev->db_count = irq - 1;
-	for (i = 0; i < irq; i++) {
+	ntb_irqs = irq - dma_irqs;
+	ndev->db_count = ntb_irqs - 1;
+	for (i = 0; i < ntb_irqs; i++) {
 		ndev->irq_ctx[i].ndev = ndev;
 		ndev->irq_ctx[i].irq_no = i;
 		ret = request_irq(pci_irq_vector(pdev, i), ntb_epf_vec_isr,
@@ -403,7 +716,7 @@ static int ntb_epf_init_isr(struct ntb_epf_dev *ndev, int msi_min, int msi_max)
 	}
 
 	ret = ntb_epf_send_command(ndev, CMD_CONFIGURE_DOORBELL,
-				   argument | irq);
+				   argument | ntb_irqs);
 	if (ret) {
 		dev_err(dev, "Failed to configure doorbell\n");
 		goto err_free_irq;
@@ -486,7 +799,7 @@ static int ntb_epf_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
 	if (bar < 0)
 		return bar;
 
-	mw_size = pci_resource_len(ntb->pdev, bar);
+	mw_size = ntb_epf_mw_size(ndev, idx, bar);
 
 	if (size > mw_size) {
 		dev_err(dev, "Size:%pa is greater than the MW size %pa\n",
@@ -520,21 +833,20 @@ static int ntb_epf_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
 				    phys_addr_t *base, resource_size_t *size)
 {
 	struct ntb_epf_dev *ndev = ntb_ndev(ntb);
-	u32 offset = 0;
+	resource_size_t offset;
 	int bar;
 
-	if (idx == 0)
-		offset = readl(ndev->ctrl_reg + NTB_EPF_MW1_OFFSET);
-
 	bar = ntb_epf_mw_to_bar(ndev, idx);
 	if (bar < 0)
 		return bar;
 
+	offset = ntb_epf_mw_offset(ndev, idx);
+
 	if (base)
 		*base = pci_resource_start(ndev->ntb.pdev, bar) + offset;
 
 	if (size)
-		*size = pci_resource_len(ndev->ntb.pdev, bar) - offset;
+		*size = ntb_epf_mw_size(ndev, idx, bar);
 
 	return 0;
 }
@@ -630,12 +942,29 @@ static int ntb_epf_init_dev(struct ntb_epf_dev *ndev)
 {
 	struct device *dev = ndev->dev;
 	int ret;
+	int i;
 
 	ndev->mw_count = readl(ndev->ctrl_reg + NTB_EPF_MW_COUNT);
 	if (ndev->mw_count > NTB_EPF_MAX_MW_COUNT) {
 		dev_err(dev, "Unsupported MW count: %u\n", ndev->mw_count);
 		return -EINVAL;
 	}
+	ret = ntb_epf_dma_parse(ndev);
+	if (ret) {
+		dev_err(dev, "Invalid endpoint DMA layout\n");
+		return ret;
+	}
+	if (ndev->dma.submap.size) {
+		for (i = 0; i < ndev->mw_count; i++) {
+			int bar = ntb_epf_mw_to_bar(ndev, i);
+
+			if (bar == ndev->dma.submap.bar &&
+			    !ntb_epf_mw_size(ndev, i, bar)) {
+				dev_err(dev, "Invalid DMA/MW boundary\n");
+				return -EINVAL;
+			}
+		}
+	}
 
 	/* One Link interrupt and rest doorbell interrupt */
 	ret = ntb_epf_init_isr(ndev, NTB_EPF_MIN_DB_COUNT + 1,
@@ -785,6 +1114,12 @@ static int ntb_epf_pci_probe(struct pci_dev *pdev,
 		goto err_init_dev;
 	}
 
+	ret = ntb_epf_dma_init(ndev);
+	if (ret) {
+		dev_err(dev, "Failed to initialize endpoint DMA\n");
+		goto err_dma_init;
+	}
+
 	ret = ntb_register_device(&ndev->ntb);
 	if (ret) {
 		dev_err(dev, "Failed to register NTB device\n");
@@ -794,6 +1129,8 @@ static int ntb_epf_pci_probe(struct pci_dev *pdev,
 	return 0;
 
 err_register_dev:
+	ntb_epf_dma_deinit(ndev);
+err_dma_init:
 	ntb_epf_cleanup_isr(ndev);
 
 err_init_dev:
@@ -807,6 +1144,7 @@ static void ntb_epf_pci_remove(struct pci_dev *pdev)
 	struct ntb_epf_dev *ndev = pci_get_drvdata(pdev);
 
 	ntb_unregister_device(&ndev->ntb);
+	ntb_epf_dma_deinit(ndev);
 	ntb_epf_cleanup_isr(ndev);
 	ntb_epf_deinit_pci(ndev);
 }
-- 
2.51.0

[PATCH v3 7/7] Documentation: PCI: endpoint: Document vNTB DMA export

[Koichiro Den]

Document the dma_bar attribute and the requirements for exporting endpoint
DMA channels. Also document how the selected BAR may be shared with an MW.

Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes in v3:
  - New patch split from the DMA export patch.
  - Document dma_bar as the only knob and describe DMA/MW BAR sharing.

 Documentation/PCI/endpoint/pci-vntb-howto.rst | 24 ++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/Documentation/PCI/endpoint/pci-vntb-howto.rst b/Documentation/PCI/endpoint/pci-vntb-howto.rst
index 3679f5c30254..3a01327208cc 100644
--- a/Documentation/PCI/endpoint/pci-vntb-howto.rst
+++ b/Documentation/PCI/endpoint/pci-vntb-howto.rst
@@ -90,9 +90,9 @@ of the function device and is populated with the following NTB specific
 attributes that can be configured by the user::
 
 	# ls functions/pci_epf_vntb/func1/pci_epf_vntb.0/
-	ctrl_bar  db_count  mw1_bar  mw2_bar  mw3_bar  mw4_bar	spad_count
-	db_bar	  mw1	    mw2      mw3      mw4      num_mws	vbus_number
-	vntb_vid  vntb_pid
+	ctrl_bar  dma_bar  mw2      mw3_bar  num_mws     vntb_pid
+	db_bar    mw1      mw2_bar  mw4      spad_count  vntb_vid
+	db_count  mw1_bar  mw3      mw4_bar  vbus_number
 
 A sample configuration for NTB function is given below::
 
@@ -105,6 +105,24 @@ By default, each construct is assigned a BAR, as needed and in order.
 Should a specific BAR setup be required by the platform, BAR may be assigned
 to each construct using the related ``XYZ_bar`` entry.
 
+To export DMA channels, select their BAR before binding the function::
+
+	# echo 5 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_bar
+
+This currently supports the unrolled DesignWare eDMA layout. If any DMA
+resource has no fixed BAR assignment, the EPC must support subrange and
+dynamic inbound mappings.
+
+Leaving ``dma_bar`` unassigned disables DMA export.
+With a separate DMA BAR, an older ``ntb_hw_epf`` peer can ignore the extension
+and continue using legacy NTB. A peer that understands the extension treats the
+advertised DMA type as required and fails probe if its support is unavailable.
+
+``dma_bar`` may instead select an MW BAR, placing the DMA resources after that
+MW. Any rounded-up tail is scratch-backed. This requires an updated peer because
+older peers treat the whole BAR as an MW. Control and doorbell BARs cannot be
+shared.
+
 A sample configuration for virtual NTB driver for virtual PCI bus::
 
 	# echo 0x1957 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_vid
-- 
2.51.0

Re: [PATCH v3 1/7] PCI: endpoint: Add DMA auxiliary resource metadata

[sashiko-bot]

> Endpoint functions that export controller-local DMA channels need the
> controller register and descriptor-memory layout.
> 
> Add DMA control and descriptor-memory resource types. The control resource
> carries the register layout and channel counts. Each descriptor region is
> associated with the static channel ID assigned by the local DMA driver.
> 
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=1

Re: [PATCH v3 2/7] PCI: dwc: Expose endpoint DMA resources

[sashiko-bot]

> Expose the endpoint-integrated eDMA register window and linked-list memory
> through the EPC auxiliary resource API. Endpoint functions can then choose
> which channels to export and map the required windows.
> 
> Associate each linked-list region with the direction-flattened static
> channel ID used by the local dw-edma device. Report these resources only
> after that device is registered and only for channels with linked-list
> memory.
> 
> When the register window is already part of a reserved BAR, report its BAR
> and offset. Otherwise report its CPU physical address.
> 
> While at it, harden the existing doorbell resource query. Report it
> only after the local dw-edma device is registered, and reject VF
> queries because DWC cannot assign DMA/HDMA registers to VFs.
> 
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=2

Re: [PATCH v3 3/7] PCI: endpoint: pci-epf-vntb: Move epf_ntb_is_bar_used() up

[sashiko-bot]

> Move epf_ntb_is_bar_used() up so later BAR selection code can call it
> without a forward declaration. No functional change.
> 
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=3

Re: [PATCH v3 4/7] PCI: endpoint: pci-epf-vntb: Export endpoint DMA channels

[sashiko-bot]

> An RC may use endpoint-local DMA read channels to transfer data directly
> to an endpoint DMA address once both sides agree to use them. Quiescing
> an unrolled eDMA channel disables its whole direction, so reserve the
> complete read direction and route its interrupts to the RC when dma_bar
> is configured.
> 
> Describe the controller and per-channel descriptor memory in a private
> control-region extension.
> 
> Add a dma_bar configfs attribute. An explicit BAR selection enables DMA
> export; leaving it unassigned keeps the feature disabled.
> 
> Keep resources already assigned to a BAR in place, and map the rest
> through the selected dma_bar.
> 
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=4

Re: [PATCH v3 5/7] PCI: endpoint: pci-epf-vntb: Allow DMA and MW to share a BAR

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [Critical] Deactivating a shared DMA BAR unconditionally clears all submaps, breaking the active Memory Window (MW) mapping and causing severe memory corruption.
- [High] The `ntb->dma` structure is freed in `epf_ntb_unbind()` before the driver is unregistered, allowing concurrent NTB subsystem operations to use a freed pointer and its lock.

Pre-existing issues:
- [High] The virtual PCI driver `vntb_pci_driver` lacks a `.remove` callback, causing the NTB device and virtual PCI bus devices to leak and potentially use freed memory.
--

commit 4680cfd6b651b64bf9225e9233e1d66310c8b0da
Author: Koichiro Den <den@valinux.co.jp>

PCI: endpoint: pci-epf-vntb: Allow DMA and MW to share a BAR

This patch modifies the PCI endpoint vNTB function to allow a DMA BAR to
select an MW BAR and place DMA ranges after it. This enables configurations
that have no spare BAR for DMA resources to share one.

> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> index 1842493c33b02..aa659ec62d931 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
[ ... ]
> @@ -621,17 +648,25 @@ static void epf_ntb_dma_release(struct epf_ntb *ntb, bool quiesce)
>  	ntb->dma = NULL;
>  }
>  
> -static int epf_ntb_dma_set_bar(struct epf_ntb *ntb, bool active)
> +static int epf_ntb_dma_set_bar_locked(struct epf_ntb *ntb, bool active,
> +				      const dma_addr_t *mw_addr)
>  {
>  	struct pci_epf_bar_submap *old_submap;
>  	struct epf_ntb_dma *dma = ntb->dma;
>  	struct pci_epf_bar *bar;
>  	unsigned int old_num_submap;
> +	dma_addr_t old_mw_addr;
>  	int restore, ret;
>  
> +	lockdep_assert_held(&dma->lock);
> +
>  	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
>  	old_submap = bar->submap;
>  	old_num_submap = bar->num_submap;
> +	if (mw_addr) {
> +		old_mw_addr = dma->submap[0].phys_addr;
> +		dma->submap[0].phys_addr = *mw_addr;
> +	}
> -	bar->submap = active ? dma->submap : NULL;
> -	bar->num_submap = active ? dma->num_submap : 0;
> +	bar->submap = active ? dma->submap : NULL;
> +	bar->num_submap = active ? dma->num_submap : 0;

[Severity: Critical]
If the host sends a COMMAND_TEARDOWN_DMA while a Memory Window is still
active, epf_ntb_dma_set_active() will call this function with active set to
false.

Does this unconditionally clear all submaps on the BAR, including submap[0]
which covers the MW prefix? 

Since pci_epf_assign_bar_space() used address 0 as the fallback base physical
address, clearing the submap would cause pci_epc_set_bar() to map the entire
BAR to CPU physical address 0 on the endpoint. Could this corrupt memory when
the host subsequently accesses the MW?

>  
>  	ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
>  			      ntb->epf->vfunc_no, bar);
[ ... ]
> @@ -1974,6 +2019,16 @@ static int vntb_epf_mw_set_trans(struct ntb_dev *ndev, int pidx, int idx,
>  	dev = &ntb->ntb.dev;
>  	barno = ntb->epf_ntb_bar[BAR_MW1 + idx];
>  	epf_bar = &ntb->epf->bar[barno];
> +	if (epf_ntb_dma_shares_bar(ntb, barno)) {
> +		/* DMA submaps follow this MW, so its extent cannot be changed. */
> +		if (size != ntb->mws_size[idx])
> +			return -EINVAL;
> +
> +		guard(mutex)(&ntb->dma->lock);

[Severity: High]
Can this lock be accessed after ntb->dma has been freed?

In epf_ntb_unbind(), epf_ntb_dma_release() is called, which frees ntb->dma
via kfree(). However, pci_unregister_driver(&vntb_pci_driver) is called
after this.

If a host MW configuration command runs concurrently with unbind, could
vntb_epf_mw_set_trans() check epf_ntb_dma_shares_bar() and then attempt to
acquire this mutex on the freed ntb->dma pointer?

> +
> +		return epf_ntb_dma_set_bar_locked(ntb, true, &addr);
> +	}
> +
>  	epf_bar->phys_addr = addr;
>  	epf_bar->barno = barno;
>  	epf_bar->size = size;

[Severity: High]
This is a pre-existing issue, but I noticed that vntb_pci_driver lacks a
remove callback:

static struct pci_driver vntb_pci_driver = {
	.name           = "pci-vntb",
	.id_table       = pci_vntb_table,
	.probe          = pci_vntb_probe,
};

When epf_ntb_unbind() calls pci_unregister_driver(), it unbinds the driver
but ntb_unregister_device() is never called because there is no cleanup
function. 

Since the struct epf_ntb instance is allocated via devm_kzalloc() tied to
the EPF device, it gets freed when the EPF device is removed. Does this
cause the NTB device and virtual PCI bus devices to leak, leading to a
use-after-free if the NTB subsystem makes subsequent calls?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=5

Re: [PATCH v3 6/7] NTB: ntb_hw_epf: Discover vNTB-embedded DMA

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

New issues:
- [High] Probe fails entirely if the endpoint advertises an unsupported DMA type or if dw-edma host support is missing.
- [High] Strictly demanding `dma_irqs` extra MSI/MSI-X vectors breaks probe on IRQ-constrained systems.

Pre-existing issues:
- [Critical] Dynamic reading of the SPAD offset without validation allows a malicious endpoint to cause arbitrary host MMIO read/write.
- [High] Pre-existing race condition in `ntb_epf_mw_set_trans` where multiple concurrent calls can interleave writes to shared memory window configuration registers.
--

commit 9b8b5f4c03fc934132c68dec22efe8a4ca1332ef
Author: Koichiro Den <den@valinux.co.jp>

NTB: ntb_hw_epf: Discover vNTB-embedded DMA

A vNTB PCI function can now expose endpoint-local DMA channels to its
host. This patch decodes the extension and registers the advertised
channels before the NTB device.

> diff --git a/drivers/ntb/hw/epf/ntb_hw_epf.c b/drivers/ntb/hw/epf/ntb_hw_epf.c
> index c47607d4f8a7f..16cde97d4bde2 100644
> --- a/drivers/ntb/hw/epf/ntb_hw_epf.c
> +++ b/drivers/ntb/hw/epf/ntb_hw_epf.c

[ ... ]

> @@ -373,15 +681,19 @@ static int ntb_epf_init_isr(struct ntb_epf_dev *ndev, int msi_min, int msi_max)
>  {
>  	struct pci_dev *pdev = ndev->ntb.pdev;
>  	struct device *dev = ndev->dev;
> +	unsigned int dma_irqs = ndev->dma.nr_irqs;
> +	unsigned int ntb_irqs;
>  	u32 argument = MSIX_ENABLE;
>  	int irq;
>  	int ret;
>  	int i;
>  
> -	irq = pci_alloc_irq_vectors(pdev, msi_min, msi_max, PCI_IRQ_MSIX);
> +	irq = pci_alloc_irq_vectors(pdev, msi_min + dma_irqs,
> +				    msi_max + dma_irqs, PCI_IRQ_MSIX);

[Severity: High]
Will strictly demanding the extra dma_irqs break probe on IRQ-constrained
host systems?

If the host platform can only supply fewer vectors than the minimum
requested here (which can inflate up to 10 if the endpoint advertises 8
DW eDMA channels), pci_alloc_irq_vectors() will return a negative error
code.

Should the driver request only the minimum required for basic NTB
operation, and selectively allocate or disable DMA channels based on the
number of vectors actually granted?

[ ... ]

> @@ -486,7 +799,7 @@ static int ntb_epf_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
>  	if (bar < 0)
>  		return bar;
>  
> -	mw_size = pci_resource_len(ntb->pdev, bar);
> +	mw_size = ntb_epf_mw_size(ndev, idx, bar);
>  
>  	if (size > mw_size) {
>  		dev_err(dev, "Size:%pa is greater than the MW size %pa\n",

[Severity: High]
This is a pre-existing issue, but can concurrent calls to
ntb_epf_mw_set_trans() corrupt the memory window configurations?

The function writes to a shared set of registers across memory windows
right after this block:

writel(lower_32_bits(addr), ndev->ctrl_reg + NTB_EPF_LOWER_ADDR);
writel(upper_32_bits(addr), ndev->ctrl_reg + NTB_EPF_UPPER_ADDR);
writel(lower_32_bits(size), ndev->ctrl_reg + NTB_EPF_LOWER_SIZE);
writel(upper_32_bits(size), ndev->ctrl_reg + NTB_EPF_UPPER_SIZE);

It then calls ntb_epf_send_command(), which is where ndev->cmd_lock is
finally acquired. If multiple threads configure different memory windows
concurrently, can their writes interleave and send a corrupted mixture of
addresses and sizes?

[ ... ]

> @@ -785,6 +1114,12 @@ static int ntb_epf_pci_probe(struct pci_dev *pdev,
>  		goto err_init_dev;
>  	}
>  
> +	ret = ntb_epf_dma_init(ndev);
> +	if (ret) {
> +		dev_err(dev, "Failed to initialize endpoint DMA\n");
> +		goto err_dma_init;
> +	}

[Severity: High]
If the endpoint advertises an unsupported DMA type, or if the host kernel
is compiled without CONFIG_DW_EDMA, ntb_epf_dma_init() will fail. Does
this mean the entire NTB device initialization will completely abort here?

Since the DMA extension is optional, would it be better to log a warning
and fall back to the existing non-DMA path instead of completely rejecting
the device?


While reviewing this file, I also noticed a pre-existing issue in
ntb_epf_spad_read() and ntb_epf_spad_write() which are not modified by
this patch:

[Severity: Critical]
This isn't a bug introduced by this patch, but is it safe to use the SPAD
offset directly from hardware for an IOMEM read or write without
validation?

In ntb_epf_spad_read(), the offset is read dynamically:

offset = readl(ndev->ctrl_reg + NTB_EPF_SPAD_OFFSET);
offset += (idx << 2);
return readl(ndev->ctrl_reg + offset);

Since the PCIe endpoint controls its configuration space, a malicious or
compromised endpoint could modify NTB_EPF_SPAD_OFFSET to an arbitrarily
large 32-bit value. Does this allow the endpoint to trigger an
unvalidated out-of-bounds MMIO read or write on the host?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=6

Re: [PATCH v3 7/7] Documentation: PCI: endpoint: Document vNTB DMA export

[sashiko-bot]

> Document the dma_bar attribute and the requirements for exporting endpoint
> DMA channels. Also document how the selected BAR may be shared with an MW.
> 
> Signed-off-by: Koichiro Den <den@valinux.co.jp>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=7

Re: [PATCH v3 5/7] PCI: endpoint: pci-epf-vntb: Allow DMA and MW to share a BAR

[Koichiro Den]

On Mon, Aug 31, 2026 at 09:05:48PM +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
> 
> New issues:
> - [Critical] Deactivating a shared DMA BAR unconditionally clears all submaps, breaking the active Memory Window (MW) mapping and causing severe memory corruption.
> - [High] The `ntb->dma` structure is freed in `epf_ntb_unbind()` before the driver is unregistered, allowing concurrent NTB subsystem operations to use a freed pointer and its lock.

Both are valid, thanks for the catch.

The second issue comes directly from the pre-existing vNTB lifetime handling. I
posted the prerequisite fix series here:

  https://lore.kernel.org/r/20260901063238.631166-1-den@valinux.co.jp/

That fix also gives the DMA BAR mapping a clear lifetime. The EP owns the DMA
resources and their BAR submaps for the whole EPF bind lifetime, so a
host-driven TEARDOWN_DMA is unnecessary and, for a shared BAR, unsafe. I will
drop it and rename CONFIGURE_DMA to SETUP_DMA_BAR in the next revision.
(The CONFIGURE_DMA naming was misleading even in this v3. Unlike CONFIGURE_MW
and TEARDOWN_MW, it installs an EP-side inbound mapping rather than managing an
outbound mapping.)

Best regards,
Koichiro

> 
> Pre-existing issues:
> - [High] The virtual PCI driver `vntb_pci_driver` lacks a `.remove` callback, causing the NTB device and virtual PCI bus devices to leak and potentially use freed memory.
> --
> 
> commit 4680cfd6b651b64bf9225e9233e1d66310c8b0da
> Author: Koichiro Den <den@valinux.co.jp>
> 
> PCI: endpoint: pci-epf-vntb: Allow DMA and MW to share a BAR
> 
> This patch modifies the PCI endpoint vNTB function to allow a DMA BAR to
> select an MW BAR and place DMA ranges after it. This enables configurations
> that have no spare BAR for DMA resources to share one.
> 
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > index 1842493c33b02..aa659ec62d931 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> [ ... ]
> > @@ -621,17 +648,25 @@ static void epf_ntb_dma_release(struct epf_ntb *ntb, bool quiesce)
> >  	ntb->dma = NULL;
> >  }
> >  
> > -static int epf_ntb_dma_set_bar(struct epf_ntb *ntb, bool active)
> > +static int epf_ntb_dma_set_bar_locked(struct epf_ntb *ntb, bool active,
> > +				      const dma_addr_t *mw_addr)
> >  {
> >  	struct pci_epf_bar_submap *old_submap;
> >  	struct epf_ntb_dma *dma = ntb->dma;
> >  	struct pci_epf_bar *bar;
> >  	unsigned int old_num_submap;
> > +	dma_addr_t old_mw_addr;
> >  	int restore, ret;
> >  
> > +	lockdep_assert_held(&dma->lock);
> > +
> >  	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
> >  	old_submap = bar->submap;
> >  	old_num_submap = bar->num_submap;
> > +	if (mw_addr) {
> > +		old_mw_addr = dma->submap[0].phys_addr;
> > +		dma->submap[0].phys_addr = *mw_addr;
> > +	}
> > -	bar->submap = active ? dma->submap : NULL;
> > -	bar->num_submap = active ? dma->num_submap : 0;
> > +	bar->submap = active ? dma->submap : NULL;
> > +	bar->num_submap = active ? dma->num_submap : 0;
> 
> [Severity: Critical]
> If the host sends a COMMAND_TEARDOWN_DMA while a Memory Window is still
> active, epf_ntb_dma_set_active() will call this function with active set to
> false.
> 
> Does this unconditionally clear all submaps on the BAR, including submap[0]
> which covers the MW prefix? 
> 
> Since pci_epf_assign_bar_space() used address 0 as the fallback base physical
> address, clearing the submap would cause pci_epc_set_bar() to map the entire
> BAR to CPU physical address 0 on the endpoint. Could this corrupt memory when
> the host subsequently accesses the MW?
> 
> >  
> >  	ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
> >  			      ntb->epf->vfunc_no, bar);
> [ ... ]
> > @@ -1974,6 +2019,16 @@ static int vntb_epf_mw_set_trans(struct ntb_dev *ndev, int pidx, int idx,
> >  	dev = &ntb->ntb.dev;
> >  	barno = ntb->epf_ntb_bar[BAR_MW1 + idx];
> >  	epf_bar = &ntb->epf->bar[barno];
> > +	if (epf_ntb_dma_shares_bar(ntb, barno)) {
> > +		/* DMA submaps follow this MW, so its extent cannot be changed. */
> > +		if (size != ntb->mws_size[idx])
> > +			return -EINVAL;
> > +
> > +		guard(mutex)(&ntb->dma->lock);
> 
> [Severity: High]
> Can this lock be accessed after ntb->dma has been freed?
> 
> In epf_ntb_unbind(), epf_ntb_dma_release() is called, which frees ntb->dma
> via kfree(). However, pci_unregister_driver(&vntb_pci_driver) is called
> after this.
> 
> If a host MW configuration command runs concurrently with unbind, could
> vntb_epf_mw_set_trans() check epf_ntb_dma_shares_bar() and then attempt to
> acquire this mutex on the freed ntb->dma pointer?
> 
> > +
> > +		return epf_ntb_dma_set_bar_locked(ntb, true, &addr);
> > +	}
> > +
> >  	epf_bar->phys_addr = addr;
> >  	epf_bar->barno = barno;
> >  	epf_bar->size = size;
> 
> [Severity: High]
> This is a pre-existing issue, but I noticed that vntb_pci_driver lacks a
> remove callback:
> 
> static struct pci_driver vntb_pci_driver = {
> 	.name           = "pci-vntb",
> 	.id_table       = pci_vntb_table,
> 	.probe          = pci_vntb_probe,
> };
> 
> When epf_ntb_unbind() calls pci_unregister_driver(), it unbinds the driver
> but ntb_unregister_device() is never called because there is no cleanup
> function. 
> 
> Since the struct epf_ntb instance is allocated via devm_kzalloc() tied to
> the EPF device, it gets freed when the EPF device is removed. Does this
> cause the NTB device and virtual PCI bus devices to leak, leading to a
> use-after-free if the NTB subsystem makes subsequent calls?
> 
> -- 
> Sashiko AI review · https://sashiko.dev/#/patchset/20260831182657.329614-1-den@valinux.co.jp?part=5

Re: [PATCH v3 0/7] PCI: endpoint: Remote DMA support via vNTB

[Manivannan Sadhasivam]

On Tue, Sep 01, 2026 at 03:26:50AM +0900, Koichiro Den wrote:
> Hi,
> 
> This series lets an RC use endpoint-local DesignWare eDMA read channels
> through vNTB. It keeps the export within pci-epf-vntb and ntb_hw_epf
> rather than defining a generic remote-DMA ABI or using a separate PCI
> DMA function.
> 
> On the endpoint, pci-epf-vntb reserves the complete eDMA read direction
> and exposes those channels as a group. HDMA is not yet supported,
> keeping the series minimal.
> 
> The DMA feature is opt-in through the pci-epf-vntb configfs attribute
> 'dma_bar', which is -1 by default. Without it, no DMA extension is
> exported. pci-epf-ntb is unchanged.
> 
> Base and dependency
> ===================
> 
> This series is based on next-20260827 (3d83758432b5) plus:
> 
>   [PATCH v2 0/3] dmaengine: dw-edma: Prepare channels for remote use
>   https://lore.kernel.org/r/20260828163611.2691264-1-den@valinux.co.jp/
> 
> The prerequisite is for the dmaengine tree and is not part of
> pci/endpoint. linux-next provides a common buildable base for both
> series. Once the prerequisite lands, this series can be rebased onto
> pci/endpoint.
> 

Just to make it clear. This series has to build time dependency with the above
mentioned dmaengine series, right? The above statement is not very clear.

EPF patches LGTM! I'm waiting for an Ack from NTB maintainer for patch 6.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v3 0/7] PCI: endpoint: Remote DMA support via vNTB

[Koichiro Den]

On Thu, Sep 03, 2026 at 01:43:45PM +0200, Manivannan Sadhasivam wrote:
> On Tue, Sep 01, 2026 at 03:26:50AM +0900, Koichiro Den wrote:
> > Hi,
> > 
> > This series lets an RC use endpoint-local DesignWare eDMA read channels
> > through vNTB. It keeps the export within pci-epf-vntb and ntb_hw_epf
> > rather than defining a generic remote-DMA ABI or using a separate PCI
> > DMA function.
> > 
> > On the endpoint, pci-epf-vntb reserves the complete eDMA read direction
> > and exposes those channels as a group. HDMA is not yet supported,
> > keeping the series minimal.
> > 
> > The DMA feature is opt-in through the pci-epf-vntb configfs attribute
> > 'dma_bar', which is -1 by default. Without it, no DMA extension is
> > exported. pci-epf-ntb is unchanged.
> > 
> > Base and dependency
> > ===================
> > 
> > This series is based on next-20260827 (3d83758432b5) plus:
> > 
> >   [PATCH v2 0/3] dmaengine: dw-edma: Prepare channels for remote use
> >   https://lore.kernel.org/r/20260828163611.2691264-1-den@valinux.co.jp/
> > 
> > The prerequisite is for the dmaengine tree and is not part of
> > pci/endpoint. linux-next provides a common buildable base for both
> > series. Once the prerequisite lands, this series can be rebased onto
> > pci/endpoint.
> > 
> 
> Just to make it clear. This series has to build time dependency with the above
> mentioned dmaengine series, right? The above statement is not very clear.

Yes, this series strictly depends on the prereq series, and as you pointed out,
it's a built-time dependency as well. Sorry for the ambiguity.

> 
> EPF patches LGTM! I'm waiting for an Ack from NTB maintainer for patch 6.

Thanks for taking a look. Note that I have already sent v4:
https://lore.kernel.org/r/20260903082327.2345602-1-den@valinux.co.jp/
(One more dependency is added as noted in v4 cover letter..)

Best regards,
Koichiro

> 
> - Mani
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v3 4/7] PCI: endpoint: pci-epf-vntb: Export endpoint DMA channels

[Frank Li]

On Tue, Sep 01, 2026 at 03:26:54AM +0900, Koichiro Den wrote:
> An RC may use endpoint-local DMA read channels to transfer data directly
> to an endpoint DMA address once both sides agree to use them. Quiescing
> an unrolled eDMA channel disables its whole direction, so reserve the
> complete read direction and route its interrupts to the RC when dma_bar
> is configured.
>
> Describe the controller and per-channel descriptor memory in a private
> control-region extension.
>
> Add a dma_bar configfs attribute. An explicit BAR selection enables DMA
> export; leaving it unassigned keeps the feature disabled.
>
> Keep resources already assigned to a BAR in place, and map the rest
> through the selected dma_bar.
>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
> Changes in v3:
>   - Use dma_bar as the opt-in and drop use_dma and automatic BAR selection.
>   - Split DMA/MW BAR sharing and the documentation into separate patches.
>   - Avoid mixing cleanup helpers with goto-based error paths. (Sashiko)
>   - Use pci_epf_assign_bar_space() and back only uncovered BAR ranges.
>     (Sashiko)
>
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 575 +++++++++++++++++-
>  1 file changed, 567 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> index d12d134ce553..1842493c33b0 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> @@ -39,8 +39,12 @@
>  #include <linux/atomic.h>
>  #include <linux/bitops.h>
>  #include <linux/delay.h>
> +#include <linux/dma/edma.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/dmaengine.h>
>  #include <linux/io.h>
>  #include <linux/module.h>
> +#include <linux/overflow.h>
>  #include <linux/slab.h>
>
>  #include <linux/pci-ep-msi.h>
> @@ -56,6 +60,8 @@ static struct workqueue_struct *kpcintb_workqueue;
>  #define COMMAND_TEARDOWN_MW		4
>  #define COMMAND_LINK_UP			5
>  #define COMMAND_LINK_DOWN		6
> +#define COMMAND_CONFIGURE_DMA		7
> +#define COMMAND_TEARDOWN_DMA		8
>
>  #define COMMAND_STATUS_OK		1
>  #define COMMAND_STATUS_ERROR		2
> @@ -69,6 +75,10 @@ static struct workqueue_struct *kpcintb_workqueue;
>  #define MSIX_ENABLE			BIT(16)
>  #define MAX_MW				4
>
> +#define EPF_NTB_DMA_MAGIC		0x414d444e /* "NDMA": NTB DMA */
> +#define EPF_NTB_DMA_REVISION		1
> +#define EPF_NTB_DMA_TYPE_DW_EDMA	1
> +
>  /* Limit per-work execution to avoid monopolizing kworker on doorbell storms. */
>  #define VNTB_PEER_DB_WORK_BUDGET	5
>
> @@ -79,6 +89,7 @@ enum epf_ntb_bar {
>  	BAR_MW2,
>  	BAR_MW3,
>  	BAR_MW4,
> +	BAR_DMA,
>  	VNTB_BAR_NUM,
>  };
>
> @@ -91,6 +102,30 @@ enum epf_irq_slot {
>  #define MIN_DB_COUNT			(EPF_IRQ_DB_START + 1)
>  #define MAX_DB_COUNT			32
>
> +/* Private wire extension consumed by ntb_hw_epf. */
> +struct epf_ntb_dma_region_ctrl {
> +	u32 bar;
> +	u32 offset;
> +	u32 size;
> +} __packed;
> +
> +struct epf_ntb_dma_chan_ctrl {
> +	struct epf_ntb_dma_region_ctrl desc;
> +	u32 desc_addr_lo;
> +	u32 desc_addr_hi;
> +} __packed;
> +
> +struct epf_ntb_dma_ctrl {
> +	u32 magic;
> +	u16 revision;
> +	u16 length;
> +	u32 type;
> +	/* BAR range occupied by resources without a fixed BAR assignment. */
> +	struct epf_ntb_dma_region_ctrl submap;
> +	struct epf_ntb_dma_region_ctrl reg;
> +	struct epf_ntb_dma_chan_ctrl chan[EDMA_MAX_RD_CH];
> +} __packed;
> +
>  /*
>   * +--------------------------------------------------+ Base
>   * |                                                  |
> @@ -129,8 +164,21 @@ struct epf_ntb_ctrl {
>  	u32 db_entry_size;
>  	u32 db_data[MAX_DB_COUNT];
>  	u32 db_offset[MAX_DB_COUNT];
> +	struct epf_ntb_dma_ctrl dma;

I suggest move magic and revision to here intead of dma, in case we need
add more in future.

It is not good without version informaiton in this struct since historical
reason.

>  } __packed;
>
> +struct epf_ntb_dma {
> +	struct epf_ntb_dma_ctrl ctrl;
> +	struct dma_chan *dchan[EDMA_MAX_RD_CH];
> +	void *bar_scratch;
> +	dma_addr_t bar_scratch_phys;
> +	size_t bar_scratch_size;
> +	struct pci_epf_bar_submap submap[EDMA_MAX_RD_CH + 2];
> +	struct pci_epf_bar_submap *reg_submap;
> +	unsigned int num_submap;
> +	u16 rd_ch_cnt;
> +};
> +
>  struct epf_ntb {
>  	struct ntb_dev ntb;
>  	struct pci_epf *epf;
> @@ -159,6 +207,7 @@ struct epf_ntb {
>  	enum pci_barno epf_ntb_bar[VNTB_BAR_NUM];
>
>  	struct epf_ntb_ctrl *reg;
> +	struct epf_ntb_dma *dma;
>
>  	u32 *epf_db;
>
> @@ -211,7 +260,8 @@ static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
>  {
>  	int i;
>
> -	for (i = 0; i < VNTB_BAR_NUM; i++) {
> +	/* BAR_DMA is checked separately because it may share an MW BAR. */
> +	for (i = 0; i < BAR_DMA; i++) {
>  		if (ntb->epf_ntb_bar[i] == barno)
>  			return true;
>  	}
> @@ -219,6 +269,404 @@ static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
>  	return false;
>  }
>
> +static int epf_ntb_dma_validate_bar(struct epf_ntb *ntb,
> +				    const struct pci_epc_features *features)
> +{
> +	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
> +
> +	if (epf_ntb_is_bar_used(ntb, barno) ||
> +	    pci_epc_get_next_free_bar(features, barno) != barno)
> +		return -EINVAL;
> +
> +	return 0;
> +}
> +
> +struct epf_ntb_dma_filter {
> +	struct device *dev;
> +	int chan_id;
> +};
> +
> +static bool epf_ntb_dma_filter(struct dma_chan *chan, void *data)
> +{
> +	struct epf_ntb_dma_filter *filter = data;
> +
> +	return chan->device->dev == filter->dev &&
> +	       chan->chan_id == filter->chan_id;
> +}
> +
> +static int epf_ntb_dma_add_region(struct epf_ntb_dma *dma,
> +				  const struct pci_epc_aux_resource *resource,
> +				  dma_addr_t target_addr,
> +				  enum pci_barno barno, size_t align, u32 *next,
> +				  struct epf_ntb_dma_region_ctrl *region)
> +{
> +	struct pci_epf_bar_submap *submap;
> +	resource_size_t delta, map_size, size;
> +	dma_addr_t base;
> +
> +	if (!resource->size || resource->size > U32_MAX)
> +		return -EINVAL;
> +
> +	region->size = resource->size;
> +	if (resource->bar != NO_BAR) {
> +		if (resource->bar < BAR_0 || resource->bar > BAR_5 ||
> +		    resource->bar_offset > U32_MAX)
> +			return -EINVAL;
> +
> +		region->bar = resource->bar;
> +		region->offset = resource->bar_offset;
> +		return 0;
> +	}
> +	submap = &dma->submap[dma->num_submap];
> +
> +	/*
> +	 * Meet the EPC alignment requirement by mapping an aligned superset
> +	 * and advertising the resource after any leading padding.
> +	 */
> +	base = ALIGN_DOWN(target_addr, align);
> +	delta = target_addr - base;
> +	if (check_add_overflow(delta, resource->size, &size))
> +		return -EOVERFLOW;
> +	map_size = ALIGN(size, align);
> +	if (map_size < size || map_size > U32_MAX - *next)
> +		return -EOVERFLOW;
> +
> +	submap->phys_addr = base;
> +	submap->size = map_size;
> +	region->bar = barno;
> +	region->offset = *next + delta;
> +	*next += map_size;
> +	dma->num_submap++;
> +
> +	return 0;
> +}
> +
> +/* DW eDMA */
> +
> +static int epf_ntb_dw_edma_claim(struct device *dev, int chan_id,
> +				 struct dma_chan **dchan)
> +{
> +	enum dw_edma_ch_irq_mode mode = DW_EDMA_CH_IRQ_REMOTE;
> +	struct epf_ntb_dma_filter filter = {
> +		.dev = dev,
> +		.chan_id = chan_id,
> +	};
> +	struct dma_slave_config config = {
> +		.peripheral_config = &mode,
> +		.peripheral_size = sizeof(mode),
> +	};
> +	dma_cap_mask_t mask;
> +	struct dma_chan *chan;
> +	int ret;
> +
> +	dma_cap_zero(mask);
> +	dma_cap_set(DMA_SLAVE, mask);
> +	chan = dma_request_channel(mask, epf_ntb_dma_filter, &filter);
> +	if (!chan)
> +		return -EBUSY;
> +
> +	ret = dmaengine_slave_config(chan, &config);
> +	if (ret) {
> +		dma_release_channel(chan);
> +		return ret;
> +	}
> +
> +	*dchan = chan;
> +
> +	return 0;
> +}
> +
> +static void epf_ntb_dw_edma_release_channels(struct epf_ntb *ntb,
> +					     struct epf_ntb_dma *dma,
> +					     bool quiesce)
> +{
> +	unsigned int i;
> +	int ret;
> +
> +	if (quiesce) {
> +		/*
> +		 * RC programming has stopped and this EPF owns the complete read
> +		 * direction, so one termination quiesces the direction.
> +		 */
> +		ret = dmaengine_terminate_sync(dma->dchan[0]);
> +		if (ret)
> +			dev_warn(&ntb->epf->dev,
> +				 "failed to terminate remote DMA: %d\n", ret);
> +	}
> +
> +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> +		if (!dma->dchan[i])
> +			continue;
> +
> +		dma_release_channel(dma->dchan[i]);
> +	}
> +}
> +
> +static const struct pci_epc_aux_resource *
> +epf_ntb_dw_edma_find_desc(const struct pci_epc_aux_resource *resources,
> +			  unsigned int count, u16 chan_id)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < count; i++)
> +		if (resources[i].type == PCI_EPC_AUX_DMA_DESC_MEM &&
> +		    resources[i].u.dma_desc.chan_id == chan_id)
> +			return &resources[i];
> +
> +	return NULL;
> +}
> +
> +static int
> +epf_ntb_dw_edma_collect(struct epf_ntb *ntb,
> +			struct epf_ntb_dma *dma,
> +			const struct pci_epc_aux_resource *ctrl,
> +			const struct pci_epc_aux_resource *resources,
> +			unsigned int count)
> +{
> +	const struct pci_epc_features *features;
> +	const struct pci_epc_aux_resource *desc[EDMA_MAX_RD_CH];
> +	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
> +	struct device *dma_dev;
> +	bool needs_submap;
> +	unsigned int i;
> +	size_t align;
> +	u32 next = 0;
> +	int ret;
> +
> +	if (ctrl->u.dma_ctrl.reg_layout_data != EDMA_MF_EDMA_UNROLL)
> +		return -EOPNOTSUPP;
> +	if (ctrl->u.dma_ctrl.ep_to_rc_ch_cnt > EDMA_MAX_WR_CH ||
> +	    !ctrl->u.dma_ctrl.rc_to_ep_ch_cnt ||
> +	    ctrl->u.dma_ctrl.rc_to_ep_ch_cnt > EDMA_MAX_RD_CH)
> +		return -EINVAL;

this should never happen, it should validate at export resources side.

> +
> +	dma->rd_ch_cnt = ctrl->u.dma_ctrl.rc_to_ep_ch_cnt;

You export all channel to RC? suppose you only need export necessary one.

I need more time to understand this patch.

Frank
> +
> +	features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no,
> +					ntb->epf->vfunc_no);
> +	if (!features)
> +		return -EOPNOTSUPP;
> +
> +	align = features->align ?: 1;
> +	if (!is_power_of_2(align))
> +		return -EINVAL;
> +
> +	needs_submap = ctrl->bar == NO_BAR;
> +	/* DW eDMA static IDs place read channels after all write channels. */
> +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> +		u16 chan_id = ctrl->u.dma_ctrl.ep_to_rc_ch_cnt + i;
> +`
> +		desc[i] = epf_ntb_dw_edma_find_desc(resources, count, chan_id);
> +		if (!desc[i])
> +			return -EINVAL;
> +		needs_submap |= desc[i]->bar == NO_BAR;
> +	}
> +	if (needs_submap) {
> +		if (!features->subrange_mapping ||
> +		    !features->dynamic_inbound_mapping)
> +			return -EOPNOTSUPP;
> +		ret = epf_ntb_dma_validate_bar(ntb, features);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	dma->ctrl.magic = EPF_NTB_DMA_MAGIC;
> +	dma->ctrl.revision = EPF_NTB_DMA_REVISION;
> +	dma->ctrl.type = EPF_NTB_DMA_TYPE_DW_EDMA;
> +	dma->ctrl.submap.bar = U32_MAX;
> +	dma->ctrl.length = offsetof(struct epf_ntb_dma_ctrl,
> +				    chan[dma->rd_ch_cnt]);
> +
> +	if (ctrl->bar == NO_BAR)
> +		dma->reg_submap = &dma->submap[dma->num_submap];
> +	ret = epf_ntb_dma_add_region(dma, ctrl, ctrl->phys_addr,
> +				     barno, align, &next, &dma->ctrl.reg);
> +	if (ret)
> +		return ret;
> +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> +		struct epf_ntb_dma_chan_ctrl *chan = &dma->ctrl.chan[i];
> +		dma_addr_t dma_addr = desc[i]->u.dma_desc.dma_addr;
> +
> +		ret = epf_ntb_dma_add_region(dma, desc[i], dma_addr, barno,
> +					     align, &next,
> +					     &chan->desc);
> +		if (ret)
> +			return ret;
> +		chan->desc_addr_lo = lower_32_bits(dma_addr);
> +		chan->desc_addr_hi = upper_32_bits(dma_addr);
> +	}
> +	if (dma->num_submap) {
> +		dma->ctrl.submap.bar = barno;
> +		dma->ctrl.submap.size = next;
> +	}
> +
> +	dma_dev = ntb->epf->epc->dev.parent;
> +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> +		u16 chan_id = ctrl->u.dma_ctrl.ep_to_rc_ch_cnt + i;
> +
> +		ret = epf_ntb_dw_edma_claim(dma_dev, chan_id, &dma->dchan[i]);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +/* Common endpoint DMA */
> +
> +static void epf_ntb_dma_release_channels(struct epf_ntb *ntb,
> +					 struct epf_ntb_dma *dma,
> +					 bool quiesce)
> +{
> +	switch (dma->ctrl.type) {
> +	case EPF_NTB_DMA_TYPE_DW_EDMA:
> +		epf_ntb_dw_edma_release_channels(ntb, dma, quiesce);
> +		break;
> +	}
> +}
> +
> +static int epf_ntb_dma_collect(struct epf_ntb *ntb)
> +{
> +	const struct pci_epc_aux_resource *ctrl = NULL;
> +	struct device *dma_dev;
> +	dma_addr_t dma_addr;
> +	unsigned int i;
> +	int count, ret;
> +
> +	if (ntb->epf_ntb_bar[BAR_DMA] == NO_BAR)
> +		return 0;
> +
> +	count = pci_epc_get_aux_resources_count(ntb->epf->epc,
> +						ntb->epf->func_no,
> +						ntb->epf->vfunc_no);
> +	if (count <= 0)
> +		return count ?: -ENODEV;
> +
> +	struct pci_epc_aux_resource *resources __free(kfree) =
> +		kcalloc(count, sizeof(*resources), GFP_KERNEL);
> +	if (!resources)
> +		return -ENOMEM;
> +
> +	ret = pci_epc_get_aux_resources(ntb->epf->epc, ntb->epf->func_no,
> +					ntb->epf->vfunc_no, resources, count);
> +	if (ret)
> +		return ret;
> +
> +	for (i = 0; i < count; i++) {
> +		if (resources[i].type != PCI_EPC_AUX_DMA_CTRL_MMIO)
> +			continue;
> +		if (ctrl)
> +			return -EINVAL;
> +		ctrl = &resources[i];
> +	}
> +	if (!ctrl)
> +		return -ENODEV;
> +
> +	struct epf_ntb_dma *dma __free(kfree) =
> +		kzalloc(sizeof(*dma), GFP_KERNEL);
> +	if (!dma)
> +		return -ENOMEM;
> +
> +	switch (ctrl->u.dma_ctrl.reg_layout) {
> +	case PCI_EPC_AUX_DMA_REG_LAYOUT_DW_EDMA:
> +		ret = epf_ntb_dw_edma_collect(ntb, dma, ctrl, resources, count);
> +		break;
> +	default:
> +		return -EOPNOTSUPP;
> +	}
> +	if (ret) {
> +		epf_ntb_dma_release_channels(ntb, dma, false);
> +		return ret;
> +	}
> +
> +	/*
> +	 * CTRL_MMIO carries a CPU physical address, while DMA_DESC_MEM already
> +	 * carries an endpoint DMA address. Convert only the former.
> +	 */
> +	if (dma->reg_submap) {
> +		dma_dev = ntb->epf->epc->dev.parent;
> +		dma_addr = dma_map_resource(dma_dev,
> +					    dma->reg_submap->phys_addr,
> +					    dma->reg_submap->size,
> +					    DMA_BIDIRECTIONAL, 0);
> +		if (dma_mapping_error(dma_dev, dma_addr)) {
> +			epf_ntb_dma_release_channels(ntb, dma, false);
> +			return -EIO;
> +		}
> +		dma->reg_submap->phys_addr = dma_addr;
> +	}
> +
> +	ntb->dma = no_free_ptr(dma);
> +
> +	return 0;
> +}
> +
> +static void epf_ntb_dma_release(struct epf_ntb *ntb, bool quiesce)
> +{
> +	struct epf_ntb_dma *dma = ntb->dma;
> +	struct device *dev;
> +
> +	if (!dma)
> +		return;
> +
> +	epf_ntb_dma_release_channels(ntb, dma, quiesce);
> +	dev = ntb->epf->epc->dev.parent;
> +	if (dma->reg_submap)
> +		dma_unmap_resource(dev, dma->reg_submap->phys_addr,
> +				   dma->reg_submap->size, DMA_BIDIRECTIONAL, 0);
> +	if (dma->bar_scratch)
> +		dma_free_coherent(dev, dma->bar_scratch_size,
> +				  dma->bar_scratch, dma->bar_scratch_phys);
> +	kfree(dma);
> +	ntb->dma = NULL;
> +}
> +
> +static int epf_ntb_dma_set_bar(struct epf_ntb *ntb, bool active)
> +{
> +	struct pci_epf_bar_submap *old_submap;
> +	struct epf_ntb_dma *dma = ntb->dma;
> +	struct pci_epf_bar *bar;
> +	unsigned int old_num_submap;
> +	int restore, ret;
> +
> +	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
> +	old_submap = bar->submap;
> +	old_num_submap = bar->num_submap;
> +	bar->submap = active ? dma->submap : NULL;
> +	bar->num_submap = active ? dma->num_submap : 0;
> +
> +	ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
> +			      ntb->epf->vfunc_no, bar);
> +	if (!ret)
> +		return 0;
> +
> +	/* A failed dynamic update may have already removed the old mapping. */
> +	bar->submap = old_submap;
> +	bar->num_submap = old_num_submap;
> +	restore = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
> +				  ntb->epf->vfunc_no, bar);
> +	if (restore)
> +		dev_warn(&ntb->epf->dev,
> +			 "failed to restore DMA BAR mapping: %d\n", restore);
> +
> +	return ret;
> +}
> +
> +static int epf_ntb_dma_set_active(struct epf_ntb *ntb, bool active)
> +{
> +	struct epf_ntb_dma *dma = ntb->dma;
> +	struct pci_epf_bar *bar;
> +
> +	if (!dma || !dma->num_submap)
> +		return 0;
> +
> +	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
> +	if (active == !!bar->num_submap)
> +		return 0;
> +
> +	return epf_ntb_dma_set_bar(ntb, active);
> +}
> +
>  /**
>   * epf_ntb_configure_mw() - Configure the Outbound Address Space for VHOST
>   *   to access the memory window of HOST
> @@ -339,6 +787,13 @@ static void epf_ntb_cmd_handler(struct work_struct *work)
>  		epf_ntb_teardown_mw(ntb, argument);
>  		ctrl->command_status = COMMAND_STATUS_OK;
>  		break;
> +	case COMMAND_CONFIGURE_DMA:
> +	case COMMAND_TEARDOWN_DMA:
> +		ret = epf_ntb_dma_set_active(ntb,
> +					     command == COMMAND_CONFIGURE_DMA);
> +		ctrl->command_status = ret ? COMMAND_STATUS_ERROR :
> +					     COMMAND_STATUS_OK;
> +		break;
>  	case COMMAND_LINK_UP:
>  		ntb->linkup = true;
>  		ret = epf_ntb_link_up(ntb, true);
> @@ -459,9 +914,8 @@ static void epf_ntb_config_spad_bar_free(struct epf_ntb *ntb)
>   *   region
>   * @ntb: NTB device that facilitates communication between HOST and VHOST
>   *
> - * Allocate the Local Memory mentioned in the above diagram. The size of
> - * CONFIG REGION is sizeof(struct epf_ntb_ctrl) and size of SCRATCHPAD REGION
> - * is obtained from "spad-count" configfs entry.
> + * Allocate the control and scratchpad regions, omitting the optional DMA
> + * extension when no channels are exported.
>   *
>   * Returns: Zero for success, or an error code in case of failure
>   */
> @@ -481,7 +935,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
>  	barno = ntb->epf_ntb_bar[BAR_CONFIG];
>  	spad_count = ntb->spad_count;
>
> -	ctrl_size = ALIGN(sizeof(struct epf_ntb_ctrl), sizeof(u32));
> +	ctrl_size = ntb->dma ? sizeof(struct epf_ntb_ctrl) :
> +			       offsetof(struct epf_ntb_ctrl, dma);
> +	ctrl_size = ALIGN(ctrl_size, sizeof(u32));
>  	spad_size = 2 * spad_count * sizeof(u32);
>
>  	base = pci_epf_alloc_space(epf, ctrl_size + spad_size,
> @@ -507,6 +963,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
>  		ntb->reg->db_offset[i] = 0;
>  	}
>
> +	if (ntb->dma)
> +		ctrl->dma = ntb->dma->ctrl;
> +
>  	return 0;
>  }
>
> @@ -738,6 +1197,83 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
>
>  static void epf_ntb_mw_bar_clear(struct epf_ntb *ntb, int num_mws);
>
> +static int epf_ntb_dma_bar_init(struct epf_ntb *ntb)
> +{
> +	const struct pci_epc_features *features;
> +	struct epf_ntb_dma *dma = ntb->dma;
> +	struct device *dev = ntb->epf->epc->dev.parent;
> +	struct pci_epf_bar *bar;
> +	enum pci_barno barno;
> +	size_t backing_size;
> +	u32 mapped_size;
> +	int ret;
> +
> +	features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no,
> +					ntb->epf->vfunc_no);
> +	if (!features)
> +		return -EOPNOTSUPP;
> +
> +	barno = ntb->epf_ntb_bar[BAR_DMA];
> +	mapped_size = dma->ctrl.submap.size;
> +	/*
> +	 * Submaps cannot be installed until the host assigns the BAR address.
> +	 * Use address 0 for the temporary BAR Match Mode mapping, as is done
> +	 * for regular vNTB MW BARs.
> +	 */
> +	ret = pci_epf_assign_bar_space(ntb->epf, mapped_size, barno, features,
> +				       PRIMARY_INTERFACE, 0);
> +	if (ret)
> +		return ret;
> +
> +	bar = &ntb->epf->bar[barno];
> +	if (bar->size > U32_MAX)
> +		return -EOVERFLOW;
> +
> +	backing_size = bar->size - mapped_size;
> +	if (backing_size) {
> +		/* Back the BAR tail added by the power-of-two size rounding. */
> +		dma->bar_scratch = dma_alloc_coherent(dev, backing_size,
> +						      &dma->bar_scratch_phys,
> +						      GFP_KERNEL);
> +		if (!dma->bar_scratch)
> +			return -ENOMEM;
> +		dma->bar_scratch_size = backing_size;
> +		if (!IS_ALIGNED(dma->bar_scratch_phys, features->align ?: 1))
> +			return -EINVAL;
> +
> +		dma->submap[dma->num_submap++] = (struct pci_epf_bar_submap) {
> +			.phys_addr = dma->bar_scratch_phys,
> +			.size = backing_size,
> +		};
> +	}
> +
> +	return pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
> +			       ntb->epf->vfunc_no, bar);
> +}
> +
> +static void epf_ntb_dma_bar_clear(struct epf_ntb *ntb)
> +{
> +	struct epf_ntb_dma *dma = ntb->dma;
> +	struct pci_epf_bar *bar;
> +	enum pci_barno barno;
> +
> +	if (!dma || !dma->num_submap)
> +		return;
> +
> +	barno = ntb->epf_ntb_bar[BAR_DMA];
> +	bar = &ntb->epf->bar[barno];
> +	pci_epc_clear_bar(ntb->epf->epc, ntb->epf->func_no,
> +			  ntb->epf->vfunc_no, bar);
> +	bar->submap = NULL;
> +	bar->num_submap = 0;
> +	bar->phys_addr = 0;
> +	bar->addr = NULL;
> +	bar->size = 0;
> +	bar->mem_size = 0;
> +	bar->barno = 0;
> +	bar->flags = 0;
> +}
> +
>  /**
>   * epf_ntb_db_bar_clear() - Clear doorbell BAR and free memory
>   *   allocated in peer's outbound address space
> @@ -877,7 +1413,8 @@ static int epf_ntb_find_bar(struct epf_ntb *ntb,
>  		 * Verify if the BAR found is not already assigned
>  		 * through the provided configuration
>  		 */
> -		if (!epf_ntb_is_bar_used(ntb, barno))
> +		if (ntb->epf_ntb_bar[BAR_DMA] != barno &&
> +		    !epf_ntb_is_bar_used(ntb, barno))
>  			ntb->epf_ntb_bar[bar] = barno;
>
>  		barno += 1;
> @@ -976,11 +1513,19 @@ static int epf_ntb_epc_init(struct epf_ntb *ntb)
>  		goto err_mw_bar_init;
>  	}
>
> +	if (ntb->dma && ntb->dma->num_submap) {
> +		ret = epf_ntb_dma_bar_init(ntb);
> +		if (ret) {
> +			dev_err(dev, "DMA BAR init failed\n");
> +			goto err_dma_bar_init;
> +		}
> +	}
> +
>  	if (vfunc_no <= 1) {
>  		ret = pci_epc_write_header(epc, func_no, vfunc_no, epf->header);
>  		if (ret) {
>  			dev_err(dev, "Configuration header write failed\n");
> -			goto err_write_header;
> +			goto err_dma_bar_init;
>  		}
>  	}
>
> @@ -992,7 +1537,8 @@ static int epf_ntb_epc_init(struct epf_ntb *ntb)
>
>  	return 0;
>
> -err_write_header:
> +err_dma_bar_init:
> +	epf_ntb_dma_bar_clear(ntb);
>  	epf_ntb_mw_bar_clear(ntb, ntb->num_mws);
>  err_mw_bar_init:
>  	epf_ntb_db_bar_clear(ntb);
> @@ -1015,6 +1561,7 @@ static void epf_ntb_epc_cleanup(struct epf_ntb *ntb)
>  	disable_delayed_work_sync(&ntb->cmd_handler);
>  	disable_work_sync(&ntb->peer_db_work);
>  	atomic64_set(&ntb->peer_db_pending, 0);
> +	epf_ntb_dma_bar_clear(ntb);
>  	epf_ntb_mw_bar_clear(ntb, ntb->num_mws);
>  	epf_ntb_db_bar_clear(ntb);
>  	epf_ntb_config_sspad_bar_clear(ntb);
> @@ -1221,10 +1768,13 @@ EPF_NTB_BAR_R(mw3_bar, BAR_MW3)
>  EPF_NTB_BAR_W(mw3_bar, BAR_MW3)
>  EPF_NTB_BAR_R(mw4_bar, BAR_MW4)
>  EPF_NTB_BAR_W(mw4_bar, BAR_MW4)
> +EPF_NTB_BAR_R(dma_bar, BAR_DMA)
> +EPF_NTB_BAR_W(dma_bar, BAR_DMA)
>
>  CONFIGFS_ATTR(epf_ntb_, spad_count);
>  CONFIGFS_ATTR(epf_ntb_, db_count);
>  CONFIGFS_ATTR(epf_ntb_, num_mws);
> +CONFIGFS_ATTR(epf_ntb_, dma_bar);
>  CONFIGFS_ATTR(epf_ntb_, mw1);
>  CONFIGFS_ATTR(epf_ntb_, mw2);
>  CONFIGFS_ATTR(epf_ntb_, mw3);
> @@ -1243,6 +1793,7 @@ static struct configfs_attribute *epf_ntb_attrs[] = {
>  	&epf_ntb_attr_spad_count,
>  	&epf_ntb_attr_db_count,
>  	&epf_ntb_attr_num_mws,
> +	&epf_ntb_attr_dma_bar,
>  	&epf_ntb_attr_mw1,
>  	&epf_ntb_attr_mw2,
>  	&epf_ntb_attr_mw3,
> @@ -1743,6 +2294,12 @@ static int epf_ntb_bind(struct pci_epf *epf)
>  		return ret;
>  	}
>
> +	ret = epf_ntb_dma_collect(ntb);
> +	if (ret) {
> +		dev_err(dev, "Failed to prepare NTB DMA export\n");
> +		return ret;
> +	}
> +
>  	ret = epf_ntb_config_spad_bar_alloc(ntb);
>  	if (ret) {
>  		dev_err(dev, "Failed to allocate BAR memory\n");
> @@ -1779,6 +2336,7 @@ static int epf_ntb_bind(struct pci_epf *epf)
>  	epf_ntb_epc_cleanup(ntb);
>  err_bar_alloc:
>  	epf_ntb_config_spad_bar_free(ntb);
> +	epf_ntb_dma_release(ntb, false);
>
>  	return ret;
>  }
> @@ -1795,6 +2353,7 @@ static void epf_ntb_unbind(struct pci_epf *epf)
>
>  	epf_ntb_epc_cleanup(ntb);
>  	epf_ntb_config_spad_bar_free(ntb);
> +	epf_ntb_dma_release(ntb, true);
>
>  	pci_unregister_driver(&vntb_pci_driver);
>  }
> --
> 2.51.0
>

Re: [PATCH v3 4/7] PCI: endpoint: pci-epf-vntb: Export endpoint DMA channels

[Koichiro Den]

On Thu, Sep 03, 2026 at 04:20:26PM -0500, Frank Li wrote:
> On Tue, Sep 01, 2026 at 03:26:54AM +0900, Koichiro Den wrote:
> > An RC may use endpoint-local DMA read channels to transfer data directly
> > to an endpoint DMA address once both sides agree to use them. Quiescing
> > an unrolled eDMA channel disables its whole direction, so reserve the
> > complete read direction and route its interrupts to the RC when dma_bar
> > is configured.
> >
> > Describe the controller and per-channel descriptor memory in a private
> > control-region extension.
> >
> > Add a dma_bar configfs attribute. An explicit BAR selection enables DMA
> > export; leaving it unassigned keeps the feature disabled.
> >
> > Keep resources already assigned to a BAR in place, and map the rest
> > through the selected dma_bar.
> >
> > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > ---
> > Changes in v3:
> >   - Use dma_bar as the opt-in and drop use_dma and automatic BAR selection.
> >   - Split DMA/MW BAR sharing and the documentation into separate patches.
> >   - Avoid mixing cleanup helpers with goto-based error paths. (Sashiko)
> >   - Use pci_epf_assign_bar_space() and back only uncovered BAR ranges.
> >     (Sashiko)
> >
> >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 575 +++++++++++++++++-
> >  1 file changed, 567 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > index d12d134ce553..1842493c33b0 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > @@ -39,8 +39,12 @@
> >  #include <linux/atomic.h>
> >  #include <linux/bitops.h>
> >  #include <linux/delay.h>
> > +#include <linux/dma/edma.h>
> > +#include <linux/dma-mapping.h>
> > +#include <linux/dmaengine.h>
> >  #include <linux/io.h>
> >  #include <linux/module.h>
> > +#include <linux/overflow.h>
> >  #include <linux/slab.h>
> >
> >  #include <linux/pci-ep-msi.h>
> > @@ -56,6 +60,8 @@ static struct workqueue_struct *kpcintb_workqueue;
> >  #define COMMAND_TEARDOWN_MW		4
> >  #define COMMAND_LINK_UP			5
> >  #define COMMAND_LINK_DOWN		6
> > +#define COMMAND_CONFIGURE_DMA		7
> > +#define COMMAND_TEARDOWN_DMA		8
> >
> >  #define COMMAND_STATUS_OK		1
> >  #define COMMAND_STATUS_ERROR		2
> > @@ -69,6 +75,10 @@ static struct workqueue_struct *kpcintb_workqueue;
> >  #define MSIX_ENABLE			BIT(16)
> >  #define MAX_MW				4
> >
> > +#define EPF_NTB_DMA_MAGIC		0x414d444e /* "NDMA": NTB DMA */
> > +#define EPF_NTB_DMA_REVISION		1
> > +#define EPF_NTB_DMA_TYPE_DW_EDMA	1
> > +
> >  /* Limit per-work execution to avoid monopolizing kworker on doorbell storms. */
> >  #define VNTB_PEER_DB_WORK_BUDGET	5
> >
> > @@ -79,6 +89,7 @@ enum epf_ntb_bar {
> >  	BAR_MW2,
> >  	BAR_MW3,
> >  	BAR_MW4,
> > +	BAR_DMA,
> >  	VNTB_BAR_NUM,
> >  };
> >
> > @@ -91,6 +102,30 @@ enum epf_irq_slot {
> >  #define MIN_DB_COUNT			(EPF_IRQ_DB_START + 1)
> >  #define MAX_DB_COUNT			32
> >
> > +/* Private wire extension consumed by ntb_hw_epf. */
> > +struct epf_ntb_dma_region_ctrl {
> > +	u32 bar;
> > +	u32 offset;
> > +	u32 size;
> > +} __packed;
> > +
> > +struct epf_ntb_dma_chan_ctrl {
> > +	struct epf_ntb_dma_region_ctrl desc;
> > +	u32 desc_addr_lo;
> > +	u32 desc_addr_hi;
> > +} __packed;
> > +
> > +struct epf_ntb_dma_ctrl {
> > +	u32 magic;
> > +	u16 revision;
> > +	u16 length;
> > +	u32 type;
> > +	/* BAR range occupied by resources without a fixed BAR assignment. */
> > +	struct epf_ntb_dma_region_ctrl submap;
> > +	struct epf_ntb_dma_region_ctrl reg;
> > +	struct epf_ntb_dma_chan_ctrl chan[EDMA_MAX_RD_CH];
> > +} __packed;
> > +
> >  /*
> >   * +--------------------------------------------------+ Base
> >   * |                                                  |
> > @@ -129,8 +164,21 @@ struct epf_ntb_ctrl {
> >  	u32 db_entry_size;
> >  	u32 db_data[MAX_DB_COUNT];
> >  	u32 db_offset[MAX_DB_COUNT];
> > +	struct epf_ntb_dma_ctrl dma;
> 
> I suggest move magic and revision to here intead of dma, in case we need
> add more in future.
> 
> It is not good without version informaiton in this struct since historical
> reason.
> 
> >  } __packed;
> >
> > +struct epf_ntb_dma {
> > +	struct epf_ntb_dma_ctrl ctrl;
> > +	struct dma_chan *dchan[EDMA_MAX_RD_CH];
> > +	void *bar_scratch;
> > +	dma_addr_t bar_scratch_phys;
> > +	size_t bar_scratch_size;
> > +	struct pci_epf_bar_submap submap[EDMA_MAX_RD_CH + 2];
> > +	struct pci_epf_bar_submap *reg_submap;
> > +	unsigned int num_submap;
> > +	u16 rd_ch_cnt;
> > +};
> > +
> >  struct epf_ntb {
> >  	struct ntb_dev ntb;
> >  	struct pci_epf *epf;
> > @@ -159,6 +207,7 @@ struct epf_ntb {
> >  	enum pci_barno epf_ntb_bar[VNTB_BAR_NUM];
> >
> >  	struct epf_ntb_ctrl *reg;
> > +	struct epf_ntb_dma *dma;
> >
> >  	u32 *epf_db;
> >
> > @@ -211,7 +260,8 @@ static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
> >  {
> >  	int i;
> >
> > -	for (i = 0; i < VNTB_BAR_NUM; i++) {
> > +	/* BAR_DMA is checked separately because it may share an MW BAR. */
> > +	for (i = 0; i < BAR_DMA; i++) {
> >  		if (ntb->epf_ntb_bar[i] == barno)
> >  			return true;
> >  	}
> > @@ -219,6 +269,404 @@ static bool epf_ntb_is_bar_used(struct epf_ntb *ntb,
> >  	return false;
> >  }
> >
> > +static int epf_ntb_dma_validate_bar(struct epf_ntb *ntb,
> > +				    const struct pci_epc_features *features)
> > +{
> > +	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
> > +
> > +	if (epf_ntb_is_bar_used(ntb, barno) ||
> > +	    pci_epc_get_next_free_bar(features, barno) != barno)
> > +		return -EINVAL;
> > +
> > +	return 0;
> > +}
> > +
> > +struct epf_ntb_dma_filter {
> > +	struct device *dev;
> > +	int chan_id;
> > +};
> > +
> > +static bool epf_ntb_dma_filter(struct dma_chan *chan, void *data)
> > +{
> > +	struct epf_ntb_dma_filter *filter = data;
> > +
> > +	return chan->device->dev == filter->dev &&
> > +	       chan->chan_id == filter->chan_id;
> > +}
> > +
> > +static int epf_ntb_dma_add_region(struct epf_ntb_dma *dma,
> > +				  const struct pci_epc_aux_resource *resource,
> > +				  dma_addr_t target_addr,
> > +				  enum pci_barno barno, size_t align, u32 *next,
> > +				  struct epf_ntb_dma_region_ctrl *region)
> > +{
> > +	struct pci_epf_bar_submap *submap;
> > +	resource_size_t delta, map_size, size;
> > +	dma_addr_t base;
> > +
> > +	if (!resource->size || resource->size > U32_MAX)
> > +		return -EINVAL;
> > +
> > +	region->size = resource->size;
> > +	if (resource->bar != NO_BAR) {
> > +		if (resource->bar < BAR_0 || resource->bar > BAR_5 ||
> > +		    resource->bar_offset > U32_MAX)
> > +			return -EINVAL;
> > +
> > +		region->bar = resource->bar;
> > +		region->offset = resource->bar_offset;
> > +		return 0;
> > +	}
> > +	submap = &dma->submap[dma->num_submap];
> > +
> > +	/*
> > +	 * Meet the EPC alignment requirement by mapping an aligned superset
> > +	 * and advertising the resource after any leading padding.
> > +	 */
> > +	base = ALIGN_DOWN(target_addr, align);
> > +	delta = target_addr - base;
> > +	if (check_add_overflow(delta, resource->size, &size))
> > +		return -EOVERFLOW;
> > +	map_size = ALIGN(size, align);
> > +	if (map_size < size || map_size > U32_MAX - *next)
> > +		return -EOVERFLOW;
> > +
> > +	submap->phys_addr = base;
> > +	submap->size = map_size;
> > +	region->bar = barno;
> > +	region->offset = *next + delta;
> > +	*next += map_size;
> > +	dma->num_submap++;
> > +
> > +	return 0;
> > +}
> > +
> > +/* DW eDMA */
> > +
> > +static int epf_ntb_dw_edma_claim(struct device *dev, int chan_id,
> > +				 struct dma_chan **dchan)
> > +{
> > +	enum dw_edma_ch_irq_mode mode = DW_EDMA_CH_IRQ_REMOTE;
> > +	struct epf_ntb_dma_filter filter = {
> > +		.dev = dev,
> > +		.chan_id = chan_id,
> > +	};
> > +	struct dma_slave_config config = {
> > +		.peripheral_config = &mode,
> > +		.peripheral_size = sizeof(mode),
> > +	};
> > +	dma_cap_mask_t mask;
> > +	struct dma_chan *chan;
> > +	int ret;
> > +
> > +	dma_cap_zero(mask);
> > +	dma_cap_set(DMA_SLAVE, mask);
> > +	chan = dma_request_channel(mask, epf_ntb_dma_filter, &filter);
> > +	if (!chan)
> > +		return -EBUSY;
> > +
> > +	ret = dmaengine_slave_config(chan, &config);
> > +	if (ret) {
> > +		dma_release_channel(chan);
> > +		return ret;
> > +	}
> > +
> > +	*dchan = chan;
> > +
> > +	return 0;
> > +}
> > +
> > +static void epf_ntb_dw_edma_release_channels(struct epf_ntb *ntb,
> > +					     struct epf_ntb_dma *dma,
> > +					     bool quiesce)
> > +{
> > +	unsigned int i;
> > +	int ret;
> > +
> > +	if (quiesce) {
> > +		/*
> > +		 * RC programming has stopped and this EPF owns the complete read
> > +		 * direction, so one termination quiesces the direction.
> > +		 */
> > +		ret = dmaengine_terminate_sync(dma->dchan[0]);
> > +		if (ret)
> > +			dev_warn(&ntb->epf->dev,
> > +				 "failed to terminate remote DMA: %d\n", ret);
> > +	}
> > +
> > +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> > +		if (!dma->dchan[i])
> > +			continue;
> > +
> > +		dma_release_channel(dma->dchan[i]);
> > +	}
> > +}
> > +
> > +static const struct pci_epc_aux_resource *
> > +epf_ntb_dw_edma_find_desc(const struct pci_epc_aux_resource *resources,
> > +			  unsigned int count, u16 chan_id)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < count; i++)
> > +		if (resources[i].type == PCI_EPC_AUX_DMA_DESC_MEM &&
> > +		    resources[i].u.dma_desc.chan_id == chan_id)
> > +			return &resources[i];
> > +
> > +	return NULL;
> > +}
> > +
> > +static int
> > +epf_ntb_dw_edma_collect(struct epf_ntb *ntb,
> > +			struct epf_ntb_dma *dma,
> > +			const struct pci_epc_aux_resource *ctrl,
> > +			const struct pci_epc_aux_resource *resources,
> > +			unsigned int count)
> > +{
> > +	const struct pci_epc_features *features;
> > +	const struct pci_epc_aux_resource *desc[EDMA_MAX_RD_CH];
> > +	enum pci_barno barno = ntb->epf_ntb_bar[BAR_DMA];
> > +	struct device *dma_dev;
> > +	bool needs_submap;
> > +	unsigned int i;
> > +	size_t align;
> > +	u32 next = 0;
> > +	int ret;
> > +
> > +	if (ctrl->u.dma_ctrl.reg_layout_data != EDMA_MF_EDMA_UNROLL)
> > +		return -EOPNOTSUPP;

...

> > +	if (ctrl->u.dma_ctrl.ep_to_rc_ch_cnt > EDMA_MAX_WR_CH ||
> > +	    !ctrl->u.dma_ctrl.rc_to_ep_ch_cnt ||
> > +	    ctrl->u.dma_ctrl.rc_to_ep_ch_cnt > EDMA_MAX_RD_CH)
> > +		return -EINVAL;
> 
> this should never happen, it should validate at export resources side.

Yes. I was on the fence about these checks. I thought this clarified the
requirements and made it future-proof, but it might be too defensive.
I can drop it.

> 
> > +
> > +	dma->rd_ch_cnt = ctrl->u.dma_ctrl.rc_to_ep_ch_cnt;
> 
> You export all channel to RC? suppose you only need export necessary one.

Yes, all read channels. For eDMA (excluding HDMA), per-direction registers
cannot be safely shared between EP and RC users, so this exports either none or
all of its channels. As noted in the cover letter, this series does not include
HDMA support.

Thanks for taking time to review this.

Best regards,
Koichiro

> 
> I need more time to understand this patch.
> 
> Frank
> > +
> > +	features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no,
> > +					ntb->epf->vfunc_no);
> > +	if (!features)
> > +		return -EOPNOTSUPP;
> > +
> > +	align = features->align ?: 1;
> > +	if (!is_power_of_2(align))
> > +		return -EINVAL;
> > +
> > +	needs_submap = ctrl->bar == NO_BAR;
> > +	/* DW eDMA static IDs place read channels after all write channels. */
> > +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> > +		u16 chan_id = ctrl->u.dma_ctrl.ep_to_rc_ch_cnt + i;
> > +`
> > +		desc[i] = epf_ntb_dw_edma_find_desc(resources, count, chan_id);
> > +		if (!desc[i])
> > +			return -EINVAL;
> > +		needs_submap |= desc[i]->bar == NO_BAR;
> > +	}
> > +	if (needs_submap) {
> > +		if (!features->subrange_mapping ||
> > +		    !features->dynamic_inbound_mapping)
> > +			return -EOPNOTSUPP;
> > +		ret = epf_ntb_dma_validate_bar(ntb, features);
> > +		if (ret)
> > +			return ret;
> > +	}
> > +
> > +	dma->ctrl.magic = EPF_NTB_DMA_MAGIC;
> > +	dma->ctrl.revision = EPF_NTB_DMA_REVISION;
> > +	dma->ctrl.type = EPF_NTB_DMA_TYPE_DW_EDMA;
> > +	dma->ctrl.submap.bar = U32_MAX;
> > +	dma->ctrl.length = offsetof(struct epf_ntb_dma_ctrl,
> > +				    chan[dma->rd_ch_cnt]);
> > +
> > +	if (ctrl->bar == NO_BAR)
> > +		dma->reg_submap = &dma->submap[dma->num_submap];
> > +	ret = epf_ntb_dma_add_region(dma, ctrl, ctrl->phys_addr,
> > +				     barno, align, &next, &dma->ctrl.reg);
> > +	if (ret)
> > +		return ret;
> > +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> > +		struct epf_ntb_dma_chan_ctrl *chan = &dma->ctrl.chan[i];
> > +		dma_addr_t dma_addr = desc[i]->u.dma_desc.dma_addr;
> > +
> > +		ret = epf_ntb_dma_add_region(dma, desc[i], dma_addr, barno,
> > +					     align, &next,
> > +					     &chan->desc);
> > +		if (ret)
> > +			return ret;
> > +		chan->desc_addr_lo = lower_32_bits(dma_addr);
> > +		chan->desc_addr_hi = upper_32_bits(dma_addr);
> > +	}
> > +	if (dma->num_submap) {
> > +		dma->ctrl.submap.bar = barno;
> > +		dma->ctrl.submap.size = next;
> > +	}
> > +
> > +	dma_dev = ntb->epf->epc->dev.parent;
> > +	for (i = 0; i < dma->rd_ch_cnt; i++) {
> > +		u16 chan_id = ctrl->u.dma_ctrl.ep_to_rc_ch_cnt + i;
> > +
> > +		ret = epf_ntb_dw_edma_claim(dma_dev, chan_id, &dma->dchan[i]);
> > +		if (ret)
> > +			return ret;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/* Common endpoint DMA */
> > +
> > +static void epf_ntb_dma_release_channels(struct epf_ntb *ntb,
> > +					 struct epf_ntb_dma *dma,
> > +					 bool quiesce)
> > +{
> > +	switch (dma->ctrl.type) {
> > +	case EPF_NTB_DMA_TYPE_DW_EDMA:
> > +		epf_ntb_dw_edma_release_channels(ntb, dma, quiesce);
> > +		break;
> > +	}
> > +}
> > +
> > +static int epf_ntb_dma_collect(struct epf_ntb *ntb)
> > +{
> > +	const struct pci_epc_aux_resource *ctrl = NULL;
> > +	struct device *dma_dev;
> > +	dma_addr_t dma_addr;
> > +	unsigned int i;
> > +	int count, ret;
> > +
> > +	if (ntb->epf_ntb_bar[BAR_DMA] == NO_BAR)
> > +		return 0;
> > +
> > +	count = pci_epc_get_aux_resources_count(ntb->epf->epc,
> > +						ntb->epf->func_no,
> > +						ntb->epf->vfunc_no);
> > +	if (count <= 0)
> > +		return count ?: -ENODEV;
> > +
> > +	struct pci_epc_aux_resource *resources __free(kfree) =
> > +		kcalloc(count, sizeof(*resources), GFP_KERNEL);
> > +	if (!resources)
> > +		return -ENOMEM;
> > +
> > +	ret = pci_epc_get_aux_resources(ntb->epf->epc, ntb->epf->func_no,
> > +					ntb->epf->vfunc_no, resources, count);
> > +	if (ret)
> > +		return ret;
> > +
> > +	for (i = 0; i < count; i++) {
> > +		if (resources[i].type != PCI_EPC_AUX_DMA_CTRL_MMIO)
> > +			continue;
> > +		if (ctrl)
> > +			return -EINVAL;
> > +		ctrl = &resources[i];
> > +	}
> > +	if (!ctrl)
> > +		return -ENODEV;
> > +
> > +	struct epf_ntb_dma *dma __free(kfree) =
> > +		kzalloc(sizeof(*dma), GFP_KERNEL);
> > +	if (!dma)
> > +		return -ENOMEM;
> > +
> > +	switch (ctrl->u.dma_ctrl.reg_layout) {
> > +	case PCI_EPC_AUX_DMA_REG_LAYOUT_DW_EDMA:
> > +		ret = epf_ntb_dw_edma_collect(ntb, dma, ctrl, resources, count);
> > +		break;
> > +	default:
> > +		return -EOPNOTSUPP;
> > +	}
> > +	if (ret) {
> > +		epf_ntb_dma_release_channels(ntb, dma, false);
> > +		return ret;
> > +	}
> > +
> > +	/*
> > +	 * CTRL_MMIO carries a CPU physical address, while DMA_DESC_MEM already
> > +	 * carries an endpoint DMA address. Convert only the former.
> > +	 */
> > +	if (dma->reg_submap) {
> > +		dma_dev = ntb->epf->epc->dev.parent;
> > +		dma_addr = dma_map_resource(dma_dev,
> > +					    dma->reg_submap->phys_addr,
> > +					    dma->reg_submap->size,
> > +					    DMA_BIDIRECTIONAL, 0);
> > +		if (dma_mapping_error(dma_dev, dma_addr)) {
> > +			epf_ntb_dma_release_channels(ntb, dma, false);
> > +			return -EIO;
> > +		}
> > +		dma->reg_submap->phys_addr = dma_addr;
> > +	}
> > +
> > +	ntb->dma = no_free_ptr(dma);
> > +
> > +	return 0;
> > +}
> > +
> > +static void epf_ntb_dma_release(struct epf_ntb *ntb, bool quiesce)
> > +{
> > +	struct epf_ntb_dma *dma = ntb->dma;
> > +	struct device *dev;
> > +
> > +	if (!dma)
> > +		return;
> > +
> > +	epf_ntb_dma_release_channels(ntb, dma, quiesce);
> > +	dev = ntb->epf->epc->dev.parent;
> > +	if (dma->reg_submap)
> > +		dma_unmap_resource(dev, dma->reg_submap->phys_addr,
> > +				   dma->reg_submap->size, DMA_BIDIRECTIONAL, 0);
> > +	if (dma->bar_scratch)
> > +		dma_free_coherent(dev, dma->bar_scratch_size,
> > +				  dma->bar_scratch, dma->bar_scratch_phys);
> > +	kfree(dma);
> > +	ntb->dma = NULL;
> > +}
> > +
> > +static int epf_ntb_dma_set_bar(struct epf_ntb *ntb, bool active)
> > +{
> > +	struct pci_epf_bar_submap *old_submap;
> > +	struct epf_ntb_dma *dma = ntb->dma;
> > +	struct pci_epf_bar *bar;
> > +	unsigned int old_num_submap;
> > +	int restore, ret;
> > +
> > +	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
> > +	old_submap = bar->submap;
> > +	old_num_submap = bar->num_submap;
> > +	bar->submap = active ? dma->submap : NULL;
> > +	bar->num_submap = active ? dma->num_submap : 0;
> > +
> > +	ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
> > +			      ntb->epf->vfunc_no, bar);
> > +	if (!ret)
> > +		return 0;
> > +
> > +	/* A failed dynamic update may have already removed the old mapping. */
> > +	bar->submap = old_submap;
> > +	bar->num_submap = old_num_submap;
> > +	restore = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
> > +				  ntb->epf->vfunc_no, bar);
> > +	if (restore)
> > +		dev_warn(&ntb->epf->dev,
> > +			 "failed to restore DMA BAR mapping: %d\n", restore);
> > +
> > +	return ret;
> > +}
> > +
> > +static int epf_ntb_dma_set_active(struct epf_ntb *ntb, bool active)
> > +{
> > +	struct epf_ntb_dma *dma = ntb->dma;
> > +	struct pci_epf_bar *bar;
> > +
> > +	if (!dma || !dma->num_submap)
> > +		return 0;
> > +
> > +	bar = &ntb->epf->bar[ntb->epf_ntb_bar[BAR_DMA]];
> > +	if (active == !!bar->num_submap)
> > +		return 0;
> > +
> > +	return epf_ntb_dma_set_bar(ntb, active);
> > +}
> > +
> >  /**
> >   * epf_ntb_configure_mw() - Configure the Outbound Address Space for VHOST
> >   *   to access the memory window of HOST
> > @@ -339,6 +787,13 @@ static void epf_ntb_cmd_handler(struct work_struct *work)
> >  		epf_ntb_teardown_mw(ntb, argument);
> >  		ctrl->command_status = COMMAND_STATUS_OK;
> >  		break;
> > +	case COMMAND_CONFIGURE_DMA:
> > +	case COMMAND_TEARDOWN_DMA:
> > +		ret = epf_ntb_dma_set_active(ntb,
> > +					     command == COMMAND_CONFIGURE_DMA);
> > +		ctrl->command_status = ret ? COMMAND_STATUS_ERROR :
> > +					     COMMAND_STATUS_OK;
> > +		break;
> >  	case COMMAND_LINK_UP:
> >  		ntb->linkup = true;
> >  		ret = epf_ntb_link_up(ntb, true);
> > @@ -459,9 +914,8 @@ static void epf_ntb_config_spad_bar_free(struct epf_ntb *ntb)
> >   *   region
> >   * @ntb: NTB device that facilitates communication between HOST and VHOST
> >   *
> > - * Allocate the Local Memory mentioned in the above diagram. The size of
> > - * CONFIG REGION is sizeof(struct epf_ntb_ctrl) and size of SCRATCHPAD REGION
> > - * is obtained from "spad-count" configfs entry.
> > + * Allocate the control and scratchpad regions, omitting the optional DMA
> > + * extension when no channels are exported.
> >   *
> >   * Returns: Zero for success, or an error code in case of failure
> >   */
> > @@ -481,7 +935,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
> >  	barno = ntb->epf_ntb_bar[BAR_CONFIG];
> >  	spad_count = ntb->spad_count;
> >
> > -	ctrl_size = ALIGN(sizeof(struct epf_ntb_ctrl), sizeof(u32));
> > +	ctrl_size = ntb->dma ? sizeof(struct epf_ntb_ctrl) :
> > +			       offsetof(struct epf_ntb_ctrl, dma);
> > +	ctrl_size = ALIGN(ctrl_size, sizeof(u32));
> >  	spad_size = 2 * spad_count * sizeof(u32);
> >
> >  	base = pci_epf_alloc_space(epf, ctrl_size + spad_size,
> > @@ -507,6 +963,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
> >  		ntb->reg->db_offset[i] = 0;
> >  	}
> >
> > +	if (ntb->dma)
> > +		ctrl->dma = ntb->dma->ctrl;
> > +
> >  	return 0;
> >  }
> >
> > @@ -738,6 +1197,83 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
> >
> >  static void epf_ntb_mw_bar_clear(struct epf_ntb *ntb, int num_mws);
> >
> > +static int epf_ntb_dma_bar_init(struct epf_ntb *ntb)
> > +{
> > +	const struct pci_epc_features *features;
> > +	struct epf_ntb_dma *dma = ntb->dma;
> > +	struct device *dev = ntb->epf->epc->dev.parent;
> > +	struct pci_epf_bar *bar;
> > +	enum pci_barno barno;
> > +	size_t backing_size;
> > +	u32 mapped_size;
> > +	int ret;
> > +
> > +	features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no,
> > +					ntb->epf->vfunc_no);
> > +	if (!features)
> > +		return -EOPNOTSUPP;
> > +
> > +	barno = ntb->epf_ntb_bar[BAR_DMA];
> > +	mapped_size = dma->ctrl.submap.size;
> > +	/*
> > +	 * Submaps cannot be installed until the host assigns the BAR address.
> > +	 * Use address 0 for the temporary BAR Match Mode mapping, as is done
> > +	 * for regular vNTB MW BARs.
> > +	 */
> > +	ret = pci_epf_assign_bar_space(ntb->epf, mapped_size, barno, features,
> > +				       PRIMARY_INTERFACE, 0);
> > +	if (ret)
> > +		return ret;
> > +
> > +	bar = &ntb->epf->bar[barno];
> > +	if (bar->size > U32_MAX)
> > +		return -EOVERFLOW;
> > +
> > +	backing_size = bar->size - mapped_size;
> > +	if (backing_size) {
> > +		/* Back the BAR tail added by the power-of-two size rounding. */
> > +		dma->bar_scratch = dma_alloc_coherent(dev, backing_size,
> > +						      &dma->bar_scratch_phys,
> > +						      GFP_KERNEL);
> > +		if (!dma->bar_scratch)
> > +			return -ENOMEM;
> > +		dma->bar_scratch_size = backing_size;
> > +		if (!IS_ALIGNED(dma->bar_scratch_phys, features->align ?: 1))
> > +			return -EINVAL;
> > +
> > +		dma->submap[dma->num_submap++] = (struct pci_epf_bar_submap) {
> > +			.phys_addr = dma->bar_scratch_phys,
> > +			.size = backing_size,
> > +		};
> > +	}
> > +
> > +	return pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no,
> > +			       ntb->epf->vfunc_no, bar);
> > +}
> > +
> > +static void epf_ntb_dma_bar_clear(struct epf_ntb *ntb)
> > +{
> > +	struct epf_ntb_dma *dma = ntb->dma;
> > +	struct pci_epf_bar *bar;
> > +	enum pci_barno barno;
> > +
> > +	if (!dma || !dma->num_submap)
> > +		return;
> > +
> > +	barno = ntb->epf_ntb_bar[BAR_DMA];
> > +	bar = &ntb->epf->bar[barno];
> > +	pci_epc_clear_bar(ntb->epf->epc, ntb->epf->func_no,
> > +			  ntb->epf->vfunc_no, bar);
> > +	bar->submap = NULL;
> > +	bar->num_submap = 0;
> > +	bar->phys_addr = 0;
> > +	bar->addr = NULL;
> > +	bar->size = 0;
> > +	bar->mem_size = 0;
> > +	bar->barno = 0;
> > +	bar->flags = 0;
> > +}
> > +
> >  /**
> >   * epf_ntb_db_bar_clear() - Clear doorbell BAR and free memory
> >   *   allocated in peer's outbound address space
> > @@ -877,7 +1413,8 @@ static int epf_ntb_find_bar(struct epf_ntb *ntb,
> >  		 * Verify if the BAR found is not already assigned
> >  		 * through the provided configuration
> >  		 */
> > -		if (!epf_ntb_is_bar_used(ntb, barno))
> > +		if (ntb->epf_ntb_bar[BAR_DMA] != barno &&
> > +		    !epf_ntb_is_bar_used(ntb, barno))
> >  			ntb->epf_ntb_bar[bar] = barno;
> >
> >  		barno += 1;
> > @@ -976,11 +1513,19 @@ static int epf_ntb_epc_init(struct epf_ntb *ntb)
> >  		goto err_mw_bar_init;
> >  	}
> >
> > +	if (ntb->dma && ntb->dma->num_submap) {
> > +		ret = epf_ntb_dma_bar_init(ntb);
> > +		if (ret) {
> > +			dev_err(dev, "DMA BAR init failed\n");
> > +			goto err_dma_bar_init;
> > +		}
> > +	}
> > +
> >  	if (vfunc_no <= 1) {
> >  		ret = pci_epc_write_header(epc, func_no, vfunc_no, epf->header);
> >  		if (ret) {
> >  			dev_err(dev, "Configuration header write failed\n");
> > -			goto err_write_header;
> > +			goto err_dma_bar_init;
> >  		}
> >  	}
> >
> > @@ -992,7 +1537,8 @@ static int epf_ntb_epc_init(struct epf_ntb *ntb)
> >
> >  	return 0;
> >
> > -err_write_header:
> > +err_dma_bar_init:
> > +	epf_ntb_dma_bar_clear(ntb);
> >  	epf_ntb_mw_bar_clear(ntb, ntb->num_mws);
> >  err_mw_bar_init:
> >  	epf_ntb_db_bar_clear(ntb);
> > @@ -1015,6 +1561,7 @@ static void epf_ntb_epc_cleanup(struct epf_ntb *ntb)
> >  	disable_delayed_work_sync(&ntb->cmd_handler);
> >  	disable_work_sync(&ntb->peer_db_work);
> >  	atomic64_set(&ntb->peer_db_pending, 0);
> > +	epf_ntb_dma_bar_clear(ntb);
> >  	epf_ntb_mw_bar_clear(ntb, ntb->num_mws);
> >  	epf_ntb_db_bar_clear(ntb);
> >  	epf_ntb_config_sspad_bar_clear(ntb);
> > @@ -1221,10 +1768,13 @@ EPF_NTB_BAR_R(mw3_bar, BAR_MW3)
> >  EPF_NTB_BAR_W(mw3_bar, BAR_MW3)
> >  EPF_NTB_BAR_R(mw4_bar, BAR_MW4)
> >  EPF_NTB_BAR_W(mw4_bar, BAR_MW4)
> > +EPF_NTB_BAR_R(dma_bar, BAR_DMA)
> > +EPF_NTB_BAR_W(dma_bar, BAR_DMA)
> >
> >  CONFIGFS_ATTR(epf_ntb_, spad_count);
> >  CONFIGFS_ATTR(epf_ntb_, db_count);
> >  CONFIGFS_ATTR(epf_ntb_, num_mws);
> > +CONFIGFS_ATTR(epf_ntb_, dma_bar);
> >  CONFIGFS_ATTR(epf_ntb_, mw1);
> >  CONFIGFS_ATTR(epf_ntb_, mw2);
> >  CONFIGFS_ATTR(epf_ntb_, mw3);
> > @@ -1243,6 +1793,7 @@ static struct configfs_attribute *epf_ntb_attrs[] = {
> >  	&epf_ntb_attr_spad_count,
> >  	&epf_ntb_attr_db_count,
> >  	&epf_ntb_attr_num_mws,
> > +	&epf_ntb_attr_dma_bar,
> >  	&epf_ntb_attr_mw1,
> >  	&epf_ntb_attr_mw2,
> >  	&epf_ntb_attr_mw3,
> > @@ -1743,6 +2294,12 @@ static int epf_ntb_bind(struct pci_epf *epf)
> >  		return ret;
> >  	}
> >
> > +	ret = epf_ntb_dma_collect(ntb);
> > +	if (ret) {
> > +		dev_err(dev, "Failed to prepare NTB DMA export\n");
> > +		return ret;
> > +	}
> > +
> >  	ret = epf_ntb_config_spad_bar_alloc(ntb);
> >  	if (ret) {
> >  		dev_err(dev, "Failed to allocate BAR memory\n");
> > @@ -1779,6 +2336,7 @@ static int epf_ntb_bind(struct pci_epf *epf)
> >  	epf_ntb_epc_cleanup(ntb);
> >  err_bar_alloc:
> >  	epf_ntb_config_spad_bar_free(ntb);
> > +	epf_ntb_dma_release(ntb, false);
> >
> >  	return ret;
> >  }
> > @@ -1795,6 +2353,7 @@ static void epf_ntb_unbind(struct pci_epf *epf)
> >
> >  	epf_ntb_epc_cleanup(ntb);
> >  	epf_ntb_config_spad_bar_free(ntb);
> > +	epf_ntb_dma_release(ntb, true);
> >
> >  	pci_unregister_driver(&vntb_pci_driver);
> >  }
> > --
> > 2.51.0
> >