* [PATCH v10 1/7] PCI: endpoint: Add auxiliary resource query API
2026-03-02 7:14 [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
@ 2026-03-02 7:14 ` Koichiro Den
2026-03-21 14:17 ` Manivannan Sadhasivam
2026-03-02 7:14 ` [PATCH v10 2/7] PCI: dwc: Record integrated eDMA register window Koichiro Den
` (5 subsequent siblings)
6 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-02 7:14 UTC (permalink / raw)
To: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Frank Li, Bhanu Seshu Kumar Valluri,
Marco Crivellari, Shin'ichiro Kawasaki, Manikanta Maddireddy
Cc: linux-pci, linux-kernel, ntb
Endpoint controller drivers may integrate auxiliary blocks (e.g. DMA
engines) whose register windows and descriptor memories metadata need to
be exposed to a remote peer. Endpoint function drivers need a generic
way to discover such resources without hard-coding controller-specific
helpers.
Add pci_epc_get_aux_resources() and the corresponding pci_epc_ops
get_aux_resources() callback. The API returns a list of resources
described by type, physical address and size, plus type-specific
metadata.
Passing resources == NULL (or num_resources == 0) returns the required
number of entries.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
drivers/pci/endpoint/pci-epc-core.c | 41 +++++++++++++++++++++++
include/linux/pci-epc.h | 52 +++++++++++++++++++++++++++++
2 files changed, 93 insertions(+)
diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
index 32cf9a9bc365..d63967622505 100644
--- a/drivers/pci/endpoint/pci-epc-core.c
+++ b/drivers/pci/endpoint/pci-epc-core.c
@@ -157,6 +157,47 @@ const struct pci_epc_features *pci_epc_get_features(struct pci_epc *epc,
}
EXPORT_SYMBOL_GPL(pci_epc_get_features);
+/**
+ * pci_epc_get_aux_resources() - query EPC-provided auxiliary resources
+ * @epc: EPC device
+ * @func_no: function number
+ * @vfunc_no: virtual function number
+ * @resources: output array (may be NULL to query required count)
+ * @num_resources: size of @resources array in entries (0 when querying count)
+ *
+ * Some EPC backends integrate auxiliary blocks (e.g. DMA engines) whose control
+ * registers and/or descriptor memories can be exposed to the host by mapping
+ * them into BAR space. This helper queries the backend for such resources.
+ *
+ * Return:
+ * * >= 0: number of resources returned (or required, if @resources is NULL)
+ * * -EOPNOTSUPP: backend does not support auxiliary resource queries
+ * * other -errno on failure
+ */
+int pci_epc_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
+ struct pci_epc_aux_resource *resources,
+ int num_resources)
+{
+ int ret;
+
+ if (!epc || !epc->ops)
+ return -EINVAL;
+
+ if (func_no >= epc->max_functions)
+ return -EINVAL;
+
+ if (!epc->ops->get_aux_resources)
+ return -EOPNOTSUPP;
+
+ mutex_lock(&epc->lock);
+ ret = epc->ops->get_aux_resources(epc, func_no, vfunc_no, resources,
+ num_resources);
+ mutex_unlock(&epc->lock);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(pci_epc_get_aux_resources);
+
/**
* pci_epc_stop() - stop the PCI link
* @epc: the link of the EPC device that has to be stopped
diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
index 63a24ebf144c..0827650acf93 100644
--- a/include/linux/pci-epc.h
+++ b/include/linux/pci-epc.h
@@ -61,6 +61,51 @@ struct pci_epc_map {
void __iomem *virt_addr;
};
+/**
+ * enum pci_epc_aux_resource_type - auxiliary resource type identifiers
+ * @PCI_EPC_AUX_DMA_CTRL_MMIO: Integrated DMA controller register window (MMIO)
+ * @PCI_EPC_AUX_DMA_CHAN_DESC: Per-channel DMA descriptor
+ * @PCI_EPC_AUX_DOORBELL_MMIO: Doorbell MMIO, that might be outside the DMA
+ * controller register window
+ *
+ * EPC backends may expose auxiliary blocks (e.g. DMA engines) by mapping their
+ * register windows and descriptor memories into BAR space. This enum
+ * identifies the type of each exposable resource.
+ */
+enum pci_epc_aux_resource_type {
+ PCI_EPC_AUX_DMA_CTRL_MMIO,
+ PCI_EPC_AUX_DMA_CHAN_DESC,
+ PCI_EPC_AUX_DOORBELL_MMIO,
+};
+
+/**
+ * struct pci_epc_aux_resource - a physical auxiliary resource that may be
+ * exposed for peer use
+ * @type: resource type, see enum pci_epc_aux_resource_type
+ * @phys_addr: physical base address of the resource
+ * @size: size of the resource in bytes
+ * @bar: BAR number where this resource is already exposed to the RC
+ * (NO_BAR if not)
+ * @bar_offset: offset within @bar where the resource starts (valid iff
+ * @bar != NO_BAR)
+ * @u: type-specific metadata
+ */
+struct pci_epc_aux_resource {
+ enum pci_epc_aux_resource_type type;
+ phys_addr_t phys_addr;
+ resource_size_t size;
+ enum pci_barno bar;
+ resource_size_t bar_offset;
+
+ union {
+ /* PCI_EPC_AUX_DOORBELL_MMIO */
+ struct {
+ int irq; /* IRQ number for the doorbell handler */
+ u32 data; /* write value to ring the doorbell */
+ } db_mmio;
+ } u;
+};
+
/**
* struct pci_epc_ops - set of function pointers for performing EPC operations
* @write_header: ops to populate configuration space header
@@ -84,6 +129,7 @@ struct pci_epc_map {
* @start: ops to start the PCI link
* @stop: ops to stop the PCI link
* @get_features: ops to get the features supported by the EPC
+ * @get_aux_resources: ops to retrieve controller-owned auxiliary resources
* @owner: the module owner containing the ops
*/
struct pci_epc_ops {
@@ -115,6 +161,9 @@ struct pci_epc_ops {
void (*stop)(struct pci_epc *epc);
const struct pci_epc_features* (*get_features)(struct pci_epc *epc,
u8 func_no, u8 vfunc_no);
+ int (*get_aux_resources)(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
+ struct pci_epc_aux_resource *resources,
+ int num_resources);
struct module *owner;
};
@@ -348,6 +397,9 @@ int pci_epc_start(struct pci_epc *epc);
void pci_epc_stop(struct pci_epc *epc);
const struct pci_epc_features *pci_epc_get_features(struct pci_epc *epc,
u8 func_no, u8 vfunc_no);
+int pci_epc_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
+ struct pci_epc_aux_resource *resources,
+ int num_resources);
enum pci_barno
pci_epc_get_first_free_bar(const struct pci_epc_features *epc_features);
enum pci_barno pci_epc_get_next_free_bar(const struct pci_epc_features
--
2.51.0
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v10 1/7] PCI: endpoint: Add auxiliary resource query API
2026-03-02 7:14 ` [PATCH v10 1/7] PCI: endpoint: Add auxiliary resource query API Koichiro Den
@ 2026-03-21 14:17 ` Manivannan Sadhasivam
2026-03-23 1:34 ` Koichiro Den
0 siblings, 1 reply; 28+ messages in thread
From: Manivannan Sadhasivam @ 2026-03-21 14:17 UTC (permalink / raw)
To: Koichiro Den
Cc: Jingoo Han, Lorenzo Pieralisi, Krzysztof Wilczyński,
Rob Herring, Bjorn Helgaas, Kishon Vijay Abraham I, Jon Mason,
Dave Jiang, Allen Hubbe, Niklas Cassel, Frank Li,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 02, 2026 at 04:14:21PM +0900, Koichiro Den wrote:
> Endpoint controller drivers may integrate auxiliary blocks (e.g. DMA
> engines) whose register windows and descriptor memories metadata need to
> be exposed to a remote peer. Endpoint function drivers need a generic
> way to discover such resources without hard-coding controller-specific
> helpers.
>
> Add pci_epc_get_aux_resources() and the corresponding pci_epc_ops
> get_aux_resources() callback. The API returns a list of resources
> described by type, physical address and size, plus type-specific
> metadata.
>
> Passing resources == NULL (or num_resources == 0) returns the required
> number of entries.
>
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Tested-by: Niklas Cassel <cassel@kernel.org>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
> drivers/pci/endpoint/pci-epc-core.c | 41 +++++++++++++++++++++++
> include/linux/pci-epc.h | 52 +++++++++++++++++++++++++++++
> 2 files changed, 93 insertions(+)
>
> diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> index 32cf9a9bc365..d63967622505 100644
> --- a/drivers/pci/endpoint/pci-epc-core.c
> +++ b/drivers/pci/endpoint/pci-epc-core.c
> @@ -157,6 +157,47 @@ const struct pci_epc_features *pci_epc_get_features(struct pci_epc *epc,
> }
> EXPORT_SYMBOL_GPL(pci_epc_get_features);
>
> +/**
> + * pci_epc_get_aux_resources() - query EPC-provided auxiliary resources
> + * @epc: EPC device
> + * @func_no: function number
> + * @vfunc_no: virtual function number
> + * @resources: output array (may be NULL to query required count)
> + * @num_resources: size of @resources array in entries (0 when querying count)
> + *
> + * Some EPC backends integrate auxiliary blocks (e.g. DMA engines) whose control
> + * registers and/or descriptor memories can be exposed to the host by mapping
> + * them into BAR space. This helper queries the backend for such resources.
> + *
> + * Return:
> + * * >= 0: number of resources returned (or required, if @resources is NULL)
> + * * -EOPNOTSUPP: backend does not support auxiliary resource queries
> + * * other -errno on failure
> + */
> +int pci_epc_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
> + struct pci_epc_aux_resource *resources,
> + int num_resources)
> +{
> + int ret;
> +
> + if (!epc || !epc->ops)
> + return -EINVAL;
> +
> + if (func_no >= epc->max_functions)
> + return -EINVAL;
> +
> + if (!epc->ops->get_aux_resources)
> + return -EOPNOTSUPP;
> +
> + mutex_lock(&epc->lock);
> + ret = epc->ops->get_aux_resources(epc, func_no, vfunc_no, resources,
> + num_resources);
> + mutex_unlock(&epc->lock);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(pci_epc_get_aux_resources);
> +
> /**
> * pci_epc_stop() - stop the PCI link
> * @epc: the link of the EPC device that has to be stopped
> diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
> index 63a24ebf144c..0827650acf93 100644
> --- a/include/linux/pci-epc.h
> +++ b/include/linux/pci-epc.h
> @@ -61,6 +61,51 @@ struct pci_epc_map {
> void __iomem *virt_addr;
> };
>
> +/**
> + * enum pci_epc_aux_resource_type - auxiliary resource type identifiers
> + * @PCI_EPC_AUX_DMA_CTRL_MMIO: Integrated DMA controller register window (MMIO)
> + * @PCI_EPC_AUX_DMA_CHAN_DESC: Per-channel DMA descriptor
> + * @PCI_EPC_AUX_DOORBELL_MMIO: Doorbell MMIO, that might be outside the DMA
> + * controller register window
> + *
> + * EPC backends may expose auxiliary blocks (e.g. DMA engines) by mapping their
> + * register windows and descriptor memories into BAR space. This enum
> + * identifies the type of each exposable resource.
> + */
> +enum pci_epc_aux_resource_type {
> + PCI_EPC_AUX_DMA_CTRL_MMIO,
> + PCI_EPC_AUX_DMA_CHAN_DESC,
I see that these resources are only used in [1], which is still in early stage.
It'd help if you can just keep DOORBELL_MMIO resource related code in this
series and add these resources when required.
- Mani
[1] https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp
--
மணிவண்ணன் சதாசிவம்
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 1/7] PCI: endpoint: Add auxiliary resource query API
2026-03-21 14:17 ` Manivannan Sadhasivam
@ 2026-03-23 1:34 ` Koichiro Den
0 siblings, 0 replies; 28+ messages in thread
From: Koichiro Den @ 2026-03-23 1:34 UTC (permalink / raw)
To: Manivannan Sadhasivam
Cc: Jingoo Han, Lorenzo Pieralisi, Krzysztof Wilczyński,
Rob Herring, Bjorn Helgaas, Kishon Vijay Abraham I, Jon Mason,
Dave Jiang, Allen Hubbe, Niklas Cassel, Frank Li,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Sat, Mar 21, 2026 at 07:47:32PM +0530, Manivannan Sadhasivam wrote:
> On Mon, Mar 02, 2026 at 04:14:21PM +0900, Koichiro Den wrote:
> > Endpoint controller drivers may integrate auxiliary blocks (e.g. DMA
> > engines) whose register windows and descriptor memories metadata need to
> > be exposed to a remote peer. Endpoint function drivers need a generic
> > way to discover such resources without hard-coding controller-specific
> > helpers.
> >
> > Add pci_epc_get_aux_resources() and the corresponding pci_epc_ops
> > get_aux_resources() callback. The API returns a list of resources
> > described by type, physical address and size, plus type-specific
> > metadata.
> >
> > Passing resources == NULL (or num_resources == 0) returns the required
> > number of entries.
> >
> > Reviewed-by: Frank Li <Frank.Li@nxp.com>
> > Tested-by: Niklas Cassel <cassel@kernel.org>
> > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > ---
> > drivers/pci/endpoint/pci-epc-core.c | 41 +++++++++++++++++++++++
> > include/linux/pci-epc.h | 52 +++++++++++++++++++++++++++++
> > 2 files changed, 93 insertions(+)
> >
> > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > index 32cf9a9bc365..d63967622505 100644
> > --- a/drivers/pci/endpoint/pci-epc-core.c
> > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > @@ -157,6 +157,47 @@ const struct pci_epc_features *pci_epc_get_features(struct pci_epc *epc,
> > }
> > EXPORT_SYMBOL_GPL(pci_epc_get_features);
> >
> > +/**
> > + * pci_epc_get_aux_resources() - query EPC-provided auxiliary resources
> > + * @epc: EPC device
> > + * @func_no: function number
> > + * @vfunc_no: virtual function number
> > + * @resources: output array (may be NULL to query required count)
> > + * @num_resources: size of @resources array in entries (0 when querying count)
> > + *
> > + * Some EPC backends integrate auxiliary blocks (e.g. DMA engines) whose control
> > + * registers and/or descriptor memories can be exposed to the host by mapping
> > + * them into BAR space. This helper queries the backend for such resources.
> > + *
> > + * Return:
> > + * * >= 0: number of resources returned (or required, if @resources is NULL)
> > + * * -EOPNOTSUPP: backend does not support auxiliary resource queries
> > + * * other -errno on failure
> > + */
> > +int pci_epc_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
> > + struct pci_epc_aux_resource *resources,
> > + int num_resources)
> > +{
> > + int ret;
> > +
> > + if (!epc || !epc->ops)
> > + return -EINVAL;
> > +
> > + if (func_no >= epc->max_functions)
> > + return -EINVAL;
> > +
> > + if (!epc->ops->get_aux_resources)
> > + return -EOPNOTSUPP;
> > +
> > + mutex_lock(&epc->lock);
> > + ret = epc->ops->get_aux_resources(epc, func_no, vfunc_no, resources,
> > + num_resources);
> > + mutex_unlock(&epc->lock);
> > +
> > + return ret;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_get_aux_resources);
> > +
> > /**
> > * pci_epc_stop() - stop the PCI link
> > * @epc: the link of the EPC device that has to be stopped
> > diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
> > index 63a24ebf144c..0827650acf93 100644
> > --- a/include/linux/pci-epc.h
> > +++ b/include/linux/pci-epc.h
> > @@ -61,6 +61,51 @@ struct pci_epc_map {
> > void __iomem *virt_addr;
> > };
> >
> > +/**
> > + * enum pci_epc_aux_resource_type - auxiliary resource type identifiers
> > + * @PCI_EPC_AUX_DMA_CTRL_MMIO: Integrated DMA controller register window (MMIO)
> > + * @PCI_EPC_AUX_DMA_CHAN_DESC: Per-channel DMA descriptor
> > + * @PCI_EPC_AUX_DOORBELL_MMIO: Doorbell MMIO, that might be outside the DMA
> > + * controller register window
> > + *
> > + * EPC backends may expose auxiliary blocks (e.g. DMA engines) by mapping their
> > + * register windows and descriptor memories into BAR space. This enum
> > + * identifies the type of each exposable resource.
> > + */
> > +enum pci_epc_aux_resource_type {
> > + PCI_EPC_AUX_DMA_CTRL_MMIO,
> > + PCI_EPC_AUX_DMA_CHAN_DESC,
>
> I see that these resources are only used in [1], which is still in early stage.
> It'd help if you can just keep DOORBELL_MMIO resource related code in this
> series and add these resources when required.
Thanks for the suggestion.
Agree. I'll drop PCI_EPC_AUX_DMA_CTRL_MMIO and PCI_EPC_AUX_DMA_CHAN_DESC from
this series, and add them later when the consumer is in place.
Best regards,
Koichiro
>
> - Mani
>
> [1] https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp
>
> --
> மணிவண்ணன் சதாசிவம்
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v10 2/7] PCI: dwc: Record integrated eDMA register window
2026-03-02 7:14 [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
2026-03-02 7:14 ` [PATCH v10 1/7] PCI: endpoint: Add auxiliary resource query API Koichiro Den
@ 2026-03-02 7:14 ` Koichiro Den
2026-03-21 14:21 ` Manivannan Sadhasivam
2026-03-02 7:14 ` [PATCH v10 3/7] PCI: dwc: ep: Expose integrated eDMA resources via EPC aux-resource API Koichiro Den
` (4 subsequent siblings)
6 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-02 7:14 UTC (permalink / raw)
To: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Frank Li, Bhanu Seshu Kumar Valluri,
Marco Crivellari, Shin'ichiro Kawasaki, Manikanta Maddireddy
Cc: linux-pci, linux-kernel, ntb
Some DesignWare PCIe controllers integrate an eDMA block whose registers
are located in a dedicated register window. Endpoint function drivers
may need the physical base and size of this window to map/expose it to a
peer.
Record the physical base and size of the integrated eDMA register window
in struct dw_pcie.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
drivers/pci/controller/dwc/pcie-designware.c | 4 ++++
drivers/pci/controller/dwc/pcie-designware.h | 2 ++
2 files changed, 6 insertions(+)
diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 5741c09dde7f..f82ed189f6ae 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -162,8 +162,12 @@ int dw_pcie_get_resources(struct dw_pcie *pci)
pci->edma.reg_base = devm_ioremap_resource(pci->dev, res);
if (IS_ERR(pci->edma.reg_base))
return PTR_ERR(pci->edma.reg_base);
+ pci->edma_reg_phys = res->start;
+ pci->edma_reg_size = resource_size(res);
} else if (pci->atu_size >= 2 * DEFAULT_DBI_DMA_OFFSET) {
pci->edma.reg_base = pci->atu_base + DEFAULT_DBI_DMA_OFFSET;
+ pci->edma_reg_phys = pci->atu_phys_addr + DEFAULT_DBI_DMA_OFFSET;
+ pci->edma_reg_size = pci->atu_size - DEFAULT_DBI_DMA_OFFSET;
}
}
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index ae6389dd9caa..52f26663e8b1 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -541,6 +541,8 @@ struct dw_pcie {
int max_link_speed;
u8 n_fts[2];
struct dw_edma_chip edma;
+ phys_addr_t edma_reg_phys;
+ resource_size_t edma_reg_size;
bool l1ss_support; /* L1 PM Substates support */
struct clk_bulk_data app_clks[DW_PCIE_NUM_APP_CLKS];
struct clk_bulk_data core_clks[DW_PCIE_NUM_CORE_CLKS];
--
2.51.0
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v10 2/7] PCI: dwc: Record integrated eDMA register window
2026-03-02 7:14 ` [PATCH v10 2/7] PCI: dwc: Record integrated eDMA register window Koichiro Den
@ 2026-03-21 14:21 ` Manivannan Sadhasivam
2026-03-23 1:46 ` Koichiro Den
0 siblings, 1 reply; 28+ messages in thread
From: Manivannan Sadhasivam @ 2026-03-21 14:21 UTC (permalink / raw)
To: Koichiro Den
Cc: Jingoo Han, Lorenzo Pieralisi, Krzysztof Wilczyński,
Rob Herring, Bjorn Helgaas, Kishon Vijay Abraham I, Jon Mason,
Dave Jiang, Allen Hubbe, Niklas Cassel, Frank Li,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 02, 2026 at 04:14:22PM +0900, Koichiro Den wrote:
> Some DesignWare PCIe controllers integrate an eDMA block whose registers
> are located in a dedicated register window. Endpoint function drivers
> may need the physical base and size of this window to map/expose it to a
> peer.
>
This sounds exactly like 'Remote eDMA' concept where the eDMA registers are
exposed over BAR and programmed by the host. So why this duplication and why
can't you reuse remote eDMA?
- Mani
> Record the physical base and size of the integrated eDMA register window
> in struct dw_pcie.
>
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
> Tested-by: Niklas Cassel <cassel@kernel.org>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
> drivers/pci/controller/dwc/pcie-designware.c | 4 ++++
> drivers/pci/controller/dwc/pcie-designware.h | 2 ++
> 2 files changed, 6 insertions(+)
>
> diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> index 5741c09dde7f..f82ed189f6ae 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.c
> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> @@ -162,8 +162,12 @@ int dw_pcie_get_resources(struct dw_pcie *pci)
> pci->edma.reg_base = devm_ioremap_resource(pci->dev, res);
> if (IS_ERR(pci->edma.reg_base))
> return PTR_ERR(pci->edma.reg_base);
> + pci->edma_reg_phys = res->start;
> + pci->edma_reg_size = resource_size(res);
> } else if (pci->atu_size >= 2 * DEFAULT_DBI_DMA_OFFSET) {
> pci->edma.reg_base = pci->atu_base + DEFAULT_DBI_DMA_OFFSET;
> + pci->edma_reg_phys = pci->atu_phys_addr + DEFAULT_DBI_DMA_OFFSET;
> + pci->edma_reg_size = pci->atu_size - DEFAULT_DBI_DMA_OFFSET;
> }
> }
>
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index ae6389dd9caa..52f26663e8b1 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -541,6 +541,8 @@ struct dw_pcie {
> int max_link_speed;
> u8 n_fts[2];
> struct dw_edma_chip edma;
> + phys_addr_t edma_reg_phys;
> + resource_size_t edma_reg_size;
> bool l1ss_support; /* L1 PM Substates support */
> struct clk_bulk_data app_clks[DW_PCIE_NUM_APP_CLKS];
> struct clk_bulk_data core_clks[DW_PCIE_NUM_CORE_CLKS];
> --
> 2.51.0
>
--
மணிவண்ணன் சதாசிவம்
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 2/7] PCI: dwc: Record integrated eDMA register window
2026-03-21 14:21 ` Manivannan Sadhasivam
@ 2026-03-23 1:46 ` Koichiro Den
2026-03-24 8:06 ` Koichiro Den
0 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-23 1:46 UTC (permalink / raw)
To: Manivannan Sadhasivam
Cc: Jingoo Han, Lorenzo Pieralisi, Krzysztof Wilczyński,
Rob Herring, Bjorn Helgaas, Kishon Vijay Abraham I, Jon Mason,
Dave Jiang, Allen Hubbe, Niklas Cassel, Frank Li,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Sat, Mar 21, 2026 at 07:51:08PM +0530, Manivannan Sadhasivam wrote:
> On Mon, Mar 02, 2026 at 04:14:22PM +0900, Koichiro Den wrote:
> > Some DesignWare PCIe controllers integrate an eDMA block whose registers
> > are located in a dedicated register window. Endpoint function drivers
> > may need the physical base and size of this window to map/expose it to a
> > peer.
> >
>
> This sounds exactly like 'Remote eDMA' concept where the eDMA registers are
> exposed over BAR and programmed by the host. So why this duplication and why
> can't you reuse remote eDMA?
Thanks for reviewing.
The intent is not to introduce another "remote eDMA" scheme. I am trying to
reuse the same model, but what is missing today is the EP-side resource
plumbing.
In-tree today, the remote path is effectively host-driven, i.e. a driver such as
dw-edma-pcie discovers and maps the remote eDMA windows after enumeration. For a
DesignWare EP, we still need a way for the EPC/EPF side to describe what
integrated eDMA resources are available to be exposed to the RC.
This patch is only the preparatory part. Today, pci->edma.reg_base is just a
local kernel VA, but the EP side needs the physical base/size so it can expose
that window in a BAR, or reuse a reserved BAR window as-is (e.g. RK3588).
Patch 3 then exports the eDMA register / LL / doorbell resources via
get_aux_resources().
So this is meant to make existing remote-eDMA style exposure usable from the PCI
endpoint framework, not to duplicate it. I can revise the commit message to
better explain this.
Best regards,
Koichiro
>
> - Mani
>
> > Record the physical base and size of the integrated eDMA register window
> > in struct dw_pcie.
> >
> > Reviewed-by: Frank Li <Frank.Li@nxp.com>
> > Tested-by: Niklas Cassel <cassel@kernel.org>
> > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > ---
> > drivers/pci/controller/dwc/pcie-designware.c | 4 ++++
> > drivers/pci/controller/dwc/pcie-designware.h | 2 ++
> > 2 files changed, 6 insertions(+)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> > index 5741c09dde7f..f82ed189f6ae 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > @@ -162,8 +162,12 @@ int dw_pcie_get_resources(struct dw_pcie *pci)
> > pci->edma.reg_base = devm_ioremap_resource(pci->dev, res);
> > if (IS_ERR(pci->edma.reg_base))
> > return PTR_ERR(pci->edma.reg_base);
> > + pci->edma_reg_phys = res->start;
> > + pci->edma_reg_size = resource_size(res);
> > } else if (pci->atu_size >= 2 * DEFAULT_DBI_DMA_OFFSET) {
> > pci->edma.reg_base = pci->atu_base + DEFAULT_DBI_DMA_OFFSET;
> > + pci->edma_reg_phys = pci->atu_phys_addr + DEFAULT_DBI_DMA_OFFSET;
> > + pci->edma_reg_size = pci->atu_size - DEFAULT_DBI_DMA_OFFSET;
> > }
> > }
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> > index ae6389dd9caa..52f26663e8b1 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -541,6 +541,8 @@ struct dw_pcie {
> > int max_link_speed;
> > u8 n_fts[2];
> > struct dw_edma_chip edma;
> > + phys_addr_t edma_reg_phys;
> > + resource_size_t edma_reg_size;
> > bool l1ss_support; /* L1 PM Substates support */
> > struct clk_bulk_data app_clks[DW_PCIE_NUM_APP_CLKS];
> > struct clk_bulk_data core_clks[DW_PCIE_NUM_CORE_CLKS];
> > --
> > 2.51.0
> >
>
> --
> மணிவண்ணன் சதாசிவம்
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 2/7] PCI: dwc: Record integrated eDMA register window
2026-03-23 1:46 ` Koichiro Den
@ 2026-03-24 8:06 ` Koichiro Den
0 siblings, 0 replies; 28+ messages in thread
From: Koichiro Den @ 2026-03-24 8:06 UTC (permalink / raw)
To: Manivannan Sadhasivam
Cc: Jingoo Han, Lorenzo Pieralisi, Krzysztof Wilczyński,
Rob Herring, Bjorn Helgaas, Kishon Vijay Abraham I, Jon Mason,
Dave Jiang, Allen Hubbe, Niklas Cassel, Frank Li,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 23, 2026 at 10:46:07AM +0900, Koichiro Den wrote:
> On Sat, Mar 21, 2026 at 07:51:08PM +0530, Manivannan Sadhasivam wrote:
> > On Mon, Mar 02, 2026 at 04:14:22PM +0900, Koichiro Den wrote:
> > > Some DesignWare PCIe controllers integrate an eDMA block whose registers
> > > are located in a dedicated register window. Endpoint function drivers
> > > may need the physical base and size of this window to map/expose it to a
> > > peer.
> > >
> >
> > This sounds exactly like 'Remote eDMA' concept where the eDMA registers are
> > exposed over BAR and programmed by the host. So why this duplication and why
> > can't you reuse remote eDMA?
>
> Thanks for reviewing.
>
> The intent is not to introduce another "remote eDMA" scheme. I am trying to
> reuse the same model, but what is missing today is the EP-side resource
> plumbing.
>
> In-tree today, the remote path is effectively host-driven, i.e. a driver such as
> dw-edma-pcie discovers and maps the remote eDMA windows after enumeration. For a
> DesignWare EP, we still need a way for the EPC/EPF side to describe what
> integrated eDMA resources are available to be exposed to the RC.
>
> This patch is only the preparatory part. Today, pci->edma.reg_base is just a
> local kernel VA, but the EP side needs the physical base/size so it can expose
> that window in a BAR, or reuse a reserved BAR window as-is (e.g. RK3588).
> Patch 3 then exports the eDMA register / LL / doorbell resources via
> get_aux_resources().
I think I missed your point in my previous reply.
Based on another feedback on [PATCH v10 1/7], the v11 scope is to be strictly
narrowed so that this series only keeps the pieces needed for the embedded
doorbell fallback, with PCI_EPC_AUX_DMA_CTRL_MMIO and PCI_EPC_AUX_DMA_CHAN_DESC
dropped.
With that in mind, the current commit message does indeed sound a bit confusing.
In this series we only expose a standalone DOORBELL_MMIO resource, whose
register location is derived as:
edma_reg_phys + db_offset
where db_offset comes from the existing dw-edma metadata. This patch records
edma_reg_phys (and the window size), which provides the base for that
calculation. The recorded window size is then used to validate that the derived
doorbell register stays within range.
I'll rewrite the Patch 2 commit message accordingly in v11 to better clarify its
purpose.
Best regards,
Koichiro
>
> So this is meant to make existing remote-eDMA style exposure usable from the PCI
> endpoint framework, not to duplicate it. I can revise the commit message to
> better explain this.
>
> Best regards,
> Koichiro
>
> >
> > - Mani
> >
> > > Record the physical base and size of the integrated eDMA register window
> > > in struct dw_pcie.
> > >
> > > Reviewed-by: Frank Li <Frank.Li@nxp.com>
> > > Tested-by: Niklas Cassel <cassel@kernel.org>
> > > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > > ---
> > > drivers/pci/controller/dwc/pcie-designware.c | 4 ++++
> > > drivers/pci/controller/dwc/pcie-designware.h | 2 ++
> > > 2 files changed, 6 insertions(+)
> > >
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> > > index 5741c09dde7f..f82ed189f6ae 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > > @@ -162,8 +162,12 @@ int dw_pcie_get_resources(struct dw_pcie *pci)
> > > pci->edma.reg_base = devm_ioremap_resource(pci->dev, res);
> > > if (IS_ERR(pci->edma.reg_base))
> > > return PTR_ERR(pci->edma.reg_base);
> > > + pci->edma_reg_phys = res->start;
> > > + pci->edma_reg_size = resource_size(res);
> > > } else if (pci->atu_size >= 2 * DEFAULT_DBI_DMA_OFFSET) {
> > > pci->edma.reg_base = pci->atu_base + DEFAULT_DBI_DMA_OFFSET;
> > > + pci->edma_reg_phys = pci->atu_phys_addr + DEFAULT_DBI_DMA_OFFSET;
> > > + pci->edma_reg_size = pci->atu_size - DEFAULT_DBI_DMA_OFFSET;
> > > }
> > > }
> > >
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> > > index ae6389dd9caa..52f26663e8b1 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > > @@ -541,6 +541,8 @@ struct dw_pcie {
> > > int max_link_speed;
> > > u8 n_fts[2];
> > > struct dw_edma_chip edma;
> > > + phys_addr_t edma_reg_phys;
> > > + resource_size_t edma_reg_size;
> > > bool l1ss_support; /* L1 PM Substates support */
> > > struct clk_bulk_data app_clks[DW_PCIE_NUM_APP_CLKS];
> > > struct clk_bulk_data core_clks[DW_PCIE_NUM_CORE_CLKS];
> > > --
> > > 2.51.0
> > >
> >
> > --
> > மணிவண்ணன் சதாசிவம்
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v10 3/7] PCI: dwc: ep: Expose integrated eDMA resources via EPC aux-resource API
2026-03-02 7:14 [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
2026-03-02 7:14 ` [PATCH v10 1/7] PCI: endpoint: Add auxiliary resource query API Koichiro Den
2026-03-02 7:14 ` [PATCH v10 2/7] PCI: dwc: Record integrated eDMA register window Koichiro Den
@ 2026-03-02 7:14 ` Koichiro Den
2026-03-23 18:36 ` Frank Li
2026-03-02 7:14 ` [PATCH v10 4/7] PCI: endpoint: pci-ep-msi: Refactor doorbell allocation for new backends Koichiro Den
` (3 subsequent siblings)
6 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-02 7:14 UTC (permalink / raw)
To: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Frank Li, Bhanu Seshu Kumar Valluri,
Marco Crivellari, Shin'ichiro Kawasaki, Manikanta Maddireddy
Cc: linux-pci, linux-kernel, ntb
Implement the EPC aux-resource API for DesignWare endpoint controllers
with integrated eDMA.
Report:
- DMA controller MMIO window (PCI_EPC_AUX_DMA_CTRL_MMIO)
- interrupt-emulation doorbell register (PCI_EPC_AUX_DOORBELL_MMIO),
including its Linux IRQ and the data value to write to trigger the
interrupt
- per-channel LL descriptor regions (PCI_EPC_AUX_DMA_CHAN_DESC)
If the DMA controller MMIO window is already exposed via a
platform-owned fixed BAR subregion, also provide the BAR number and
offset so EPF drivers can reuse it without reprogramming the BAR.
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
.../pci/controller/dwc/pcie-designware-ep.c | 151 ++++++++++++++++++
1 file changed, 151 insertions(+)
diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 386bfb7b2bf6..eec20800a745 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -9,6 +9,7 @@
#include <linux/align.h>
#include <linux/bitfield.h>
#include <linux/of.h>
+#include <linux/overflow.h>
#include <linux/platform_device.h>
#include "pcie-designware.h"
@@ -817,6 +818,155 @@ dw_pcie_ep_get_features(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
return ep->ops->get_features(ep);
}
+static const struct pci_epc_bar_rsvd_region *
+dw_pcie_ep_find_bar_rsvd_region(struct dw_pcie_ep *ep,
+ enum pci_epc_bar_rsvd_region_type type,
+ enum pci_barno *bar,
+ resource_size_t *bar_offset)
+{
+ const struct pci_epc_features *features;
+ const struct pci_epc_bar_desc *bar_desc;
+ const struct pci_epc_bar_rsvd_region *r;
+ int i, j;
+
+ if (!ep->ops->get_features)
+ return NULL;
+
+ features = ep->ops->get_features(ep);
+ if (!features)
+ return NULL;
+
+ for (i = BAR_0; i <= BAR_5; i++) {
+ bar_desc = &features->bar[i];
+
+ if (!bar_desc->nr_rsvd_regions || !bar_desc->rsvd_regions)
+ continue;
+
+ for (j = 0; j < bar_desc->nr_rsvd_regions; j++) {
+ r = &bar_desc->rsvd_regions[j];
+
+ if (r->type != type)
+ continue;
+
+ if (bar)
+ *bar = i;
+ if (bar_offset)
+ *bar_offset = r->offset;
+ return r;
+ }
+ }
+
+ return NULL;
+}
+
+static int
+dw_pcie_ep_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
+ struct pci_epc_aux_resource *resources,
+ int num_resources)
+{
+ struct dw_pcie_ep *ep = epc_get_drvdata(epc);
+ struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+ const struct pci_epc_bar_rsvd_region *rsvd;
+ struct dw_edma_chip *edma = &pci->edma;
+ enum pci_barno dma_ctrl_bar = NO_BAR;
+ int ll_cnt = 0, needed, idx = 0;
+ resource_size_t db_offset = edma->db_offset;
+ resource_size_t dma_ctrl_bar_offset = 0;
+ resource_size_t dma_reg_size;
+ unsigned int i;
+
+ if (!pci->edma_reg_size)
+ return 0;
+
+ dma_reg_size = pci->edma_reg_size;
+
+ for (i = 0; i < edma->ll_wr_cnt; i++)
+ if (edma->ll_region_wr[i].sz)
+ ll_cnt++;
+
+ for (i = 0; i < edma->ll_rd_cnt; i++)
+ if (edma->ll_region_rd[i].sz)
+ ll_cnt++;
+
+ needed = 1 + ll_cnt + (db_offset != ~0 ? 1 : 0);
+
+ /* Count query mode */
+ if (!resources || !num_resources)
+ return needed;
+
+ if (num_resources < needed)
+ return -ENOSPC;
+
+ rsvd = dw_pcie_ep_find_bar_rsvd_region(ep,
+ PCI_EPC_BAR_RSVD_DMA_CTRL_MMIO,
+ &dma_ctrl_bar,
+ &dma_ctrl_bar_offset);
+ if (rsvd && rsvd->size < dma_reg_size)
+ dma_reg_size = rsvd->size;
+
+ /* DMA register block */
+ resources[idx++] = (struct pci_epc_aux_resource) {
+ .type = PCI_EPC_AUX_DMA_CTRL_MMIO,
+ .phys_addr = pci->edma_reg_phys,
+ .size = dma_reg_size,
+ .bar = dma_ctrl_bar,
+ .bar_offset = dma_ctrl_bar_offset,
+ };
+
+ /*
+ * For interrupt-emulation doorbells, report a standalone resource
+ * instead of bundling it into the DMA controller MMIO resource.
+ */
+ if (db_offset != ~0) {
+ if (range_end_overflows_t(resource_size_t, db_offset,
+ sizeof(u32), dma_reg_size))
+ return -EINVAL;
+
+ resources[idx++] = (struct pci_epc_aux_resource) {
+ .type = PCI_EPC_AUX_DOORBELL_MMIO,
+ .phys_addr = pci->edma_reg_phys + db_offset,
+ .size = sizeof(u32),
+ .bar = dma_ctrl_bar,
+ .bar_offset = dma_ctrl_bar != NO_BAR ?
+ dma_ctrl_bar_offset + db_offset : 0,
+ .u.db_mmio = {
+ .irq = edma->db_irq,
+ .data = 0, /* write 0 to assert */
+ },
+ };
+ }
+
+ /* One LL region per write channel */
+ for (i = 0; i < edma->ll_wr_cnt; i++) {
+ if (!edma->ll_region_wr[i].sz)
+ continue;
+
+ resources[idx++] = (struct pci_epc_aux_resource) {
+ .type = PCI_EPC_AUX_DMA_CHAN_DESC,
+ .phys_addr = edma->ll_region_wr[i].paddr,
+ .size = edma->ll_region_wr[i].sz,
+ .bar = NO_BAR,
+ .bar_offset = 0,
+ };
+ }
+
+ /* One LL region per read channel */
+ for (i = 0; i < edma->ll_rd_cnt; i++) {
+ if (!edma->ll_region_rd[i].sz)
+ continue;
+
+ resources[idx++] = (struct pci_epc_aux_resource) {
+ .type = PCI_EPC_AUX_DMA_CHAN_DESC,
+ .phys_addr = edma->ll_region_rd[i].paddr,
+ .size = edma->ll_region_rd[i].sz,
+ .bar = NO_BAR,
+ .bar_offset = 0,
+ };
+ }
+
+ return idx;
+}
+
static const struct pci_epc_ops epc_ops = {
.write_header = dw_pcie_ep_write_header,
.set_bar = dw_pcie_ep_set_bar,
@@ -832,6 +982,7 @@ static const struct pci_epc_ops epc_ops = {
.start = dw_pcie_ep_start,
.stop = dw_pcie_ep_stop,
.get_features = dw_pcie_ep_get_features,
+ .get_aux_resources = dw_pcie_ep_get_aux_resources,
};
/**
--
2.51.0
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v10 3/7] PCI: dwc: ep: Expose integrated eDMA resources via EPC aux-resource API
2026-03-02 7:14 ` [PATCH v10 3/7] PCI: dwc: ep: Expose integrated eDMA resources via EPC aux-resource API Koichiro Den
@ 2026-03-23 18:36 ` Frank Li
2026-03-24 8:45 ` Koichiro Den
0 siblings, 1 reply; 28+ messages in thread
From: Frank Li @ 2026-03-23 18:36 UTC (permalink / raw)
To: Koichiro Den
Cc: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 02, 2026 at 04:14:23PM +0900, Koichiro Den wrote:
> Implement the EPC aux-resource API for DesignWare endpoint controllers
> with integrated eDMA.
>
> Report:
> - DMA controller MMIO window (PCI_EPC_AUX_DMA_CTRL_MMIO)
> - interrupt-emulation doorbell register (PCI_EPC_AUX_DOORBELL_MMIO),
> including its Linux IRQ and the data value to write to trigger the
> interrupt
> - per-channel LL descriptor regions (PCI_EPC_AUX_DMA_CHAN_DESC)
>
> If the DMA controller MMIO window is already exposed via a
> platform-owned fixed BAR subregion, also provide the BAR number and
> offset so EPF drivers can reuse it without reprogramming the BAR.
>
> Tested-by: Niklas Cassel <cassel@kernel.org>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
Reviewed-by: Frank Li <Frank.Li@nxp.com>
> .../pci/controller/dwc/pcie-designware-ep.c | 151 ++++++++++++++++++
> 1 file changed, 151 insertions(+)
>
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index 386bfb7b2bf6..eec20800a745 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -9,6 +9,7 @@
> #include <linux/align.h>
> #include <linux/bitfield.h>
> #include <linux/of.h>
> +#include <linux/overflow.h>
> #include <linux/platform_device.h>
>
> #include "pcie-designware.h"
> @@ -817,6 +818,155 @@ dw_pcie_ep_get_features(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> return ep->ops->get_features(ep);
> }
>
> +static const struct pci_epc_bar_rsvd_region *
> +dw_pcie_ep_find_bar_rsvd_region(struct dw_pcie_ep *ep,
> + enum pci_epc_bar_rsvd_region_type type,
> + enum pci_barno *bar,
> + resource_size_t *bar_offset)
> +{
> + const struct pci_epc_features *features;
> + const struct pci_epc_bar_desc *bar_desc;
> + const struct pci_epc_bar_rsvd_region *r;
> + int i, j;
> +
> + if (!ep->ops->get_features)
> + return NULL;
> +
> + features = ep->ops->get_features(ep);
> + if (!features)
> + return NULL;
> +
> + for (i = BAR_0; i <= BAR_5; i++) {
> + bar_desc = &features->bar[i];
> +
> + if (!bar_desc->nr_rsvd_regions || !bar_desc->rsvd_regions)
> + continue;
> +
> + for (j = 0; j < bar_desc->nr_rsvd_regions; j++) {
> + r = &bar_desc->rsvd_regions[j];
> +
> + if (r->type != type)
> + continue;
> +
> + if (bar)
> + *bar = i;
> + if (bar_offset)
> + *bar_offset = r->offset;
> + return r;
> + }
> + }
> +
> + return NULL;
> +}
> +
> +static int
> +dw_pcie_ep_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
> + struct pci_epc_aux_resource *resources,
> + int num_resources)
> +{
> + struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> + struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> + const struct pci_epc_bar_rsvd_region *rsvd;
> + struct dw_edma_chip *edma = &pci->edma;
> + enum pci_barno dma_ctrl_bar = NO_BAR;
> + int ll_cnt = 0, needed, idx = 0;
> + resource_size_t db_offset = edma->db_offset;
> + resource_size_t dma_ctrl_bar_offset = 0;
> + resource_size_t dma_reg_size;
> + unsigned int i;
> +
> + if (!pci->edma_reg_size)
> + return 0;
> +
> + dma_reg_size = pci->edma_reg_size;
> +
> + for (i = 0; i < edma->ll_wr_cnt; i++)
> + if (edma->ll_region_wr[i].sz)
> + ll_cnt++;
> +
> + for (i = 0; i < edma->ll_rd_cnt; i++)
> + if (edma->ll_region_rd[i].sz)
> + ll_cnt++;
> +
> + needed = 1 + ll_cnt + (db_offset != ~0 ? 1 : 0);
> +
> + /* Count query mode */
> + if (!resources || !num_resources)
> + return needed;
> +
> + if (num_resources < needed)
> + return -ENOSPC;
> +
> + rsvd = dw_pcie_ep_find_bar_rsvd_region(ep,
> + PCI_EPC_BAR_RSVD_DMA_CTRL_MMIO,
> + &dma_ctrl_bar,
> + &dma_ctrl_bar_offset);
> + if (rsvd && rsvd->size < dma_reg_size)
> + dma_reg_size = rsvd->size;
> +
> + /* DMA register block */
> + resources[idx++] = (struct pci_epc_aux_resource) {
> + .type = PCI_EPC_AUX_DMA_CTRL_MMIO,
> + .phys_addr = pci->edma_reg_phys,
> + .size = dma_reg_size,
> + .bar = dma_ctrl_bar,
> + .bar_offset = dma_ctrl_bar_offset,
> + };
> +
> + /*
> + * For interrupt-emulation doorbells, report a standalone resource
> + * instead of bundling it into the DMA controller MMIO resource.
> + */
> + if (db_offset != ~0) {
> + if (range_end_overflows_t(resource_size_t, db_offset,
> + sizeof(u32), dma_reg_size))
> + return -EINVAL;
> +
> + resources[idx++] = (struct pci_epc_aux_resource) {
> + .type = PCI_EPC_AUX_DOORBELL_MMIO,
> + .phys_addr = pci->edma_reg_phys + db_offset,
> + .size = sizeof(u32),
> + .bar = dma_ctrl_bar,
> + .bar_offset = dma_ctrl_bar != NO_BAR ?
> + dma_ctrl_bar_offset + db_offset : 0,
> + .u.db_mmio = {
> + .irq = edma->db_irq,
> + .data = 0, /* write 0 to assert */
> + },
> + };
> + }
> +
> + /* One LL region per write channel */
> + for (i = 0; i < edma->ll_wr_cnt; i++) {
> + if (!edma->ll_region_wr[i].sz)
> + continue;
> +
> + resources[idx++] = (struct pci_epc_aux_resource) {
> + .type = PCI_EPC_AUX_DMA_CHAN_DESC,
> + .phys_addr = edma->ll_region_wr[i].paddr,
> + .size = edma->ll_region_wr[i].sz,
> + .bar = NO_BAR,
> + .bar_offset = 0,
> + };
> + }
> +
> + /* One LL region per read channel */
> + for (i = 0; i < edma->ll_rd_cnt; i++) {
> + if (!edma->ll_region_rd[i].sz)
> + continue;
> +
> + resources[idx++] = (struct pci_epc_aux_resource) {
> + .type = PCI_EPC_AUX_DMA_CHAN_DESC,
> + .phys_addr = edma->ll_region_rd[i].paddr,
> + .size = edma->ll_region_rd[i].sz,
> + .bar = NO_BAR,
> + .bar_offset = 0,
> + };
> + }
> +
> + return idx;
> +}
> +
> static const struct pci_epc_ops epc_ops = {
> .write_header = dw_pcie_ep_write_header,
> .set_bar = dw_pcie_ep_set_bar,
> @@ -832,6 +982,7 @@ static const struct pci_epc_ops epc_ops = {
> .start = dw_pcie_ep_start,
> .stop = dw_pcie_ep_stop,
> .get_features = dw_pcie_ep_get_features,
> + .get_aux_resources = dw_pcie_ep_get_aux_resources,
> };
>
> /**
> --
> 2.51.0
>
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 3/7] PCI: dwc: ep: Expose integrated eDMA resources via EPC aux-resource API
2026-03-23 18:36 ` Frank Li
@ 2026-03-24 8:45 ` Koichiro Den
0 siblings, 0 replies; 28+ messages in thread
From: Koichiro Den @ 2026-03-24 8:45 UTC (permalink / raw)
To: Frank Li
Cc: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 23, 2026 at 02:36:11PM -0400, Frank Li wrote:
> On Mon, Mar 02, 2026 at 04:14:23PM +0900, Koichiro Den wrote:
> > Implement the EPC aux-resource API for DesignWare endpoint controllers
> > with integrated eDMA.
> >
> > Report:
> > - DMA controller MMIO window (PCI_EPC_AUX_DMA_CTRL_MMIO)
> > - interrupt-emulation doorbell register (PCI_EPC_AUX_DOORBELL_MMIO),
> > including its Linux IRQ and the data value to write to trigger the
> > interrupt
> > - per-channel LL descriptor regions (PCI_EPC_AUX_DMA_CHAN_DESC)
> >
> > If the DMA controller MMIO window is already exposed via a
> > platform-owned fixed BAR subregion, also provide the BAR number and
> > offset so EPF drivers can reuse it without reprogramming the BAR.
> >
> > Tested-by: Niklas Cassel <cassel@kernel.org>
> > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > ---
>
> Reviewed-by: Frank Li <Frank.Li@nxp.com>
Thank you for reviewing, Frank.
After Mani's feedback at:
https://lore.kernel.org/linux-pci/45yj3oztsa4dpy2nyc32td76umhd6dxxkmr5tsqtqvlgy6v4ju@6o4ov7nck7ca/
the code in patch 1 and 3 has changed in v11, so I did not carry over your tag.
When you have a chance, I'd appreciate it if you could take another look at
patches 1 and 3:
https://lore.kernel.org/linux-pci/20260324083728.3744734-2-den@valinux.co.jp/
https://lore.kernel.org/linux-pci/20260324083728.3744734-4-den@valinux.co.jp/
Best regards,
Koichiro
>
> > .../pci/controller/dwc/pcie-designware-ep.c | 151 ++++++++++++++++++
> > 1 file changed, 151 insertions(+)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > index 386bfb7b2bf6..eec20800a745 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > @@ -9,6 +9,7 @@
> > #include <linux/align.h>
> > #include <linux/bitfield.h>
> > #include <linux/of.h>
> > +#include <linux/overflow.h>
> > #include <linux/platform_device.h>
> >
> > #include "pcie-designware.h"
> > @@ -817,6 +818,155 @@ dw_pcie_ep_get_features(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > return ep->ops->get_features(ep);
> > }
> >
> > +static const struct pci_epc_bar_rsvd_region *
> > +dw_pcie_ep_find_bar_rsvd_region(struct dw_pcie_ep *ep,
> > + enum pci_epc_bar_rsvd_region_type type,
> > + enum pci_barno *bar,
> > + resource_size_t *bar_offset)
> > +{
> > + const struct pci_epc_features *features;
> > + const struct pci_epc_bar_desc *bar_desc;
> > + const struct pci_epc_bar_rsvd_region *r;
> > + int i, j;
> > +
> > + if (!ep->ops->get_features)
> > + return NULL;
> > +
> > + features = ep->ops->get_features(ep);
> > + if (!features)
> > + return NULL;
> > +
> > + for (i = BAR_0; i <= BAR_5; i++) {
> > + bar_desc = &features->bar[i];
> > +
> > + if (!bar_desc->nr_rsvd_regions || !bar_desc->rsvd_regions)
> > + continue;
> > +
> > + for (j = 0; j < bar_desc->nr_rsvd_regions; j++) {
> > + r = &bar_desc->rsvd_regions[j];
> > +
> > + if (r->type != type)
> > + continue;
> > +
> > + if (bar)
> > + *bar = i;
> > + if (bar_offset)
> > + *bar_offset = r->offset;
> > + return r;
> > + }
> > + }
> > +
> > + return NULL;
> > +}
> > +
> > +static int
> > +dw_pcie_ep_get_aux_resources(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
> > + struct pci_epc_aux_resource *resources,
> > + int num_resources)
> > +{
> > + struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > + struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > + const struct pci_epc_bar_rsvd_region *rsvd;
> > + struct dw_edma_chip *edma = &pci->edma;
> > + enum pci_barno dma_ctrl_bar = NO_BAR;
> > + int ll_cnt = 0, needed, idx = 0;
> > + resource_size_t db_offset = edma->db_offset;
> > + resource_size_t dma_ctrl_bar_offset = 0;
> > + resource_size_t dma_reg_size;
> > + unsigned int i;
> > +
> > + if (!pci->edma_reg_size)
> > + return 0;
> > +
> > + dma_reg_size = pci->edma_reg_size;
> > +
> > + for (i = 0; i < edma->ll_wr_cnt; i++)
> > + if (edma->ll_region_wr[i].sz)
> > + ll_cnt++;
> > +
> > + for (i = 0; i < edma->ll_rd_cnt; i++)
> > + if (edma->ll_region_rd[i].sz)
> > + ll_cnt++;
> > +
> > + needed = 1 + ll_cnt + (db_offset != ~0 ? 1 : 0);
> > +
> > + /* Count query mode */
> > + if (!resources || !num_resources)
> > + return needed;
> > +
> > + if (num_resources < needed)
> > + return -ENOSPC;
> > +
> > + rsvd = dw_pcie_ep_find_bar_rsvd_region(ep,
> > + PCI_EPC_BAR_RSVD_DMA_CTRL_MMIO,
> > + &dma_ctrl_bar,
> > + &dma_ctrl_bar_offset);
> > + if (rsvd && rsvd->size < dma_reg_size)
> > + dma_reg_size = rsvd->size;
> > +
> > + /* DMA register block */
> > + resources[idx++] = (struct pci_epc_aux_resource) {
> > + .type = PCI_EPC_AUX_DMA_CTRL_MMIO,
> > + .phys_addr = pci->edma_reg_phys,
> > + .size = dma_reg_size,
> > + .bar = dma_ctrl_bar,
> > + .bar_offset = dma_ctrl_bar_offset,
> > + };
> > +
> > + /*
> > + * For interrupt-emulation doorbells, report a standalone resource
> > + * instead of bundling it into the DMA controller MMIO resource.
> > + */
> > + if (db_offset != ~0) {
> > + if (range_end_overflows_t(resource_size_t, db_offset,
> > + sizeof(u32), dma_reg_size))
> > + return -EINVAL;
> > +
> > + resources[idx++] = (struct pci_epc_aux_resource) {
> > + .type = PCI_EPC_AUX_DOORBELL_MMIO,
> > + .phys_addr = pci->edma_reg_phys + db_offset,
> > + .size = sizeof(u32),
> > + .bar = dma_ctrl_bar,
> > + .bar_offset = dma_ctrl_bar != NO_BAR ?
> > + dma_ctrl_bar_offset + db_offset : 0,
> > + .u.db_mmio = {
> > + .irq = edma->db_irq,
> > + .data = 0, /* write 0 to assert */
> > + },
> > + };
> > + }
> > +
> > + /* One LL region per write channel */
> > + for (i = 0; i < edma->ll_wr_cnt; i++) {
> > + if (!edma->ll_region_wr[i].sz)
> > + continue;
> > +
> > + resources[idx++] = (struct pci_epc_aux_resource) {
> > + .type = PCI_EPC_AUX_DMA_CHAN_DESC,
> > + .phys_addr = edma->ll_region_wr[i].paddr,
> > + .size = edma->ll_region_wr[i].sz,
> > + .bar = NO_BAR,
> > + .bar_offset = 0,
> > + };
> > + }
> > +
> > + /* One LL region per read channel */
> > + for (i = 0; i < edma->ll_rd_cnt; i++) {
> > + if (!edma->ll_region_rd[i].sz)
> > + continue;
> > +
> > + resources[idx++] = (struct pci_epc_aux_resource) {
> > + .type = PCI_EPC_AUX_DMA_CHAN_DESC,
> > + .phys_addr = edma->ll_region_rd[i].paddr,
> > + .size = edma->ll_region_rd[i].sz,
> > + .bar = NO_BAR,
> > + .bar_offset = 0,
> > + };
> > + }
> > +
> > + return idx;
> > +}
> > +
> > static const struct pci_epc_ops epc_ops = {
> > .write_header = dw_pcie_ep_write_header,
> > .set_bar = dw_pcie_ep_set_bar,
> > @@ -832,6 +982,7 @@ static const struct pci_epc_ops epc_ops = {
> > .start = dw_pcie_ep_start,
> > .stop = dw_pcie_ep_stop,
> > .get_features = dw_pcie_ep_get_features,
> > + .get_aux_resources = dw_pcie_ep_get_aux_resources,
> > };
> >
> > /**
> > --
> > 2.51.0
> >
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v10 4/7] PCI: endpoint: pci-ep-msi: Refactor doorbell allocation for new backends
2026-03-02 7:14 [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
` (2 preceding siblings ...)
2026-03-02 7:14 ` [PATCH v10 3/7] PCI: dwc: ep: Expose integrated eDMA resources via EPC aux-resource API Koichiro Den
@ 2026-03-02 7:14 ` Koichiro Den
2026-03-02 7:14 ` [PATCH v10 5/7] PCI: endpoint: pci-epf-vntb: Reuse pre-exposed doorbells and IRQ flags Koichiro Den
` (2 subsequent siblings)
6 siblings, 0 replies; 28+ messages in thread
From: Koichiro Den @ 2026-03-02 7:14 UTC (permalink / raw)
To: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Frank Li, Bhanu Seshu Kumar Valluri,
Marco Crivellari, Shin'ichiro Kawasaki, Manikanta Maddireddy
Cc: linux-pci, linux-kernel, ntb
Prepare pci-ep-msi for non-MSI doorbell backends.
Factor MSI doorbell allocation into a helper and extend struct
pci_epf_doorbell_msg with:
- irq_flags: required IRQ request flags (e.g. IRQF_SHARED for some
backends)
- type: doorbell backend type
- bar/offset: pre-exposed doorbell target location, if any
Initialize these fields for the existing MSI-backed doorbell
implementation.
Also add PCI_EPF_DOORBELL_EMBEDDED type, which is to be implemented in a
follow-up patch.
No functional changes.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
drivers/pci/endpoint/pci-ep-msi.c | 54 ++++++++++++++++++++++---------
include/linux/pci-epf.h | 23 +++++++++++--
2 files changed, 60 insertions(+), 17 deletions(-)
diff --git a/drivers/pci/endpoint/pci-ep-msi.c b/drivers/pci/endpoint/pci-ep-msi.c
index 1395919571f8..85fe46103220 100644
--- a/drivers/pci/endpoint/pci-ep-msi.c
+++ b/drivers/pci/endpoint/pci-ep-msi.c
@@ -8,6 +8,7 @@
#include <linux/device.h>
#include <linux/export.h>
+#include <linux/interrupt.h>
#include <linux/irqdomain.h>
#include <linux/module.h>
#include <linux/msi.h>
@@ -35,23 +36,13 @@ static void pci_epf_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
pci_epc_put(epc);
}
-int pci_epf_alloc_doorbell(struct pci_epf *epf, u16 num_db)
+static int pci_epf_alloc_doorbell_msi(struct pci_epf *epf, u16 num_db)
{
- struct pci_epc *epc = epf->epc;
+ struct pci_epf_doorbell_msg *msg;
struct device *dev = &epf->dev;
+ struct pci_epc *epc = epf->epc;
struct irq_domain *domain;
- void *msg;
- int ret;
- int i;
-
- /* TODO: Multi-EPF support */
- if (list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list) != epf) {
- dev_err(dev, "MSI doorbell doesn't support multiple EPF\n");
- return -EINVAL;
- }
-
- if (epf->db_msg)
- return -EBUSY;
+ int ret, i;
domain = of_msi_map_get_device_domain(epc->dev.parent, 0,
DOMAIN_BUS_PLATFORM_MSI);
@@ -74,6 +65,12 @@ int pci_epf_alloc_doorbell(struct pci_epf *epf, u16 num_db)
if (!msg)
return -ENOMEM;
+ for (i = 0; i < num_db; i++)
+ msg[i] = (struct pci_epf_doorbell_msg) {
+ .type = PCI_EPF_DOORBELL_MSI,
+ .bar = NO_BAR,
+ };
+
epf->num_db = num_db;
epf->db_msg = msg;
@@ -90,13 +87,40 @@ int pci_epf_alloc_doorbell(struct pci_epf *epf, u16 num_db)
for (i = 0; i < num_db; i++)
epf->db_msg[i].virq = msi_get_virq(epc->dev.parent, i);
+ return 0;
+}
+
+int pci_epf_alloc_doorbell(struct pci_epf *epf, u16 num_db)
+{
+ struct pci_epc *epc = epf->epc;
+ struct device *dev = &epf->dev;
+ int ret;
+
+ /* TODO: Multi-EPF support */
+ if (list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list) != epf) {
+ dev_err(dev, "Doorbell doesn't support multiple EPF\n");
+ return -EINVAL;
+ }
+
+ if (epf->db_msg)
+ return -EBUSY;
+
+ ret = pci_epf_alloc_doorbell_msi(epf, num_db);
+ if (!ret)
+ return 0;
+
+ dev_err(dev, "Failed to allocate doorbell: %d\n", ret);
return ret;
}
EXPORT_SYMBOL_GPL(pci_epf_alloc_doorbell);
void pci_epf_free_doorbell(struct pci_epf *epf)
{
- platform_device_msi_free_irqs_all(epf->epc->dev.parent);
+ if (!epf->db_msg)
+ return;
+
+ if (epf->db_msg[0].type == PCI_EPF_DOORBELL_MSI)
+ platform_device_msi_free_irqs_all(epf->epc->dev.parent);
kfree(epf->db_msg);
epf->db_msg = NULL;
diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
index 7737a7c03260..cd747447a1ea 100644
--- a/include/linux/pci-epf.h
+++ b/include/linux/pci-epf.h
@@ -152,14 +152,33 @@ struct pci_epf_bar {
struct pci_epf_bar_submap *submap;
};
+enum pci_epf_doorbell_type {
+ PCI_EPF_DOORBELL_MSI = 0,
+ PCI_EPF_DOORBELL_EMBEDDED,
+};
+
/**
* struct pci_epf_doorbell_msg - represents doorbell message
- * @msg: MSI message
- * @virq: IRQ number of this doorbell MSI message
+ * @msg: Doorbell address/data pair to be mapped into BAR space.
+ * For MSI-backed doorbells this is the MSI message, while for
+ * "embedded" doorbells this represents an MMIO write that asserts
+ * an interrupt on the EP side.
+ * @virq: IRQ number of this doorbell message
+ * @irq_flags: Required flags for request_irq()/request_threaded_irq().
+ * Callers may OR-in additional flags (e.g. IRQF_ONESHOT).
+ * @type: Doorbell type.
+ * @bar: BAR number where the doorbell target is already exposed to the RC
+ * (NO_BAR if not)
+ * @offset: offset within @bar for the doorbell target (valid iff
+ * @bar != NO_BAR)
*/
struct pci_epf_doorbell_msg {
struct msi_msg msg;
int virq;
+ unsigned long irq_flags;
+ enum pci_epf_doorbell_type type;
+ enum pci_barno bar;
+ resource_size_t offset;
};
/**
--
2.51.0
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH v10 5/7] PCI: endpoint: pci-epf-vntb: Reuse pre-exposed doorbells and IRQ flags
2026-03-02 7:14 [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
` (3 preceding siblings ...)
2026-03-02 7:14 ` [PATCH v10 4/7] PCI: endpoint: pci-ep-msi: Refactor doorbell allocation for new backends Koichiro Den
@ 2026-03-02 7:14 ` Koichiro Den
2026-03-23 18:39 ` Frank Li
2026-03-02 7:14 ` [PATCH v10 6/7] PCI: endpoint: pci-epf-test: Reuse pre-exposed doorbell targets Koichiro Den
2026-03-02 7:14 ` [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
6 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-02 7:14 UTC (permalink / raw)
To: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Frank Li, Bhanu Seshu Kumar Valluri,
Marco Crivellari, Shin'ichiro Kawasaki, Manikanta Maddireddy
Cc: linux-pci, linux-kernel, ntb
Support doorbell backends where the doorbell target is already exposed
via a platform-owned fixed BAR mapping and/or where the doorbell IRQ
must be requested with specific flags.
When pci_epf_alloc_doorbell() provides db_msg[].bar/offset, reuse the
pre-exposed BAR window and skip programming a new inbound mapping. Also
honor db_msg[].irq_flags when requesting the doorbell IRQ.
Multiple doorbells may share the same Linux IRQ. Avoid duplicate
request_irq() calls by requesting each unique virq once.
Make pci-epf-vntb work with platform-defined or embedded doorbell
backends without exposing backend-specific details to the consumer
layer.
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
drivers/pci/endpoint/functions/pci-epf-vntb.c | 61 ++++++++++++++++++-
1 file changed, 58 insertions(+), 3 deletions(-)
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 148a3b160812..8e9a3048c2e5 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -134,6 +134,11 @@ struct epf_ntb {
u16 vntb_vid;
bool linkup;
+
+ /*
+ * True when doorbells are interrupt-driven (MSI or embedded), false
+ * when polled.
+ */
bool msi_doorbell;
u32 spad_size;
@@ -517,6 +522,17 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)
return 0;
}
+static bool epf_ntb_db_irq_is_duplicated(const struct pci_epf *epf, unsigned int idx)
+{
+ unsigned int i;
+
+ for (i = 0; i < idx; i++)
+ if (epf->db_msg[i].virq == epf->db_msg[idx].virq)
+ return true;
+
+ return false;
+}
+
static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
struct pci_epf_bar *db_bar,
const struct pci_epc_features *epc_features,
@@ -533,9 +549,24 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
if (ret)
return ret;
+ /*
+ * The doorbell target may already be exposed by a platform-owned fixed
+ * BAR. In that case, we must reuse it and the requested db_bar must
+ * match.
+ */
+ if (epf->db_msg[0].bar != NO_BAR && epf->db_msg[0].bar != barno) {
+ ret = -EINVAL;
+ goto err_free_doorbell;
+ }
+
for (req = 0; req < ntb->db_count; req++) {
+ /* Avoid requesting duplicate handlers */
+ if (epf_ntb_db_irq_is_duplicated(epf, req))
+ continue;
+
ret = request_irq(epf->db_msg[req].virq, epf_ntb_doorbell_handler,
- 0, "pci_epf_vntb_db", ntb);
+ epf->db_msg[req].irq_flags, "pci_epf_vntb_db",
+ ntb);
if (ret) {
dev_err(&epf->dev,
@@ -545,6 +576,22 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
}
}
+ if (epf->db_msg[0].bar != NO_BAR) {
+ for (i = 0; i < ntb->db_count; i++) {
+ msg = &epf->db_msg[i].msg;
+
+ if (epf->db_msg[i].bar != barno) {
+ ret = -EINVAL;
+ goto err_free_irq;
+ }
+
+ ntb->reg->db_data[i] = msg->data;
+ ntb->reg->db_offset[i] = epf->db_msg[i].offset;
+ }
+ goto out;
+ }
+
+ /* Program inbound mapping for the doorbell */
msg = &epf->db_msg[0].msg;
high = 0;
@@ -591,6 +638,7 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
ntb->reg->db_offset[i] = offset;
}
+out:
ntb->reg->db_entry_size = 0;
ntb->msi_doorbell = true;
@@ -598,9 +646,13 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
return 0;
err_free_irq:
- for (req--; req >= 0; req--)
+ for (req--; req >= 0; req--) {
+ if (epf_ntb_db_irq_is_duplicated(epf, req))
+ continue;
free_irq(epf->db_msg[req].virq, ntb);
+ }
+err_free_doorbell:
pci_epf_free_doorbell(ntb->epf);
return ret;
}
@@ -666,8 +718,11 @@ static void epf_ntb_db_bar_clear(struct epf_ntb *ntb)
if (ntb->msi_doorbell) {
int i;
- for (i = 0; i < ntb->db_count; i++)
+ for (i = 0; i < ntb->db_count; i++) {
+ if (epf_ntb_db_irq_is_duplicated(ntb->epf, i))
+ continue;
free_irq(ntb->epf->db_msg[i].virq, ntb);
+ }
}
if (ntb->epf->db_msg)
--
2.51.0
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v10 5/7] PCI: endpoint: pci-epf-vntb: Reuse pre-exposed doorbells and IRQ flags
2026-03-02 7:14 ` [PATCH v10 5/7] PCI: endpoint: pci-epf-vntb: Reuse pre-exposed doorbells and IRQ flags Koichiro Den
@ 2026-03-23 18:39 ` Frank Li
0 siblings, 0 replies; 28+ messages in thread
From: Frank Li @ 2026-03-23 18:39 UTC (permalink / raw)
To: Koichiro Den
Cc: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 02, 2026 at 04:14:25PM +0900, Koichiro Den wrote:
> Support doorbell backends where the doorbell target is already exposed
> via a platform-owned fixed BAR mapping and/or where the doorbell IRQ
> must be requested with specific flags.
>
> When pci_epf_alloc_doorbell() provides db_msg[].bar/offset, reuse the
> pre-exposed BAR window and skip programming a new inbound mapping. Also
> honor db_msg[].irq_flags when requesting the doorbell IRQ.
>
> Multiple doorbells may share the same Linux IRQ. Avoid duplicate
> request_irq() calls by requesting each unique virq once.
>
> Make pci-epf-vntb work with platform-defined or embedded doorbell
> backends without exposing backend-specific details to the consumer
> layer.
>
> Tested-by: Niklas Cassel <cassel@kernel.org>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
Reviewed-by: Frank Li <Frank.Li@nxp.com>
> drivers/pci/endpoint/functions/pci-epf-vntb.c | 61 ++++++++++++++++++-
> 1 file changed, 58 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> index 148a3b160812..8e9a3048c2e5 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> @@ -134,6 +134,11 @@ struct epf_ntb {
> u16 vntb_vid;
>
> bool linkup;
> +
> + /*
> + * True when doorbells are interrupt-driven (MSI or embedded), false
> + * when polled.
> + */
> bool msi_doorbell;
> u32 spad_size;
>
> @@ -517,6 +522,17 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)
> return 0;
> }
>
> +static bool epf_ntb_db_irq_is_duplicated(const struct pci_epf *epf, unsigned int idx)
> +{
> + unsigned int i;
> +
> + for (i = 0; i < idx; i++)
> + if (epf->db_msg[i].virq == epf->db_msg[idx].virq)
> + return true;
> +
> + return false;
> +}
> +
> static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
> struct pci_epf_bar *db_bar,
> const struct pci_epc_features *epc_features,
> @@ -533,9 +549,24 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
> if (ret)
> return ret;
>
> + /*
> + * The doorbell target may already be exposed by a platform-owned fixed
> + * BAR. In that case, we must reuse it and the requested db_bar must
> + * match.
> + */
> + if (epf->db_msg[0].bar != NO_BAR && epf->db_msg[0].bar != barno) {
> + ret = -EINVAL;
> + goto err_free_doorbell;
> + }
> +
> for (req = 0; req < ntb->db_count; req++) {
> + /* Avoid requesting duplicate handlers */
> + if (epf_ntb_db_irq_is_duplicated(epf, req))
> + continue;
> +
> ret = request_irq(epf->db_msg[req].virq, epf_ntb_doorbell_handler,
> - 0, "pci_epf_vntb_db", ntb);
> + epf->db_msg[req].irq_flags, "pci_epf_vntb_db",
> + ntb);
>
> if (ret) {
> dev_err(&epf->dev,
> @@ -545,6 +576,22 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
> }
> }
>
> + if (epf->db_msg[0].bar != NO_BAR) {
> + for (i = 0; i < ntb->db_count; i++) {
> + msg = &epf->db_msg[i].msg;
> +
> + if (epf->db_msg[i].bar != barno) {
> + ret = -EINVAL;
> + goto err_free_irq;
> + }
> +
> + ntb->reg->db_data[i] = msg->data;
> + ntb->reg->db_offset[i] = epf->db_msg[i].offset;
> + }
> + goto out;
> + }
> +
> + /* Program inbound mapping for the doorbell */
> msg = &epf->db_msg[0].msg;
>
> high = 0;
> @@ -591,6 +638,7 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
> ntb->reg->db_offset[i] = offset;
> }
>
> +out:
> ntb->reg->db_entry_size = 0;
>
> ntb->msi_doorbell = true;
> @@ -598,9 +646,13 @@ static int epf_ntb_db_bar_init_msi_doorbell(struct epf_ntb *ntb,
> return 0;
>
> err_free_irq:
> - for (req--; req >= 0; req--)
> + for (req--; req >= 0; req--) {
> + if (epf_ntb_db_irq_is_duplicated(epf, req))
> + continue;
> free_irq(epf->db_msg[req].virq, ntb);
> + }
>
> +err_free_doorbell:
> pci_epf_free_doorbell(ntb->epf);
> return ret;
> }
> @@ -666,8 +718,11 @@ static void epf_ntb_db_bar_clear(struct epf_ntb *ntb)
> if (ntb->msi_doorbell) {
> int i;
>
> - for (i = 0; i < ntb->db_count; i++)
> + for (i = 0; i < ntb->db_count; i++) {
> + if (epf_ntb_db_irq_is_duplicated(ntb->epf, i))
> + continue;
> free_irq(ntb->epf->db_msg[i].virq, ntb);
> + }
> }
>
> if (ntb->epf->db_msg)
> --
> 2.51.0
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v10 6/7] PCI: endpoint: pci-epf-test: Reuse pre-exposed doorbell targets
2026-03-02 7:14 [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
` (4 preceding siblings ...)
2026-03-02 7:14 ` [PATCH v10 5/7] PCI: endpoint: pci-epf-vntb: Reuse pre-exposed doorbells and IRQ flags Koichiro Den
@ 2026-03-02 7:14 ` Koichiro Den
2026-03-23 18:41 ` Frank Li
2026-03-02 7:14 ` [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
6 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-02 7:14 UTC (permalink / raw)
To: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Frank Li, Bhanu Seshu Kumar Valluri,
Marco Crivellari, Shin'ichiro Kawasaki, Manikanta Maddireddy
Cc: linux-pci, linux-kernel, ntb
pci-epf-test advertises the doorbell target to the RC as a BAR number
and an offset, and the RC rings the doorbell with a single DWORD MMIO
write.
Some doorbell backends may report that the doorbell target is already
exposed via a platform-owned fixed BAR (db_msg[0].bar/offset). In that
case, reuse the pre-exposed window and do not reprogram the BAR with
pci_epc_set_bar().
Also honor db_msg[0].irq_flags when requesting the doorbell IRQ, and
only restore the original BAR mapping on disable if pci-epf-test
programmed it.
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
drivers/pci/endpoint/functions/pci-epf-test.c | 84 +++++++++++++------
1 file changed, 57 insertions(+), 27 deletions(-)
diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
index 14e61ebe1f11..b6c865b0883d 100644
--- a/drivers/pci/endpoint/functions/pci-epf-test.c
+++ b/drivers/pci/endpoint/functions/pci-epf-test.c
@@ -93,6 +93,7 @@ struct pci_epf_test {
bool dma_private;
const struct pci_epc_features *epc_features;
struct pci_epf_bar db_bar;
+ bool db_bar_programmed;
size_t bar_size[PCI_STD_NUM_BARS];
};
@@ -732,7 +733,9 @@ static void pci_epf_test_enable_doorbell(struct pci_epf_test *epf_test,
{
u32 status = le32_to_cpu(reg->status);
struct pci_epf *epf = epf_test->epf;
+ struct pci_epf_doorbell_msg *db;
struct pci_epc *epc = epf->epc;
+ unsigned long irq_flags;
struct msi_msg *msg;
enum pci_barno bar;
size_t offset;
@@ -742,13 +745,28 @@ static void pci_epf_test_enable_doorbell(struct pci_epf_test *epf_test,
if (ret)
goto set_status_err;
- msg = &epf->db_msg[0].msg;
- bar = pci_epc_get_next_free_bar(epf_test->epc_features, epf_test->test_reg_bar + 1);
- if (bar < BAR_0)
- goto err_doorbell_cleanup;
+ db = &epf->db_msg[0];
+ msg = &db->msg;
+ epf_test->db_bar_programmed = false;
+
+ if (db->bar != NO_BAR) {
+ /*
+ * The doorbell target is already exposed via a platform-owned
+ * fixed BAR
+ */
+ bar = db->bar;
+ offset = db->offset;
+ } else {
+ bar = pci_epc_get_next_free_bar(epf_test->epc_features,
+ epf_test->test_reg_bar + 1);
+ if (bar < BAR_0)
+ goto err_doorbell_cleanup;
+ }
+
+ irq_flags = epf->db_msg[0].irq_flags | IRQF_ONESHOT;
ret = request_threaded_irq(epf->db_msg[0].virq, NULL,
- pci_epf_test_doorbell_handler, IRQF_ONESHOT,
+ pci_epf_test_doorbell_handler, irq_flags,
"pci-ep-test-doorbell", epf_test);
if (ret) {
dev_err(&epf->dev,
@@ -760,22 +778,30 @@ static void pci_epf_test_enable_doorbell(struct pci_epf_test *epf_test,
reg->doorbell_data = cpu_to_le32(msg->data);
reg->doorbell_bar = cpu_to_le32(bar);
- msg = &epf->db_msg[0].msg;
- ret = pci_epf_align_inbound_addr(epf, bar, ((u64)msg->address_hi << 32) | msg->address_lo,
- &epf_test->db_bar.phys_addr, &offset);
+ if (db->bar == NO_BAR) {
+ ret = pci_epf_align_inbound_addr(epf, bar,
+ ((u64)msg->address_hi << 32) |
+ msg->address_lo,
+ &epf_test->db_bar.phys_addr,
+ &offset);
- if (ret)
- goto err_free_irq;
+ if (ret)
+ goto err_free_irq;
+ }
reg->doorbell_offset = cpu_to_le32(offset);
- epf_test->db_bar.barno = bar;
- epf_test->db_bar.size = epf->bar[bar].size;
- epf_test->db_bar.flags = epf->bar[bar].flags;
+ if (db->bar == NO_BAR) {
+ epf_test->db_bar.barno = bar;
+ epf_test->db_bar.size = epf->bar[bar].size;
+ epf_test->db_bar.flags = epf->bar[bar].flags;
- ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf_test->db_bar);
- if (ret)
- goto err_free_irq;
+ ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf_test->db_bar);
+ if (ret)
+ goto err_free_irq;
+
+ epf_test->db_bar_programmed = true;
+ }
status |= STATUS_DOORBELL_ENABLE_SUCCESS;
reg->status = cpu_to_le32(status);
@@ -805,17 +831,21 @@ static void pci_epf_test_disable_doorbell(struct pci_epf_test *epf_test,
free_irq(epf->db_msg[0].virq, epf_test);
pci_epf_test_doorbell_cleanup(epf_test);
- /*
- * The doorbell feature temporarily overrides the inbound translation
- * to point to the address stored in epf_test->db_bar.phys_addr, i.e.,
- * it calls set_bar() twice without ever calling clear_bar(), as
- * calling clear_bar() would clear the BAR's PCI address assigned by
- * the host. Thus, when disabling the doorbell, restore the inbound
- * translation to point to the memory allocated for the BAR.
- */
- ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf->bar[bar]);
- if (ret)
- goto set_status_err;
+ if (epf_test->db_bar_programmed) {
+ /*
+ * The doorbell feature temporarily overrides the inbound translation
+ * to point to the address stored in epf_test->db_bar.phys_addr, i.e.,
+ * it calls set_bar() twice without ever calling clear_bar(), as
+ * calling clear_bar() would clear the BAR's PCI address assigned by
+ * the host. Thus, when disabling the doorbell, restore the inbound
+ * translation to point to the memory allocated for the BAR.
+ */
+ ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf->bar[bar]);
+ if (ret)
+ goto set_status_err;
+
+ epf_test->db_bar_programmed = false;
+ }
status |= STATUS_DOORBELL_DISABLE_SUCCESS;
reg->status = cpu_to_le32(status);
--
2.51.0
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v10 6/7] PCI: endpoint: pci-epf-test: Reuse pre-exposed doorbell targets
2026-03-02 7:14 ` [PATCH v10 6/7] PCI: endpoint: pci-epf-test: Reuse pre-exposed doorbell targets Koichiro Den
@ 2026-03-23 18:41 ` Frank Li
0 siblings, 0 replies; 28+ messages in thread
From: Frank Li @ 2026-03-23 18:41 UTC (permalink / raw)
To: Koichiro Den
Cc: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 02, 2026 at 04:14:26PM +0900, Koichiro Den wrote:
> pci-epf-test advertises the doorbell target to the RC as a BAR number
> and an offset, and the RC rings the doorbell with a single DWORD MMIO
> write.
>
> Some doorbell backends may report that the doorbell target is already
> exposed via a platform-owned fixed BAR (db_msg[0].bar/offset). In that
> case, reuse the pre-exposed window and do not reprogram the BAR with
> pci_epc_set_bar().
>
> Also honor db_msg[0].irq_flags when requesting the doorbell IRQ, and
> only restore the original BAR mapping on disable if pci-epf-test
> programmed it.
>
> Tested-by: Niklas Cassel <cassel@kernel.org>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
Reviewed-by: Frank Li <Frank.Li@nxp.com>
> drivers/pci/endpoint/functions/pci-epf-test.c | 84 +++++++++++++------
> 1 file changed, 57 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> index 14e61ebe1f11..b6c865b0883d 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> @@ -93,6 +93,7 @@ struct pci_epf_test {
> bool dma_private;
> const struct pci_epc_features *epc_features;
> struct pci_epf_bar db_bar;
> + bool db_bar_programmed;
> size_t bar_size[PCI_STD_NUM_BARS];
> };
>
> @@ -732,7 +733,9 @@ static void pci_epf_test_enable_doorbell(struct pci_epf_test *epf_test,
> {
> u32 status = le32_to_cpu(reg->status);
> struct pci_epf *epf = epf_test->epf;
> + struct pci_epf_doorbell_msg *db;
> struct pci_epc *epc = epf->epc;
> + unsigned long irq_flags;
> struct msi_msg *msg;
> enum pci_barno bar;
> size_t offset;
> @@ -742,13 +745,28 @@ static void pci_epf_test_enable_doorbell(struct pci_epf_test *epf_test,
> if (ret)
> goto set_status_err;
>
> - msg = &epf->db_msg[0].msg;
> - bar = pci_epc_get_next_free_bar(epf_test->epc_features, epf_test->test_reg_bar + 1);
> - if (bar < BAR_0)
> - goto err_doorbell_cleanup;
> + db = &epf->db_msg[0];
> + msg = &db->msg;
> + epf_test->db_bar_programmed = false;
> +
> + if (db->bar != NO_BAR) {
> + /*
> + * The doorbell target is already exposed via a platform-owned
> + * fixed BAR
> + */
> + bar = db->bar;
> + offset = db->offset;
> + } else {
> + bar = pci_epc_get_next_free_bar(epf_test->epc_features,
> + epf_test->test_reg_bar + 1);
> + if (bar < BAR_0)
> + goto err_doorbell_cleanup;
> + }
> +
> + irq_flags = epf->db_msg[0].irq_flags | IRQF_ONESHOT;
>
> ret = request_threaded_irq(epf->db_msg[0].virq, NULL,
> - pci_epf_test_doorbell_handler, IRQF_ONESHOT,
> + pci_epf_test_doorbell_handler, irq_flags,
> "pci-ep-test-doorbell", epf_test);
> if (ret) {
> dev_err(&epf->dev,
> @@ -760,22 +778,30 @@ static void pci_epf_test_enable_doorbell(struct pci_epf_test *epf_test,
> reg->doorbell_data = cpu_to_le32(msg->data);
> reg->doorbell_bar = cpu_to_le32(bar);
>
> - msg = &epf->db_msg[0].msg;
> - ret = pci_epf_align_inbound_addr(epf, bar, ((u64)msg->address_hi << 32) | msg->address_lo,
> - &epf_test->db_bar.phys_addr, &offset);
> + if (db->bar == NO_BAR) {
> + ret = pci_epf_align_inbound_addr(epf, bar,
> + ((u64)msg->address_hi << 32) |
> + msg->address_lo,
> + &epf_test->db_bar.phys_addr,
> + &offset);
>
> - if (ret)
> - goto err_free_irq;
> + if (ret)
> + goto err_free_irq;
> + }
>
> reg->doorbell_offset = cpu_to_le32(offset);
>
> - epf_test->db_bar.barno = bar;
> - epf_test->db_bar.size = epf->bar[bar].size;
> - epf_test->db_bar.flags = epf->bar[bar].flags;
> + if (db->bar == NO_BAR) {
> + epf_test->db_bar.barno = bar;
> + epf_test->db_bar.size = epf->bar[bar].size;
> + epf_test->db_bar.flags = epf->bar[bar].flags;
>
> - ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf_test->db_bar);
> - if (ret)
> - goto err_free_irq;
> + ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf_test->db_bar);
> + if (ret)
> + goto err_free_irq;
> +
> + epf_test->db_bar_programmed = true;
> + }
>
> status |= STATUS_DOORBELL_ENABLE_SUCCESS;
> reg->status = cpu_to_le32(status);
> @@ -805,17 +831,21 @@ static void pci_epf_test_disable_doorbell(struct pci_epf_test *epf_test,
> free_irq(epf->db_msg[0].virq, epf_test);
> pci_epf_test_doorbell_cleanup(epf_test);
>
> - /*
> - * The doorbell feature temporarily overrides the inbound translation
> - * to point to the address stored in epf_test->db_bar.phys_addr, i.e.,
> - * it calls set_bar() twice without ever calling clear_bar(), as
> - * calling clear_bar() would clear the BAR's PCI address assigned by
> - * the host. Thus, when disabling the doorbell, restore the inbound
> - * translation to point to the memory allocated for the BAR.
> - */
> - ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf->bar[bar]);
> - if (ret)
> - goto set_status_err;
> + if (epf_test->db_bar_programmed) {
> + /*
> + * The doorbell feature temporarily overrides the inbound translation
> + * to point to the address stored in epf_test->db_bar.phys_addr, i.e.,
> + * it calls set_bar() twice without ever calling clear_bar(), as
> + * calling clear_bar() would clear the BAR's PCI address assigned by
> + * the host. Thus, when disabling the doorbell, restore the inbound
> + * translation to point to the memory allocated for the BAR.
> + */
> + ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, &epf->bar[bar]);
> + if (ret)
> + goto set_status_err;
> +
> + epf_test->db_bar_programmed = false;
> + }
>
> status |= STATUS_DOORBELL_DISABLE_SUCCESS;
> reg->status = cpu_to_le32(status);
> --
> 2.51.0
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-02 7:14 [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
` (5 preceding siblings ...)
2026-03-02 7:14 ` [PATCH v10 6/7] PCI: endpoint: pci-epf-test: Reuse pre-exposed doorbell targets Koichiro Den
@ 2026-03-02 7:14 ` Koichiro Den
2026-03-02 10:07 ` Niklas Cassel
2026-03-23 18:48 ` Frank Li
6 siblings, 2 replies; 28+ messages in thread
From: Koichiro Den @ 2026-03-02 7:14 UTC (permalink / raw)
To: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Frank Li, Bhanu Seshu Kumar Valluri,
Marco Crivellari, Shin'ichiro Kawasaki, Manikanta Maddireddy
Cc: linux-pci, linux-kernel, ntb
Some endpoint platforms cannot use platform MSI / GIC ITS to implement
EP-side doorbells. In those cases, EPF drivers cannot provide an
interrupt-driven doorbell and often fall back to polling.
Add an "embedded" doorbell backend that uses a controller-integrated
doorbell target (e.g. DesignWare integrated eDMA interrupt-emulation
doorbell).
The backend locates the doorbell register and a corresponding Linux IRQ
via the EPC aux-resource API. If the doorbell register is already
exposed via a fixed BAR mapping, provide BAR+offset. Otherwise provide
the DMA address returned by dma_map_resource() (which may be an IOVA
when an IOMMU is enabled) so EPF drivers can map it into BAR space.
When MSI doorbell allocation fails with -ENODEV,
pci_epf_alloc_doorbell() falls back to this embedded backend.
Signed-off-by: Koichiro Den <den@valinux.co.jp>
---
Changes since v9:
- Report the dma_map_resource() DMA address instead of the raw
physical address, to match the semantics of pre-existing MSI
doorbell when it is IOMMU-backed.
drivers/pci/endpoint/pci-ep-msi.c | 139 +++++++++++++++++++++++++++++-
include/linux/pci-epf.h | 8 ++
2 files changed, 144 insertions(+), 3 deletions(-)
diff --git a/drivers/pci/endpoint/pci-ep-msi.c b/drivers/pci/endpoint/pci-ep-msi.c
index 85fe46103220..331d84a79193 100644
--- a/drivers/pci/endpoint/pci-ep-msi.c
+++ b/drivers/pci/endpoint/pci-ep-msi.c
@@ -6,6 +6,8 @@
* Author: Frank Li <Frank.Li@nxp.com>
*/
+#include <linux/align.h>
+#include <linux/cleanup.h>
#include <linux/device.h>
#include <linux/export.h>
#include <linux/interrupt.h>
@@ -36,6 +38,117 @@ static void pci_epf_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
pci_epc_put(epc);
}
+static int pci_epf_alloc_doorbell_embedded(struct pci_epf *epf, u16 num_db)
+{
+ const struct pci_epc_aux_resource *doorbell = NULL;
+ struct pci_epf_doorbell_msg *msg;
+ struct pci_epc *epc = epf->epc;
+ struct device *dev = &epf->dev;
+ size_t map_size = 0, off = 0;
+ dma_addr_t iova_base = 0;
+ phys_addr_t phys_base;
+ int count, ret, i;
+ u64 addr;
+
+ count = pci_epc_get_aux_resources(epc, epf->func_no, epf->vfunc_no,
+ NULL, 0);
+ if (count == -EOPNOTSUPP || count == 0)
+ return -ENODEV;
+ if (count < 0)
+ return count;
+
+ struct pci_epc_aux_resource *res __free(kfree) =
+ kcalloc(count, sizeof(*res), GFP_KERNEL);
+ if (!res)
+ return -ENOMEM;
+
+ ret = pci_epc_get_aux_resources(epc, epf->func_no, epf->vfunc_no,
+ res, count);
+ if (ret == -EOPNOTSUPP || ret == 0)
+ return -ENODEV;
+ if (ret < 0)
+ return ret;
+
+ count = ret;
+
+ for (i = 0; i < count; i++) {
+ if (res[i].type == PCI_EPC_AUX_DOORBELL_MMIO) {
+ if (doorbell) {
+ dev_warn(dev,
+ "Duplicate DOORBELL_MMIO resource found\n");
+ continue;
+ }
+ doorbell = &res[i];
+ }
+ }
+ if (!doorbell)
+ return -ENODEV;
+
+ addr = doorbell->phys_addr;
+ if (!IS_ALIGNED(addr, sizeof(u32)))
+ return -EINVAL;
+
+ /*
+ * Reuse the pre-exposed BAR window if available. Otherwise map the MMIO
+ * doorbell resource here. Any required IOMMU mapping is handled
+ * internally, matching the MSI doorbell semantics.
+ */
+ if (doorbell->bar == NO_BAR) {
+ phys_base = addr & PAGE_MASK;
+ off = addr - phys_base;
+ map_size = PAGE_ALIGN(off + sizeof(u32));
+
+ iova_base = dma_map_resource(epc->dev.parent, phys_base,
+ map_size, DMA_FROM_DEVICE, 0);
+ if (dma_mapping_error(epc->dev.parent, iova_base))
+ return -EIO;
+
+ addr = iova_base + off;
+ }
+
+ msg = kcalloc(num_db, sizeof(*msg), GFP_KERNEL);
+ if (!msg) {
+ ret = -ENOMEM;
+ goto err_unmap;
+ }
+
+ /*
+ * Embedded doorbell backends (e.g. DesignWare eDMA interrupt emulation)
+ * typically provide a single IRQ and do not offer per-doorbell
+ * distinguishable address/data pairs. The EPC aux resource therefore
+ * exposes one DOORBELL_MMIO entry (u.db_mmio.irq).
+ *
+ * Still, pci_epf_alloc_doorbell() allows requesting multiple doorbells.
+ * For such backends we replicate the same address/data for each entry
+ * and mark the IRQ as shared (IRQF_SHARED). Consumers must treat them
+ * as equivalent "kick" doorbells.
+ */
+ for (i = 0; i < num_db; i++)
+ msg[i] = (struct pci_epf_doorbell_msg) {
+ .msg.address_lo = (u32)addr,
+ .msg.address_hi = (u32)(addr >> 32),
+ .msg.data = doorbell->u.db_mmio.data,
+ .virq = doorbell->u.db_mmio.irq,
+ .irq_flags = IRQF_SHARED,
+ .type = PCI_EPF_DOORBELL_EMBEDDED,
+ .bar = doorbell->bar,
+ .offset = (doorbell->bar == NO_BAR) ? 0 :
+ doorbell->bar_offset,
+ .iova_base = iova_base,
+ .iova_size = map_size,
+ };
+
+ epf->num_db = num_db;
+ epf->db_msg = msg;
+ return 0;
+
+err_unmap:
+ if (map_size)
+ dma_unmap_resource(epc->dev.parent, iova_base, map_size,
+ DMA_FROM_DEVICE, 0);
+ return ret;
+}
+
static int pci_epf_alloc_doorbell_msi(struct pci_epf *epf, u16 num_db)
{
struct pci_epf_doorbell_msg *msg;
@@ -109,18 +222,38 @@ int pci_epf_alloc_doorbell(struct pci_epf *epf, u16 num_db)
if (!ret)
return 0;
- dev_err(dev, "Failed to allocate doorbell: %d\n", ret);
- return ret;
+ /*
+ * Fall back to embedded doorbell only when platform MSI is unavailable
+ * for this EPC.
+ */
+ if (ret != -ENODEV)
+ return ret;
+
+ ret = pci_epf_alloc_doorbell_embedded(epf, num_db);
+ if (ret) {
+ dev_err(dev, "Failed to allocate doorbell: %d\n", ret);
+ return ret;
+ }
+
+ dev_info(dev, "Using embedded (DMA) doorbell fallback\n");
+ return 0;
}
EXPORT_SYMBOL_GPL(pci_epf_alloc_doorbell);
void pci_epf_free_doorbell(struct pci_epf *epf)
{
+ struct pci_epf_doorbell_msg *msg0;
+ struct pci_epc *epc = epf->epc;
+
if (!epf->db_msg)
return;
- if (epf->db_msg[0].type == PCI_EPF_DOORBELL_MSI)
+ msg0 = &epf->db_msg[0];
+ if (msg0->type == PCI_EPF_DOORBELL_MSI)
platform_device_msi_free_irqs_all(epf->epc->dev.parent);
+ else if (msg0->type == PCI_EPF_DOORBELL_EMBEDDED && msg0->iova_size)
+ dma_unmap_resource(epc->dev.parent, msg0->iova_base,
+ msg0->iova_size, DMA_FROM_DEVICE, 0);
kfree(epf->db_msg);
epf->db_msg = NULL;
diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
index cd747447a1ea..8a6c64a35890 100644
--- a/include/linux/pci-epf.h
+++ b/include/linux/pci-epf.h
@@ -171,6 +171,12 @@ enum pci_epf_doorbell_type {
* (NO_BAR if not)
* @offset: offset within @bar for the doorbell target (valid iff
* @bar != NO_BAR)
+ * @iova_base: Internal: base DMA address returned by dma_map_resource() for the
+ * embedded doorbell MMIO window (used only for unmapping). Valid
+ * when @type is PCI_EPF_DOORBELL_EMBEDDED and @iova_size is
+ * non-zero.
+ * @iova_size: Internal: size of the dma_map_resource() mapping at @iova_base.
+ * Zero when no mapping was created (e.g. pre-exposed fixed BAR).
*/
struct pci_epf_doorbell_msg {
struct msi_msg msg;
@@ -179,6 +185,8 @@ struct pci_epf_doorbell_msg {
enum pci_epf_doorbell_type type;
enum pci_barno bar;
resource_size_t offset;
+ dma_addr_t iova_base;
+ size_t iova_size;
};
/**
--
2.51.0
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-02 7:14 ` [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
@ 2026-03-02 10:07 ` Niklas Cassel
2026-03-23 18:48 ` Frank Li
1 sibling, 0 replies; 28+ messages in thread
From: Niklas Cassel @ 2026-03-02 10:07 UTC (permalink / raw)
To: Koichiro Den
Cc: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Frank Li, Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 02, 2026 at 04:14:27PM +0900, Koichiro Den wrote:
> Some endpoint platforms cannot use platform MSI / GIC ITS to implement
> EP-side doorbells. In those cases, EPF drivers cannot provide an
> interrupt-driven doorbell and often fall back to polling.
>
> Add an "embedded" doorbell backend that uses a controller-integrated
> doorbell target (e.g. DesignWare integrated eDMA interrupt-emulation
> doorbell).
>
> The backend locates the doorbell register and a corresponding Linux IRQ
> via the EPC aux-resource API. If the doorbell register is already
> exposed via a fixed BAR mapping, provide BAR+offset. Otherwise provide
> the DMA address returned by dma_map_resource() (which may be an IOVA
> when an IOMMU is enabled) so EPF drivers can map it into BAR space.
>
> When MSI doorbell allocation fails with -ENODEV,
> pci_epf_alloc_doorbell() falls back to this embedded backend.
>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
Tested-by: Niklas Cassel <cassel@kernel.org>
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-02 7:14 ` [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback Koichiro Den
2026-03-02 10:07 ` Niklas Cassel
@ 2026-03-23 18:48 ` Frank Li
2026-03-24 1:40 ` Koichiro Den
1 sibling, 1 reply; 28+ messages in thread
From: Frank Li @ 2026-03-23 18:48 UTC (permalink / raw)
To: Koichiro Den
Cc: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 02, 2026 at 04:14:27PM +0900, Koichiro Den wrote:
> Some endpoint platforms cannot use platform MSI / GIC ITS to implement
> EP-side doorbells. In those cases, EPF drivers cannot provide an
> interrupt-driven doorbell and often fall back to polling.
>
> Add an "embedded" doorbell backend that uses a controller-integrated
> doorbell target (e.g. DesignWare integrated eDMA interrupt-emulation
> doorbell).
>
> The backend locates the doorbell register and a corresponding Linux IRQ
> via the EPC aux-resource API. If the doorbell register is already
> exposed via a fixed BAR mapping, provide BAR+offset. Otherwise provide
> the DMA address returned by dma_map_resource() (which may be an IOVA
> when an IOMMU is enabled) so EPF drivers can map it into BAR space.
>
> When MSI doorbell allocation fails with -ENODEV,
> pci_epf_alloc_doorbell() falls back to this embedded backend.
>
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
...
> + */
> + if (doorbell->bar == NO_BAR) {
> + phys_base = addr & PAGE_MASK;
> + off = addr - phys_base;
> + map_size = PAGE_ALIGN(off + sizeof(u32));
> +
> + iova_base = dma_map_resource(epc->dev.parent, phys_base,
> + map_size, DMA_FROM_DEVICE, 0);
DB should write. Is it DMA_TO_DEVICE?
Frank
>
> /**
> --
> 2.51.0
>
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-23 18:48 ` Frank Li
@ 2026-03-24 1:40 ` Koichiro Den
2026-03-25 7:06 ` Niklas Cassel
0 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-24 1:40 UTC (permalink / raw)
To: Frank Li
Cc: Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Niklas Cassel, Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Mon, Mar 23, 2026 at 02:48:10PM -0400, Frank Li wrote:
> On Mon, Mar 02, 2026 at 04:14:27PM +0900, Koichiro Den wrote:
> > Some endpoint platforms cannot use platform MSI / GIC ITS to implement
> > EP-side doorbells. In those cases, EPF drivers cannot provide an
> > interrupt-driven doorbell and often fall back to polling.
> >
> > Add an "embedded" doorbell backend that uses a controller-integrated
> > doorbell target (e.g. DesignWare integrated eDMA interrupt-emulation
> > doorbell).
> >
> > The backend locates the doorbell register and a corresponding Linux IRQ
> > via the EPC aux-resource API. If the doorbell register is already
> > exposed via a fixed BAR mapping, provide BAR+offset. Otherwise provide
> > the DMA address returned by dma_map_resource() (which may be an IOVA
> > when an IOMMU is enabled) so EPF drivers can map it into BAR space.
> >
> > When MSI doorbell allocation fails with -ENODEV,
> > pci_epf_alloc_doorbell() falls back to this embedded backend.
> >
> > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > ---
> ...
> > + */
> > + if (doorbell->bar == NO_BAR) {
> > + phys_base = addr & PAGE_MASK;
> > + off = addr - phys_base;
> > + map_size = PAGE_ALIGN(off + sizeof(u32));
> > +
> > + iova_base = dma_map_resource(epc->dev.parent, phys_base,
> > + map_size, DMA_FROM_DEVICE, 0);
>
> DB should write. Is it DMA_TO_DEVICE?
Thanks for reviewing.
The write is initiated by the peer (RC) and lands in this EP-side doorbell
target, so from the EP side this is a DMA_FROM_DEVICE mapping.
Best regards,
Koichiro
>
> Frank
> >
> > /**
> > --
> > 2.51.0
> >
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-24 1:40 ` Koichiro Den
@ 2026-03-25 7:06 ` Niklas Cassel
2026-03-25 8:43 ` Koichiro Den
0 siblings, 1 reply; 28+ messages in thread
From: Niklas Cassel @ 2026-03-25 7:06 UTC (permalink / raw)
To: Koichiro Den
Cc: Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Tue, Mar 24, 2026 at 10:40:24AM +0900, Koichiro Den wrote:
> On Mon, Mar 23, 2026 at 02:48:10PM -0400, Frank Li wrote:
> > On Mon, Mar 02, 2026 at 04:14:27PM +0900, Koichiro Den wrote:
> > > Some endpoint platforms cannot use platform MSI / GIC ITS to implement
> > > EP-side doorbells. In those cases, EPF drivers cannot provide an
> > > interrupt-driven doorbell and often fall back to polling.
> > >
> > > Add an "embedded" doorbell backend that uses a controller-integrated
> > > doorbell target (e.g. DesignWare integrated eDMA interrupt-emulation
> > > doorbell).
> > >
> > > The backend locates the doorbell register and a corresponding Linux IRQ
> > > via the EPC aux-resource API. If the doorbell register is already
> > > exposed via a fixed BAR mapping, provide BAR+offset. Otherwise provide
> > > the DMA address returned by dma_map_resource() (which may be an IOVA
> > > when an IOMMU is enabled) so EPF drivers can map it into BAR space.
> > >
> > > When MSI doorbell allocation fails with -ENODEV,
> > > pci_epf_alloc_doorbell() falls back to this embedded backend.
> > >
> > > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > > ---
> > ...
> > > + */
> > > + if (doorbell->bar == NO_BAR) {
> > > + phys_base = addr & PAGE_MASK;
> > > + off = addr - phys_base;
> > > + map_size = PAGE_ALIGN(off + sizeof(u32));
> > > +
> > > + iova_base = dma_map_resource(epc->dev.parent, phys_base,
> > > + map_size, DMA_FROM_DEVICE, 0);
> >
> > DB should write. Is it DMA_TO_DEVICE?
>
> Thanks for reviewing.
>
> The write is initiated by the peer (RC) and lands in this EP-side doorbell
> target, so from the EP side this is a DMA_FROM_DEVICE mapping.
I don't understand.
If the RC side does a PCI TLP write, the iATU on the EP side should translate
that to an AXI write, no?
If the target address (doorbell register in this case) is only mapped as a
DMA_FROM_DEVICE (read), I would have expected that AXI write to result in an
IOMMU fault.
Is the IOMMU really running in strict mode and not in passthrough mode on
your platform?
Do you get an IOMMU fault if you change this code to DMA_TO_DEVICE ?
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-25 7:06 ` Niklas Cassel
@ 2026-03-25 8:43 ` Koichiro Den
2026-03-25 16:56 ` Niklas Cassel
0 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-25 8:43 UTC (permalink / raw)
To: Niklas Cassel
Cc: Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb
On Wed, Mar 25, 2026 at 08:06:24AM +0100, Niklas Cassel wrote:
> On Tue, Mar 24, 2026 at 10:40:24AM +0900, Koichiro Den wrote:
> > On Mon, Mar 23, 2026 at 02:48:10PM -0400, Frank Li wrote:
> > > On Mon, Mar 02, 2026 at 04:14:27PM +0900, Koichiro Den wrote:
> > > > Some endpoint platforms cannot use platform MSI / GIC ITS to implement
> > > > EP-side doorbells. In those cases, EPF drivers cannot provide an
> > > > interrupt-driven doorbell and often fall back to polling.
> > > >
> > > > Add an "embedded" doorbell backend that uses a controller-integrated
> > > > doorbell target (e.g. DesignWare integrated eDMA interrupt-emulation
> > > > doorbell).
> > > >
> > > > The backend locates the doorbell register and a corresponding Linux IRQ
> > > > via the EPC aux-resource API. If the doorbell register is already
> > > > exposed via a fixed BAR mapping, provide BAR+offset. Otherwise provide
> > > > the DMA address returned by dma_map_resource() (which may be an IOVA
> > > > when an IOMMU is enabled) so EPF drivers can map it into BAR space.
> > > >
> > > > When MSI doorbell allocation fails with -ENODEV,
> > > > pci_epf_alloc_doorbell() falls back to this embedded backend.
> > > >
> > > > Signed-off-by: Koichiro Den <den@valinux.co.jp>
> > > > ---
> > > ...
> > > > + */
> > > > + if (doorbell->bar == NO_BAR) {
> > > > + phys_base = addr & PAGE_MASK;
> > > > + off = addr - phys_base;
> > > > + map_size = PAGE_ALIGN(off + sizeof(u32));
> > > > +
> > > > + iova_base = dma_map_resource(epc->dev.parent, phys_base,
> > > > + map_size, DMA_FROM_DEVICE, 0);
> > >
> > > DB should write. Is it DMA_TO_DEVICE?
> >
> > Thanks for reviewing.
> >
> > The write is initiated by the peer (RC) and lands in this EP-side doorbell
> > target, so from the EP side this is a DMA_FROM_DEVICE mapping.
>
> I don't understand.
>
> If the RC side does a PCI TLP write, the iATU on the EP side should translate
> that to an AXI write, no?
Yes.
>
> If the target address (doorbell register in this case) is only mapped as a
> DMA_FROM_DEVICE (read), I would have expected that AXI write to result in an
> IOMMU fault.
My understanding is that the inbound MWr becomes a local write on the EP side.
Here, the EP controller needs write permission to the doorbell target, so
DMA_FROM_DEVICE is intentional.
As noted in the cover letter, IOMMU coverage was tested.
>
> Is the IOMMU really running in strict mode and not in passthrough mode on
> your platform?
>
> Do you get an IOMMU fault if you change this code to DMA_TO_DEVICE ?
Yes, changing it to DMA_TO_DEVICE triggers an IOMMU fault. I've confirmed this
experimentally. This also indicates that it's not passthrough for this mapping
on my setup.
Best regards,
Koichiro
>
>
> Kind regards,
> Niklas
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-25 8:43 ` Koichiro Den
@ 2026-03-25 16:56 ` Niklas Cassel
2026-03-26 8:49 ` Koichiro Den
0 siblings, 1 reply; 28+ messages in thread
From: Niklas Cassel @ 2026-03-25 16:56 UTC (permalink / raw)
To: Koichiro Den
Cc: Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb, robin.murphy
On Wed, Mar 25, 2026 at 05:43:58PM +0900, Koichiro Den wrote:
> > > > ...
> > > > > + */
> > > > > + if (doorbell->bar == NO_BAR) {
> > > > > + phys_base = addr & PAGE_MASK;
> > > > > + off = addr - phys_base;
> > > > > + map_size = PAGE_ALIGN(off + sizeof(u32));
> > > > > +
> > > > > + iova_base = dma_map_resource(epc->dev.parent, phys_base,
> > > > > + map_size, DMA_FROM_DEVICE, 0);
> > > >
> > > > DB should write. Is it DMA_TO_DEVICE?
> > >
> > > Thanks for reviewing.
> > >
> > > The write is initiated by the peer (RC) and lands in this EP-side doorbell
> > > target, so from the EP side this is a DMA_FROM_DEVICE mapping.
> >
> > I don't understand.
> >
> > If the RC side does a PCI TLP write, the iATU on the EP side should translate
> > that to an AXI write, no?
>
> Yes.
>
> >
> > If the target address (doorbell register in this case) is only mapped as a
> > DMA_FROM_DEVICE (read), I would have expected that AXI write to result in an
> > IOMMU fault.
>
> My understanding is that the inbound MWr becomes a local write on the EP side.
> Here, the EP controller needs write permission to the doorbell target, so
> DMA_FROM_DEVICE is intentional.
>
> As noted in the cover letter, IOMMU coverage was tested.
>
> >
> > Is the IOMMU really running in strict mode and not in passthrough mode on
> > your platform?
> >
> > Do you get an IOMMU fault if you change this code to DMA_TO_DEVICE ?
>
> Yes, changing it to DMA_TO_DEVICE triggers an IOMMU fault. I've confirmed this
> experimentally. This also indicates that it's not passthrough for this mapping
> on my setup.
I still think it sounds wrong as per:
https://docs.kernel.org/core-api/dma-api-howto.html#dma-direction
The transaction is a write from
PCIe bus -> PCIe controller iATU -> internal bus -> IOMMU -> PCIe controller
(the same controller as initiated the transaction).
But I guess I'm just an idiot :)
Would be interesting why this is not working like normal (when using buffers):
"For Networking drivers, it’s a rather simple affair.
For transmit packets, map/unmap them with the DMA_TO_DEVICE direction specifier.
For receive packets, just the opposite, map/unmap them with the DMA_FROM_DEVICE
direction specifier."
Pehaps it is because it is the same device that does a write, that ends up
writing to it a self? so source an destination is the same?
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-25 16:56 ` Niklas Cassel
@ 2026-03-26 8:49 ` Koichiro Den
2026-03-26 9:59 ` Niklas Cassel
0 siblings, 1 reply; 28+ messages in thread
From: Koichiro Den @ 2026-03-26 8:49 UTC (permalink / raw)
To: Niklas Cassel
Cc: Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb, robin.murphy
On Wed, Mar 25, 2026 at 05:56:36PM +0100, Niklas Cassel wrote:
> On Wed, Mar 25, 2026 at 05:43:58PM +0900, Koichiro Den wrote:
> > > > > ...
> > > > > > + */
> > > > > > + if (doorbell->bar == NO_BAR) {
> > > > > > + phys_base = addr & PAGE_MASK;
> > > > > > + off = addr - phys_base;
> > > > > > + map_size = PAGE_ALIGN(off + sizeof(u32));
> > > > > > +
> > > > > > + iova_base = dma_map_resource(epc->dev.parent, phys_base,
> > > > > > + map_size, DMA_FROM_DEVICE, 0);
> > > > >
> > > > > DB should write. Is it DMA_TO_DEVICE?
> > > >
> > > > Thanks for reviewing.
> > > >
> > > > The write is initiated by the peer (RC) and lands in this EP-side doorbell
> > > > target, so from the EP side this is a DMA_FROM_DEVICE mapping.
> > >
> > > I don't understand.
> > >
> > > If the RC side does a PCI TLP write, the iATU on the EP side should translate
> > > that to an AXI write, no?
> >
> > Yes.
> >
> > >
> > > If the target address (doorbell register in this case) is only mapped as a
> > > DMA_FROM_DEVICE (read), I would have expected that AXI write to result in an
> > > IOMMU fault.
> >
> > My understanding is that the inbound MWr becomes a local write on the EP side.
> > Here, the EP controller needs write permission to the doorbell target, so
> > DMA_FROM_DEVICE is intentional.
> >
> > As noted in the cover letter, IOMMU coverage was tested.
> >
> > >
> > > Is the IOMMU really running in strict mode and not in passthrough mode on
> > > your platform?
> > >
> > > Do you get an IOMMU fault if you change this code to DMA_TO_DEVICE ?
> >
> > Yes, changing it to DMA_TO_DEVICE triggers an IOMMU fault. I've confirmed this
> > experimentally. This also indicates that it's not passthrough for this mapping
> > on my setup.
>
> I still think it sounds wrong as per:
> https://docs.kernel.org/core-api/dma-api-howto.html#dma-direction
>
> The transaction is a write from
> PCIe bus -> PCIe controller iATU -> internal bus -> IOMMU -> PCIe controller
> (the same controller as initiated the transaction).
Yes, I think we're on the same page about this path itself.
On my R-Car S4 setup, changing this to DMA_TO_DEVICE consistently triggers an
IOMMU fault, so at least on this platform the local path used for the doorbell
mapping is IOMMU-visible. That is the case this dma_map_resource() is intended
to cover.
For that path, my understanding is that the doorbell access ends up as a local
write on the EP side, so it needs write permission, hence DMA_FROM_DEVICE.
> Would be interesting why this is not working like normal (when using buffers):
> "For Networking drivers, it’s a rather simple affair.
> For transmit packets, map/unmap them with the DMA_TO_DEVICE direction specifier.
> For receive packets, just the opposite, map/unmap them with the DMA_FROM_DEVICE
> direction specifier."
I think the closer analogy is RX: the data comes from outside, but the device
writes to the target, so it needs write permission.
>
> Pehaps it is because it is the same device that does a write, that ends up
> writing to it a self? so source an destination is the same?
I'm not sure that "same device" aspect matters here.
(Separately, for the DWC case, I'm also re-checking whether the embedded doorbell
write should ideally stay on the native TRGT0->LBC->CDM path instead, which
would be IOMMU-invisible. I don't want to overstate that point yet, though.)
Best regards,
Koichiro
>
>
> Kind regards,
> Niklas
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-26 8:49 ` Koichiro Den
@ 2026-03-26 9:59 ` Niklas Cassel
2026-03-26 10:25 ` Niklas Cassel
0 siblings, 1 reply; 28+ messages in thread
From: Niklas Cassel @ 2026-03-26 9:59 UTC (permalink / raw)
To: Koichiro Den, Robin Murphy, Marek Szyprowski, Joerg Roedel,
Will Deacon
Cc: Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb, iommu
On Thu, Mar 26, 2026 at 05:49:13PM +0900, Koichiro Den wrote:
> >
> > The transaction is a write from
> > PCIe bus -> PCIe controller iATU -> internal bus -> IOMMU -> PCIe controller
> > (the same controller as initiated the transaction).
>
> Yes, I think we're on the same page about this path itself.
>
> On my R-Car S4 setup, changing this to DMA_TO_DEVICE consistently triggers an
> IOMMU fault, so at least on this platform the local path used for the doorbell
> mapping is IOMMU-visible. That is the case this dma_map_resource() is intended
> to cover.
>
> For that path, my understanding is that the doorbell access ends up as a local
> write on the EP side, so it needs write permission, hence DMA_FROM_DEVICE.
>
> > Would be interesting why this is not working like normal (when using buffers):
> > "For Networking drivers, it’s a rather simple affair.
> > For transmit packets, map/unmap them with the DMA_TO_DEVICE direction specifier.
> > For receive packets, just the opposite, map/unmap them with the DMA_FROM_DEVICE
> > direction specifier."
>
> I think the closer analogy is RX: the data comes from outside, but the device
> writes to the target, so it needs write permission.
I think it is from the PoV from the IOMMU, is the transaction a Read by a device
or a Write by a device?
For a NIC driver:
For a RX packet, the data is coming from the device to the memory.
device is doing a transaction to memory.
For a TX packet, the data is going from the memory to the device.
In our case, the data is coming from the device, to a device.
Almost like a P2P DMA, but in our case, both devices are the same, so
using the P2P DMA API like pci_p2pdma_add_resource() seems unnecessary.
So should it be DMA_BIDIRECTIONAL ? :)
I understand that for the R-Car S4 Spider IOMMU, it is sufficient to map
it as DMA_FROM_DEVICE. I just want to be sure that on some other IOMMU,
they might consider it sufficient to map this as DMA_TO_DEVICE (because
it is also a transaction going to a device).
I just want to make sure that the code works on more than one IOMMU.
Perhaps some IOMMU experts could help chime in.
Note that I am happy to merge the code as is, as it obviously works on the
only platform that this has been tested on (R-Car S4 Spider), and if other
platform tries to run this test case, if their IOMMU works differently, it
will scream and they will report it to the list. So all good.
I'm mostly want to know how the DMA-API is supposed to be used in this
specific scenario (device doing a write transaction to the same device).
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-26 9:59 ` Niklas Cassel
@ 2026-03-26 10:25 ` Niklas Cassel
2026-03-26 12:12 ` Robin Murphy
0 siblings, 1 reply; 28+ messages in thread
From: Niklas Cassel @ 2026-03-26 10:25 UTC (permalink / raw)
To: Koichiro Den, Robin Murphy, Marek Szyprowski, Joerg Roedel,
Will Deacon
Cc: Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb, iommu
On Thu, Mar 26, 2026 at 10:59:26AM +0100, Niklas Cassel wrote:
> On Thu, Mar 26, 2026 at 05:49:13PM +0900, Koichiro Den wrote:
> > >
> > > The transaction is a write from
> > > PCIe bus -> PCIe controller iATU -> internal bus -> IOMMU -> PCIe controller
> > > (the same controller as initiated the transaction).
> >
> > Yes, I think we're on the same page about this path itself.
> >
> > On my R-Car S4 setup, changing this to DMA_TO_DEVICE consistently triggers an
> > IOMMU fault, so at least on this platform the local path used for the doorbell
> > mapping is IOMMU-visible. That is the case this dma_map_resource() is intended
> > to cover.
> >
> > For that path, my understanding is that the doorbell access ends up as a local
> > write on the EP side, so it needs write permission, hence DMA_FROM_DEVICE.
> >
> > > Would be interesting why this is not working like normal (when using buffers):
> > > "For Networking drivers, it’s a rather simple affair.
> > > For transmit packets, map/unmap them with the DMA_TO_DEVICE direction specifier.
> > > For receive packets, just the opposite, map/unmap them with the DMA_FROM_DEVICE
> > > direction specifier."
> >
> > I think the closer analogy is RX: the data comes from outside, but the device
> > writes to the target, so it needs write permission.
>
> I think it is from the PoV from the IOMMU, is the transaction a Read by a device
> or a Write by a device?
>
> For a NIC driver:
> For a RX packet, the data is coming from the device to the memory.
> device is doing a transaction to memory.
> For a TX packet, the data is going from the memory to the device.
>
> In our case, the data is coming from the device, to a device.
>
> Almost like a P2P DMA, but in our case, both devices are the same, so
> using the P2P DMA API like pci_p2pdma_add_resource() seems unnecessary.
>
> So should it be DMA_BIDIRECTIONAL ? :)
>
> I understand that for the R-Car S4 Spider IOMMU, it is sufficient to map
> it as DMA_FROM_DEVICE. I just want to be sure that on some other IOMMU,
> they might consider it sufficient to map this as DMA_TO_DEVICE (because
> it is also a transaction going to a device).
>
> I just want to make sure that the code works on more than one IOMMU.
>
> Perhaps some IOMMU experts could help chime in.
>
>
> Note that I am happy to merge the code as is, as it obviously works on the
> only platform that this has been tested on (R-Car S4 Spider), and if other
> platform tries to run this test case, if their IOMMU works differently, it
> will scream and they will report it to the list. So all good.
>
> I'm mostly want to know how the DMA-API is supposed to be used in this
> specific scenario (device doing a write transaction to the same device).
I guess if a device will be reading or writing from this IOVA that is
created by IOMMU by the dma_map() call...
The device will only be writing to this IOVA.
The device will never be reading from the IOVA (since the physical address
is a register in the device itself, we will never supply this IOVA for the
device to read from).
DMA_FROM_DEVICE seems correct in all cases. DMA_BIDIRECTIONAL seems wrong
since the device will never read from this IOVA. Sorry for the noise.
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-26 10:25 ` Niklas Cassel
@ 2026-03-26 12:12 ` Robin Murphy
2026-03-26 14:38 ` Koichiro Den
0 siblings, 1 reply; 28+ messages in thread
From: Robin Murphy @ 2026-03-26 12:12 UTC (permalink / raw)
To: Niklas Cassel, Koichiro Den, Marek Szyprowski, Joerg Roedel,
Will Deacon
Cc: Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb, iommu
On 2026-03-26 10:25 am, Niklas Cassel wrote:
> On Thu, Mar 26, 2026 at 10:59:26AM +0100, Niklas Cassel wrote:
>> On Thu, Mar 26, 2026 at 05:49:13PM +0900, Koichiro Den wrote:
>>>>
>>>> The transaction is a write from
>>>> PCIe bus -> PCIe controller iATU -> internal bus -> IOMMU -> PCIe controller
>>>> (the same controller as initiated the transaction).
>>>
>>> Yes, I think we're on the same page about this path itself.
>>>
>>> On my R-Car S4 setup, changing this to DMA_TO_DEVICE consistently triggers an
>>> IOMMU fault, so at least on this platform the local path used for the doorbell
>>> mapping is IOMMU-visible. That is the case this dma_map_resource() is intended
>>> to cover.
>>>
>>> For that path, my understanding is that the doorbell access ends up as a local
>>> write on the EP side, so it needs write permission, hence DMA_FROM_DEVICE.
>>>
>>>> Would be interesting why this is not working like normal (when using buffers):
>>>> "For Networking drivers, it’s a rather simple affair.
>>>> For transmit packets, map/unmap them with the DMA_TO_DEVICE direction specifier.
>>>> For receive packets, just the opposite, map/unmap them with the DMA_FROM_DEVICE
>>>> direction specifier."
>>>
>>> I think the closer analogy is RX: the data comes from outside, but the device
>>> writes to the target, so it needs write permission.
>>
>> I think it is from the PoV from the IOMMU, is the transaction a Read by a device
>> or a Write by a device?
>>
>> For a NIC driver:
>> For a RX packet, the data is coming from the device to the memory.
>> device is doing a transaction to memory.
>> For a TX packet, the data is going from the memory to the device.
>>
>> In our case, the data is coming from the device, to a device.
>>
>> Almost like a P2P DMA, but in our case, both devices are the same, so
>> using the P2P DMA API like pci_p2pdma_add_resource() seems unnecessary.
>>
>> So should it be DMA_BIDIRECTIONAL ? :)
>>
>> I understand that for the R-Car S4 Spider IOMMU, it is sufficient to map
>> it as DMA_FROM_DEVICE. I just want to be sure that on some other IOMMU,
>> they might consider it sufficient to map this as DMA_TO_DEVICE (because
>> it is also a transaction going to a device).
>>
>> I just want to make sure that the code works on more than one IOMMU.
>>
>> Perhaps some IOMMU experts could help chime in.
>>
>>
>> Note that I am happy to merge the code as is, as it obviously works on the
>> only platform that this has been tested on (R-Car S4 Spider), and if other
>> platform tries to run this test case, if their IOMMU works differently, it
>> will scream and they will report it to the list. So all good.
>>
>> I'm mostly want to know how the DMA-API is supposed to be used in this
>> specific scenario (device doing a write transaction to the same device).
>
> I guess if a device will be reading or writing from this IOVA that is
> created by IOMMU by the dma_map() call...
>
> The device will only be writing to this IOVA.
> The device will never be reading from the IOVA (since the physical address
> is a register in the device itself, we will never supply this IOVA for the
> device to read from).
>
> DMA_FROM_DEVICE seems correct in all cases. DMA_BIDIRECTIONAL seems wrong
> since the device will never read from this IOVA. Sorry for the noise.
DMA_FROM_DEVICE means the device will be sending write requests to the
address; DMA_TO_DEVICE means the device will be sending read requests to
the address; DMA_BIDIRECTIONAL means the device might send either or
both. Never does the *address* try to pull or push data anywhere of its
own accord, regardless of whether it's RAM or MMIO or whatever. Hence
why we don't have or need DMA_{TO,FROM}_MEMORY either ;)
Indeed if the endpoint controller could somehow recognise that an
incoming write was a "doorbell" write based on the incoming BAR offset
and internally short-circuit that to directly raise its own interrupt
*without* propagating a transaction into the rest of the system, then it
wouldn't need the DMA mapping. However, if it's just set up to forward
ranges of BAR address to ranges of system addresses, and there's an
IOMMU in that path (such that the iATU's "system addresses" at that
point are actually IOVAs), then all that really matters is what requests
the IOMMU is going to see coming out of the iATU, irrespective of
whether a particular system address after translation happens to be an
MSI controller, and that MSI controller coincidentally happens to live
right next to the iATU itself.
(Of course the DMA API itself does also care about the general
distinction of the target address being "memory" or "not memory", as
implied by dma_map_page/single/sg() vs. dma_map_resource(), but that's
mostly just for the software aspects of knowing not to attempt cache
maintenance etc. on the "not memory" where it wouldn't be meaningful or
valid.)
Thanks,
Robin.
>
>
> Kind regards,
> Niklas
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v10 7/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
2026-03-26 12:12 ` Robin Murphy
@ 2026-03-26 14:38 ` Koichiro Den
0 siblings, 0 replies; 28+ messages in thread
From: Koichiro Den @ 2026-03-26 14:38 UTC (permalink / raw)
To: Robin Murphy
Cc: Niklas Cassel, Marek Szyprowski, Joerg Roedel, Will Deacon,
Frank Li, Jingoo Han, Manivannan Sadhasivam, Lorenzo Pieralisi,
Krzysztof Wilczyński, Rob Herring, Bjorn Helgaas,
Kishon Vijay Abraham I, Jon Mason, Dave Jiang, Allen Hubbe,
Bhanu Seshu Kumar Valluri, Marco Crivellari,
Shin'ichiro Kawasaki, Manikanta Maddireddy, linux-pci,
linux-kernel, ntb, iommu
On Thu, Mar 26, 2026 at 12:12:25PM +0000, Robin Murphy wrote:
> On 2026-03-26 10:25 am, Niklas Cassel wrote:
> > On Thu, Mar 26, 2026 at 10:59:26AM +0100, Niklas Cassel wrote:
> > > On Thu, Mar 26, 2026 at 05:49:13PM +0900, Koichiro Den wrote:
> > > > >
> > > > > The transaction is a write from
> > > > > PCIe bus -> PCIe controller iATU -> internal bus -> IOMMU -> PCIe controller
> > > > > (the same controller as initiated the transaction).
> > > >
> > > > Yes, I think we're on the same page about this path itself.
> > > >
> > > > On my R-Car S4 setup, changing this to DMA_TO_DEVICE consistently triggers an
> > > > IOMMU fault, so at least on this platform the local path used for the doorbell
> > > > mapping is IOMMU-visible. That is the case this dma_map_resource() is intended
> > > > to cover.
> > > >
> > > > For that path, my understanding is that the doorbell access ends up as a local
> > > > write on the EP side, so it needs write permission, hence DMA_FROM_DEVICE.
> > > >
> > > > > Would be interesting why this is not working like normal (when using buffers):
> > > > > "For Networking drivers, it’s a rather simple affair.
> > > > > For transmit packets, map/unmap them with the DMA_TO_DEVICE direction specifier.
> > > > > For receive packets, just the opposite, map/unmap them with the DMA_FROM_DEVICE
> > > > > direction specifier."
> > > >
> > > > I think the closer analogy is RX: the data comes from outside, but the device
> > > > writes to the target, so it needs write permission.
> > >
> > > I think it is from the PoV from the IOMMU, is the transaction a Read by a device
> > > or a Write by a device?
> > >
> > > For a NIC driver:
> > > For a RX packet, the data is coming from the device to the memory.
> > > device is doing a transaction to memory.
> > > For a TX packet, the data is going from the memory to the device.
> > >
> > > In our case, the data is coming from the device, to a device.
> > >
> > > Almost like a P2P DMA, but in our case, both devices are the same, so
> > > using the P2P DMA API like pci_p2pdma_add_resource() seems unnecessary.
> > >
> > > So should it be DMA_BIDIRECTIONAL ? :)
> > >
> > > I understand that for the R-Car S4 Spider IOMMU, it is sufficient to map
> > > it as DMA_FROM_DEVICE. I just want to be sure that on some other IOMMU,
> > > they might consider it sufficient to map this as DMA_TO_DEVICE (because
> > > it is also a transaction going to a device).
> > >
> > > I just want to make sure that the code works on more than one IOMMU.
> > >
> > > Perhaps some IOMMU experts could help chime in.
> > >
> > >
> > > Note that I am happy to merge the code as is, as it obviously works on the
> > > only platform that this has been tested on (R-Car S4 Spider), and if other
> > > platform tries to run this test case, if their IOMMU works differently, it
> > > will scream and they will report it to the list. So all good.
> > >
> > > I'm mostly want to know how the DMA-API is supposed to be used in this
> > > specific scenario (device doing a write transaction to the same device).
> >
> > I guess if a device will be reading or writing from this IOVA that is
> > created by IOMMU by the dma_map() call...
> >
> > The device will only be writing to this IOVA.
> > The device will never be reading from the IOVA (since the physical address
> > is a register in the device itself, we will never supply this IOVA for the
> > device to read from).
> >
> > DMA_FROM_DEVICE seems correct in all cases. DMA_BIDIRECTIONAL seems wrong
> > since the device will never read from this IOVA. Sorry for the noise.
>
> DMA_FROM_DEVICE means the device will be sending write requests to the
> address; DMA_TO_DEVICE means the device will be sending read requests to the
> address; DMA_BIDIRECTIONAL means the device might send either or both. Never
> does the *address* try to pull or push data anywhere of its own accord,
> regardless of whether it's RAM or MMIO or whatever. Hence why we don't have
> or need DMA_{TO,FROM}_MEMORY either ;)
>
> Indeed if the endpoint controller could somehow recognise that an incoming
> write was a "doorbell" write based on the incoming BAR offset and internally
> short-circuit that to directly raise its own interrupt *without* propagating
> a transaction into the rest of the system, then it wouldn't need the DMA
> mapping. However, if it's just set up to forward ranges of BAR address to
> ranges of system addresses, and there's an IOMMU in that path (such that the
> iATU's "system addresses" at that point are actually IOVAs), then all that
> really matters is what requests the IOMMU is going to see coming out of the
> iATU, irrespective of whether a particular system address after translation
> happens to be an MSI controller, and that MSI controller coincidentally
> happens to live right next to the iATU itself.
Thanks for the follow-up comment.
One reason I added dma_map_resource() for the embedded fallback in v10 was to
keep it aligned with the (pre-existing) pure MSI doorbell case under
CONFIG_IRQ_MSI_IOMMU=y. The fallback to the embedded doorbell occurs behind
pci_epf_alloc_doorbell(), so the API caller can expect IOVA when appropriate
regardless of whether it's pure MSI or the fallback.
iommu_dma_get_msi_page() maps with IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO, so
using dma_map_resource(..., DMA_FROM_DEVICE) for the embedded doorbell case
also follows the same write-permission model when the forwarding path is
IOMMU-visible.
Best regards,
Koichiro
>
> (Of course the DMA API itself does also care about the general distinction
> of the target address being "memory" or "not memory", as implied by
> dma_map_page/single/sg() vs. dma_map_resource(), but that's mostly just for
> the software aspects of knowing not to attempt cache maintenance etc. on the
> "not memory" where it wouldn't be meaningful or valid.)
>
> Thanks,
> Robin.
>
> >
> >
> > Kind regards,
> > Niklas
>
^ permalink raw reply [flat|nested] 28+ messages in thread