[PATCH 0/2] PCI: dwc: Support large BARs on UltraRISC DP1000

[PATCH 0/2] PCI: dwc: Support large BARs on UltraRISC DP1000

[Xincheng Zhang]

This series lets DWC host drivers skip generic outbound iATU programming
for host bridge MEM ranges, which is useful for some compute cards' very
large BAR size demand (>100GB).

DP1000 has 16 outbound iATU windows, and each window can map up to
4GB. This limits its allocatable MEM64 BAR size to 4GB*(16-3)=52GB
(-3 for CFG, I/O and MEM32).

But some EPs like compute cards demand large BAR size that exceeds the
mapping limit of the 13 iATUs. DWC IP spec allows RC to skip MEM iATU setup
and simply uses a direct mapping. By introducing a `bypass_ob_mem_iatu`
flag at the RC level, this patch enables large BARs on UR-DP1000
RISC-V CPU.

Tested on UltraRISC DP1000 EVB and Titan board. The system booted
successfully with PCIe links up on all three controllers and NVMe rootfs
mounted. MMIO read/write speed test within the large BAR space goes well.

Xincheng Zhang (2):
  PCI: dwc: Allow drivers to skip MEM outbound iATU setup
  PCI: ultrarisc: Skip MEM outbound iATU setup on DP1000

 drivers/pci/controller/dwc/pcie-designware-host.c | 6 +++---
 drivers/pci/controller/dwc/pcie-designware.h      | 1 +
 drivers/pci/controller/dwc/pcie-ultrarisc.c       | 6 ++++++
 3 files changed, 10 insertions(+), 3 deletions(-)

-- 
2.34.1

[PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[Xincheng Zhang]

Some DWC-based controllers do not require outbound iATU windows for MEM
transactions. For those platforms, programming outbound iATU windows for
host bridge MEM ranges is unnecessary and may constrain the available MEM
aperture.

Add dw_pcie_rp::bypass_ob_mem_iatu so drivers can skip generic MEM iATU
setup while using the common DWC host init path.

Existing drivers keep the current behavior because the flag defaults to
false. The I/O iATU setup does not use the MEM resource iterator, and the
iterator may be uninitialized when MEM setup is skipped. Avoid using it in
the I/O iATU error path.

Signed-off-by: Xincheng Zhang <zhangxincheng@ultrarisc.com>
---
 drivers/pci/controller/dwc/pcie-designware-host.c | 6 +++---
 drivers/pci/controller/dwc/pcie-designware.h      | 1 +
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index f5a38e6fd8d79..3c5f5ff080818 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -911,7 +911,8 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 	resource_list_for_each_entry(entry, &pp->bridge->windows) {
 		resource_size_t res_size;
 
-		if (resource_type(entry->res) != IORESOURCE_MEM)
+		if (pp->bypass_ob_mem_iatu ||
+		    resource_type(entry->res) != IORESOURCE_MEM)
 			continue;
 
 		atu.type = PCIE_TLP_TYPE_MEM_RDWR;
@@ -964,8 +965,7 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
 			ret = dw_pcie_prog_outbound_atu(pci, &atu);
 			if (ret) {
-				dev_err(pci->dev, "Failed to set IO range %pr\n",
-					entry->res);
+				dev_err(pci->dev, "Failed to set IO range\n");
 				return ret;
 			}
 			ob_iatu_index++;
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 0735ae9409240..d89badcca2689 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -469,6 +469,7 @@ struct dw_pcie_rp {
 	struct pci_config_window *cfg;
 	bool			ecam_enabled;
 	bool			native_ecam;
+	bool			bypass_ob_mem_iatu;
 	bool                    skip_l23_ready;
 	bool			skip_pwrctrl_off;
 };
-- 
2.34.1

[PATCH 2/2] PCI: ultrarisc: Skip MEM outbound iATU setup on DP1000

[Xincheng Zhang]

DP1000 has 16 outbound iATU windows and each window can map up to 4GB. One
window is used for CFG accesses, one for I/O space and one for the 32-bit
MEM aperture, leaving only 13 windows for the 64-bit MEM aperture.

This limits the 64-bit MEM aperture that can be described with outbound
iATU windows. DP1000 routes MEM transactions without using outbound iATU
windows, so skip generic DWC MEM outbound iATU setup to allow a larger
64-bit MEM aperture.

Set dw_pcie_rp::bypass_ob_mem_iatu for this controller.

Signed-off-by: Xincheng Zhang <zhangxincheng@ultrarisc.com>
---
 drivers/pci/controller/dwc/pcie-ultrarisc.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-ultrarisc.c b/drivers/pci/controller/dwc/pcie-ultrarisc.c
index 51e89180490d4..0d4989a0f3afc 100644
--- a/drivers/pci/controller/dwc/pcie-ultrarisc.c
+++ b/drivers/pci/controller/dwc/pcie-ultrarisc.c
@@ -154,6 +154,12 @@ static int ultrarisc_pcie_probe(struct platform_device *pdev)
 
 	pp = &pci->pp;
 
+	/*
+	 * DP1000 routes MEM transactions without outbound iATU windows, which
+	 * avoids the 16-window/4GB-per-window aperture limit.
+	 */
+	pp->bypass_ob_mem_iatu = true;
+
 	platform_set_drvdata(pdev, ultra);
 
 	ret = ultrarisc_pcie_init_clks(ultra);
-- 
2.34.1

Re: [PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[sashiko-bot]

> Some DWC-based controllers do not require outbound iATU windows for MEM
> transactions. For those platforms, programming outbound iATU windows for
> host bridge MEM ranges is unnecessary and may constrain the available MEM
> aperture.
> 
> Add dw_pcie_rp::bypass_ob_mem_iatu so drivers can skip generic MEM iATU
> setup while using the common DWC host init path.
> 
> Existing drivers keep the current behavior because the flag defaults to
> false. The I/O iATU setup does not use the MEM resource iterator, and the
> iterator may be uninitialized when MEM setup is skipped. Avoid using it in
> the I/O iATU error path.
> 
> Signed-off-by: Xincheng Zhang <zhangxincheng@ultrarisc.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260827055448.1818858-1-zhangxincheng@ultrarisc.com?part=1

Re: [PATCH 2/2] PCI: ultrarisc: Skip MEM outbound iATU setup on DP1000

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing driver `.remove` callback causes a Use-After-Free of PCI host bridge structures when the module is unloaded.
--

commit e72827a32cf8b1eb87517edbf6b7e64e76f36134
Author: Xincheng Zhang <zhangxincheng@ultrarisc.com>

PCI: ultrarisc: Skip MEM outbound iATU setup on DP1000

The DP1000 has 16 outbound iATU windows, one of which is used for the
32-bit MEM aperture. To avoid limiting the 64-bit MEM aperture to the
remaining 13 windows, this commit skips generic DWC MEM outbound iATU
setup, allowing transactions to bypass outbound iATU windows.

> diff --git a/drivers/pci/controller/dwc/pcie-ultrarisc.c b/drivers/pci/controller/dwc/pcie-ultrarisc.c
> index 51e89180490d4..0d4989a0f3afc 100644
> --- a/drivers/pci/controller/dwc/pcie-ultrarisc.c
> +++ b/drivers/pci/controller/dwc/pcie-ultrarisc.c
> @@ -154,6 +154,12 @@ static int ultrarisc_pcie_probe(struct platform_device *pdev)
>  
>  	pp = &pci->pp;
>  
> +	/*
> +	 * DP1000 routes MEM transactions without outbound iATU windows, which
> +	 * avoids the 16-window/4GB-per-window aperture limit.
> +	 */
> +	pp->bypass_ob_mem_iatu = true;
> +
>  	platform_set_drvdata(pdev, ultra);
>  
>  	ret = ultrarisc_pcie_init_clks(ultra);

[Severity: High]
This is a pre-existing issue, but since you are modifying the probe path,
I noticed that ultrarisc_pcie_probe() calls dw_pcie_host_init() which allocates
a PCI host bridge via devres, yet the platform driver lacks a remove callback.

Because the driver registers with module_platform_driver():

static struct platform_driver ultrarisc_pcie_driver = {
	.driver = {
		.name = "ultrarisc-pcie",
		...
	},
	.probe = ultrarisc_pcie_probe,
};
module_platform_driver(ultrarisc_pcie_driver);

Is it possible for a privileged user to unload the module and trigger a
use-after-free?

When the module is unloaded, devres will automatically free the memory for the
pci_host_bridge. Because there is no remove callback in the driver,
dw_pcie_host_deinit() is never invoked to call pci_stop_root_bus() and
pci_remove_root_bus().

Does this leave the PCI core with active references to the freed host bridge
structures after the module is removed?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260827055448.1818858-1-zhangxincheng@ultrarisc.com?part=2

Re: [PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[Manivannan Sadhasivam]

On Thu, Aug 27, 2026 at 01:54:47PM +0800, Xincheng Zhang wrote:
> Some DWC-based controllers do not require outbound iATU windows for MEM
> transactions. For those platforms, programming outbound iATU windows for
> host bridge MEM ranges is unnecessary and may constrain the available MEM
> aperture.
> 

Can you elaborate why the outbound iATU mapping is not required? Are they using
ECAM or some other custom mapping scheme?

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[Xincheng Zhang]

Hi Mani,

On Wed, Sep 02, 2026 at 04:18:58PM +0200, Manivannan Sadhasivam wrote:
> Can you elaborate why the outbound iATU mapping is not required? Are they using
> ECAM or some other custom mapping scheme?

DP1000 is not using ECAM for this controller. Config space accesses still
use the DWC own config access path, so CFG accesses continue to use the
normal DWC outbound iATU programming for CFG0/CFG1.

The bypass is only for host bridge MEM windows. The DP1000 integration has
SoC wrapper address decode for the PCIe MEM apertures described by the
"ranges" property. CPU physical accesses to those MEM apertures are routed
directly as PCIe MEM transactions with the corresponding PCIe bus address,
without consuming outbound iATU windows.

I/O space is also unchanged and still uses an outbound iATU window. So the
new flag only skips generic MEM-window programming; it does not change CFG
or I/O iATU setup.

This direct MEM path has been verified on DP1000 EVB and Titan with large
BAR MMIO read/write tests.

I will make this clearer in v2 by updating the commit messages/comments to
say that the bypass is MEM-only, not ECAM, and that CFG/I/O iATU setup is
unchanged.

Thanks,
Xincheng

Re: [PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[Niklas Cassel]

On Thu, Aug 27, 2026 at 01:54:47PM +0800, Xincheng Zhang wrote:
> Some DWC-based controllers do not require outbound iATU windows for MEM
> transactions. For those platforms, programming outbound iATU windows for
> host bridge MEM ranges is unnecessary and may constrain the available MEM
> aperture.

Please give a reference to a specific section in a specific version of the
DWC databook.

My guess is that you are relying on the behavior defined in:

DWC EP Databook version 5.96a, section "3.10.5.5 No Address Match Result":
""""
Overview: When there is no address match then the address is untranslated but the TLP header information
(for fields that are programmable) comes from the relevant fields on the application transmit interface
XALI*1.
""""


> 
> Add dw_pcie_rp::bypass_ob_mem_iatu so drivers can skip generic MEM iATU
> setup while using the common DWC host init path.
> 
> Existing drivers keep the current behavior because the flag defaults to
> false. The I/O iATU setup does not use the MEM resource iterator, and the
> iterator may be uninitialized when MEM setup is skipped. Avoid using it in
> the I/O iATU error path.
> 
> Signed-off-by: Xincheng Zhang <zhangxincheng@ultrarisc.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware-host.c | 6 +++---
>  drivers/pci/controller/dwc/pcie-designware.h      | 1 +
>  2 files changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index f5a38e6fd8d79..3c5f5ff080818 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -911,7 +911,8 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
>  	resource_list_for_each_entry(entry, &pp->bridge->windows) {
>  		resource_size_t res_size;
>  
> -		if (resource_type(entry->res) != IORESOURCE_MEM)
> +		if (pp->bypass_ob_mem_iatu ||
> +		    resource_type(entry->res) != IORESOURCE_MEM)
>  			continue;

To consider allowing this, at the bare minimum you would have to:

if pp->bypass_ob_mem_iatu is set:

1) Verify that the glue driver has no .cpu_addr_fixup callback defined.

2) For each resource entry that you skip, verify that the atu.parent_bus_addr
is equal to the atu.pci_addr.

Just because pp->bypass_ob_mem_iatu is set in the driver, does not mean that
all device trees for SoCs that is using that driver have defined the PCI range
and the parent CPU range in a way where this would work.


A concrete example:
rk3588 pcie3x4 ranges in v7.0:
https://github.com/torvalds/linux/blob/v7.0/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi#L378

<0x03000000 0x9 0x00000000 0x9 0x00000000 0x0 0x40000000>;

PCI address range: 0x900000000-0x93fffffff
parent CPU range:  0x900000000-0x93fffffff


rk3588 pcie3x4 ranges in v6.19:
https://github.com/torvalds/linux/blob/v6.19/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi#L378

<0x03000000 0x0 0x40000000 0x9 0x00000000 0x0 0x40000000>;

PCI address range: 0x040000000-0x07fffffff
parent CPU range:  0x900000000-0x93fffffff


E.g. for rk3588, a kernel which has pp->bypass_ob_mem_iatu set in the driver
would work with a device tree from v7.0 and newer, but would send TLPs with
an invalid PCI address when used with a device tree from v6.19 or older.

Since the device tree can be updated independently of the kernel, having this
safety check would be very important IMO.


Kind regards,
Niklas

Re: [PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[Niklas Cassel]

On Thu, Sep 03, 2026 at 05:17:13PM +0200, Niklas Cassel wrote:
> 
> To consider allowing this, at the bare minimum you would have to:
> 
> if pp->bypass_ob_mem_iatu is set:
> 
> 1) Verify that the glue driver has no .cpu_addr_fixup callback defined.

Looking at dw_pcie_parent_bus_offset(), I can see that it already computes
pci->parent_bus_offset correctly, regardless of pci->ops->cpu_addr_fixup and
pci->use_parent_dt_ranges, so atu.parent_bus_addr will be set correctly.

Thus, you can ignore 1).

But 2) would still be very much needed.


Kind regards,
Niklas

Re: [PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[Xincheng Zhang]

Hi Niklas,

On Thu, Sep 03, 2026 at 05:30:18PM +0200, Niklas Cassel wrote:
> Looking at dw_pcie_parent_bus_offset(), I can see that it already computes
> pci->parent_bus_offset correctly, regardless of pci->ops->cpu_addr_fixup and
> pci->use_parent_dt_ranges, so atu.parent_bus_addr will be set correctly.
>
> Thus, you can ignore 1).
>
> But 2) would still be very much needed.

Thanks for checking this.

Yes, I agree that the skipped MEM windows need an identity mapping check.
In v2 I will move the bypass decision until after calculating the same
outbound addresses used by the normal iATU programming path, and fail the
host init if a skipped MEM range is not identity-mapped:

	atu.parent_bus_addr = entry->res->start - pci->parent_bus_offset;
	atu.pci_addr = entry->res->start - entry->offset;

	if (pp->bypass_ob_mem_iatu &&
	    atu.parent_bus_addr != atu.pci_addr)
		return -EINVAL;

This should prevent the driver from silently relying on no-match pass-through
when the DT "ranges" describe different parent-bus and PCI addresses.

For the databook reference, the relevant text I found is in the DWC PCIe RP
Controller Databook v6.00a, June 2022. Section 3.12.2 says the default
outbound no-match behavior is pass-through, and Section 3.12.5.6 "No Address
Match Result" says the address is not translated and programmable TLP header
information comes from the application transmit interface. I will cite those
RP Databook sections in the v2 commit message instead of referring to this
generically.

For DP1000, this remains MEM-only. CFG and I/O still use the normal DWC
outbound iATU programming path.

Thanks,
Xincheng

Re: [PATCH 1/2] PCI: dwc: Allow drivers to skip MEM outbound iATU setup

[Niklas Cassel]

On Fri, Sep 04, 2026 at 09:51:34AM +0800, Xincheng Zhang wrote:
> On Thu, Sep 03, 2026 at 05:30:18PM +0200, Niklas Cassel wrote:
> 
> Yes, I agree that the skipped MEM windows need an identity mapping check.
> In v2 I will move the bypass decision until after calculating the same
> outbound addresses used by the normal iATU programming path, and fail the
> host init if a skipped MEM range is not identity-mapped:
> 
> 	atu.parent_bus_addr = entry->res->start - pci->parent_bus_offset;
> 	atu.pci_addr = entry->res->start - entry->offset;
> 
> 	if (pp->bypass_ob_mem_iatu &&
> 	    atu.parent_bus_addr != atu.pci_addr)
> 		return -EINVAL;

Looks good.
I think it also makes sense to add a short code comment above the
if-statement.


> 
> This should prevent the driver from silently relying on no-match pass-through
> when the DT "ranges" describe different parent-bus and PCI addresses.
> 
> For the databook reference, the relevant text I found is in the DWC PCIe RP
> Controller Databook v6.00a, June 2022. Section 3.12.2 says the default
> outbound no-match behavior is pass-through, and Section 3.12.5.6 "No Address
> Match Result" says the address is not translated and programmable TLP header
> information comes from the application transmit interface. I will cite those
> RP Databook sections in the v2 commit message instead of referring to this
> generically.

Sounds good.


Kind regards,
Niklas