[PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Geramy Loveless]

Firmware typically sizes prefetchable bridge windows for the boot-time
BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
no room for a driver to grow a Resizable BAR afterwards: every window
from the leaf up to the root is sized for the small BAR, so
pci_resize_resource() fails with -ENOSPC. 

Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
VRAM BAR) pushes the required window size past the large BAR's alignment. 
Because bridge windows round up to a power of two, this forces a massive 
alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
wasting ~32 GiB per GPU).

This patch solves both issues to enable ReBAR on cascaded switch fabrics:

1. Reserve Headroom: 
Reserve prefetchable window headroom for the maximum size of each 
downstream Resizable BAR during the bridge sizing pass. The device BAR 
and hardware ReBAR are left at their boot size to prevent tearing down 
firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.

2. Pack Oversized Windows (BAR Demotion): 
The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR 
to be assigned from the non-prefetchable window. By clearing the PREFETCH 
flag on small control BARs during enumeration, they are placed below 4 GiB. 
The prefetchable window is then cleanly sized to the large BAR alone.

Why this is safe for 32-bit (< 4G) MMIO space:
To prevent exhausting legacy < 4G space, demotion is strictly bounded. It 
only triggers beside a massive prefetchable BAR (>= 64 MiB), and the demoted 
BAR is strictly capped at 16 MiB (PCI_BAR_PACK_CAP). Even on an 8x GPU 
system, this consumes at most 128 MiB of < 4G space. 

Fallback / Alignment logic:
If the 32-bit space is exhausted, small BARs remain in the prefetchable 
window. The alignment logic gracefully handles this by rounding the bridge 
window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
properly aligned slots.

Reserving is safe by default: __assign_resources_sorted() satisfies all
required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
to restore previous behaviours.

Testing Context:
- ASUS K14PG-D24
- 2x EPYC 9354
- 8x Radeon AI PRO R9700 (Navi 48, 1002:7551)
- Broadcom PEX890xx switches
- Kernel base: 45c13f3f9e3b (tested with SEV-SNP passthrough=1)

Signed-off-by: Geramy Loveless <gloveless@jqluv.com>
---
 .../admin-guide/kernel-parameters.txt         |   8 ++
 drivers/pci/pci.c                             |  10 ++
 drivers/pci/pci.h                             |   1 +
 drivers/pci/setup-bus.c                       | 118 ++++++++++++++++++
 4 files changed, 137 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt
b/Documentation/admin-guide/kernel-parameters.txt
index 37006fc3eac3..82d2b340fb98 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5246,6 +5246,14 @@ Kernel parameters
 		hpbussize=nn	The minimum amount of additional bus numbers
 				reserved for buses below a hotplug bridge.
 				Default is 1.
+		no_rebar_reserve	Do not reserve prefetchable bridge-window
+				space for the maximum size of downstream Resizable
+				BARs. By default such space is reserved so a driver
+				can grow a BAR later (e.g. GPU VRAM BAR) even behind
+				fixed PCIe switch fabrics that firmware sized for the
+				boot-time BAR; the device BAR is left at its boot size
+				for the driver to resize and only bridge windows are
+				enlarged. Use this to restore the old behaviour.
 		realloc=	Enable/disable reallocating PCI bridge resources
 				if allocations done by BIOS are too small to
 				accommodate resources required by all child
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index b2879a6be5f8..c9901b15432a 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -120,6 +120,14 @@ unsigned long pci_hotplug_mmio_pref_size =
DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
 #define DEFAULT_HOTPLUG_BUS_SIZE	1
 unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;

+/*
+ * Prefetchable bridge-window headroom is reserved for the maximum size of
+ * downstream Resizable BARs by default, so a driver can grow one later
+ * (e.g. amdgpu VRAM BAR) even behind fixed PCIe switch fabrics.
+ * pci=no_rebar_reserve disables this.
+ */
+bool pci_rebar_no_reserve;
+

 /* PCIe MPS/MRRS strategy; can be overridden by kernel command-line
param */
 enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
@@ -6840,6 +6848,8 @@ static int __init pci_setup(char *str)
 					simple_strtoul(str + 10, &str, 0);
 				if (pci_hotplug_bus_size > 0xff)
 					pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
+			} else if (!strncmp(str, "no_rebar_reserve", 16)) {
+				pci_rebar_no_reserve = true;
 			} else if (!strncmp(str, "pcie_bus_tune_off", 17)) {
 				pcie_bus_config = PCIE_BUS_TUNE_OFF;
 			} else if (!strncmp(str, "pcie_bus_safe", 13)) {
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index ba3c3fddddc2..6eddcb492425 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -409,6 +409,7 @@ extern unsigned long pci_hotplug_io_size;
 extern unsigned long pci_hotplug_mmio_size;
 extern unsigned long pci_hotplug_mmio_pref_size;
 extern unsigned long pci_hotplug_bus_size;
+extern bool pci_rebar_no_reserve;

 static inline bool pci_is_cardbus_bridge(struct pci_dev *dev)
 {
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index e8c94aa1d3c1..aa02b8454269 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -19,6 +19,7 @@
 #include <linux/bug.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
+#include <linux/log2.h>
 #include <linux/minmax.h>
 #include <linux/module.h>
 #include <linux/pci.h>
@@ -1258,6 +1259,53 @@ static bool pbus_size_mem_optional(struct pci_dev
*dev, int resno,
 	return true;
 }

+/*
+ * pci_rebar_reserve_size - prefetchable window headroom to reserve for
a BAR
+ *
+ * Reserve enough prefetchable window space to later grow @resno of @dev to
+ * its maximum Resizable BAR size. This is done for every ReBAR-capable
device
+ * by default (disable with pci=no_rebar_reserve). Only the optional
(add_size)
+ * part of the enclosing window is inflated here; the device BAR
resource and
+ * the hardware ReBAR are left at their boot size, so the driver still
performs
+ * the actual resize into the reserved window. Reserving the window up
front
+ * avoids -ENOSPC in pci_resize_resource() on fixed switch fabrics, where a
+ * window shared with a sibling's still-assigned BAR cannot be released and
+ * regrown at resize time.
+ *
+ * The reservation is strictly optional: __assign_resources_sorted()
satisfies
+ * all required resources first and only then places add_size on leftover
+ * space, so reserving by default cannot make a required BAR or window
fail.
+ *
+ * Return the extra bytes to reserve and raise *add_align to the BAR's
+ * alignment so the reserved space can actually hold the grown BAR.
+ */
+static resource_size_t pci_rebar_reserve_size(struct pci_dev *dev, int
resno,
+					      resource_size_t *add_align)
+{
+	struct resource *res = pci_resource_n(dev, resno);
+	resource_size_t max_size, cur_size;
+	int max;
+
+	if (pci_rebar_no_reserve || resno >= PCI_STD_NUM_BARS)
+		return 0;
+
+	if ((res->flags & (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH)) !=
+	    (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH))
+		return 0;
+
+	max = pci_rebar_get_max_size(dev, resno);
+	if (max < 0)
+		return 0;
+
+	max_size = pci_rebar_size_to_bytes(max);
+	cur_size = resource_size(res);
+	if (max_size <= cur_size)
+		return 0;
+
+	*add_align = max(*add_align, max_size);
+	return max_size - cur_size;
+}
+
 /**
  * pbus_size_mem() - Size the memory window of a given bus
  *
@@ -1285,6 +1333,7 @@ static void pbus_size_mem(struct pci_bus *bus,
struct resource *b_res,
 	int order, max_order;
 	resource_size_t children_add_size = 0;
 	resource_size_t add_align = 0;
+	resource_size_t rebar_align = 0;

 	if (!b_res)
 		return;
@@ -1343,15 +1392,29 @@ static void pbus_size_mem(struct pci_bus *bus,
struct resource *b_res,
 				aligns[order] += align;
 			if (order > max_order)
 				max_order = order;
+
+			size += pci_rebar_reserve_size(dev, i, &rebar_align);
 		}
 	}

 	win_align = pci_min_window_alignment(bus, b_res->flags);
 	min_align = calculate_head_align(aligns, max_order);
 	min_align = max(min_align, win_align);
+	min_align = max(min_align, rebar_align);
 	size0 = calculate_memsize(size, realloc_head ? 0 : add_size,
 				  0, win_align);

+	/*
+	 * A window that reserves ReBAR headroom is larger than its own
+	 * alignment (e.g. 32 GiB BAR + doorbell => 33 GiB > 32 GiB). The
+	 * parent only reserves alignment padding for a child window when
+	 * its size <= its alignment, so round such a window's alignment up
+	 * to a power of two >= its size; sibling 32 GiB BARs behind one
+	 * switch then each land in a properly aligned slot.
+	 */
+	if (rebar_align && size0)
+		min_align = max(min_align, roundup_pow_of_two(size0));
+
 	if (size0) {
 		resource_set_range(b_res, min_align, size0);
 		b_res->flags &= ~IORESOURCE_DISABLED;
@@ -2179,6 +2242,52 @@ static void pci_prepare_next_assign_round(struct
list_head *fail_head,
  * Second and later try will clear small leaf bridge res.
  * Will stop till to the max depth if can not find good one.
  */
+/*
+ * Release the 64-bit prefetchable BARs of resizable-BAR display
devices so the
+ * windows enclosing them empty out and can be re-sized. The BAR *size*
is left
+ * untouched (only released and re-placed), so the hardware ReBAR is never
+ * programmed here - the driver still owns the actual resize.
+ */
+static int pci_release_rebar_bars_cb(struct pci_dev *dev, void *data)
+{
+	struct resource *r;
+	unsigned int i;
+
+	if ((dev->class >> 16) != PCI_BASE_CLASS_DISPLAY)
+		return 0;
+	if (pci_rebar_get_max_size(dev, 0) <= pci_rebar_get_current_size(dev, 0))
+		return 0;
+	pci_dev_for_each_resource(dev, r, i) {
+		if (i >= PCI_BRIDGE_RESOURCES)
+			break;
+		if (r->parent && (r->flags & IORESOURCE_MEM_64) &&
+		    (r->flags & IORESOURCE_PREFETCH))
+			pci_release_resource(dev, i);
+	}
+	return 0;
+}
+
+/*
+ * Release the now-empty prefetchable bridge windows bottom-up so the
sizing
+ * pass re-sizes them (with pci_rebar_pref_reserve() reservation) to
fit the
+ * BARs a driver will later grow.
+ */
+static void pci_release_rebar_windows(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+	struct resource *w;
+
+	list_for_each_entry(dev, &bus->devices, bus_list)
+		if (dev->subordinate)
+			pci_release_rebar_windows(dev->subordinate);
+
+	if (!bus->self)
+		return;
+	w = &bus->self->resource[PCI_BRIDGE_PREF_MEM_WINDOW];
+	if (w->parent && (w->flags & IORESOURCE_MEM_64) && !w->child)
+		pci_release_resource(bus->self, PCI_BRIDGE_PREF_MEM_WINDOW);
+}
+
 void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 {
 	LIST_HEAD(realloc_head);
@@ -2190,6 +2299,15 @@ void
pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 	int pci_try_num = 1;
 	enum enable_type enable_local;

+	/*
+	 * Release resizable device BARs and their prefetchable windows so the
+	 * sizing pass re-sizes those windows large enough
(pci_rebar_pref_reserve)
+	 * for the BARs a driver will later grow. Only released and re-placed
- the
+	 * BAR size is left untouched, so the hardware ReBAR is never programmed.
+	 */
+	pci_walk_bus(bus, pci_release_rebar_bars_cb, NULL);
+	pci_release_rebar_windows(bus);
+
 	/* Don't realloc if asked to do so */
 	enable_local = pci_realloc_detect(bus, pci_realloc_enable);
 	if (pci_realloc_enabled(enable_local)) {

base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229
-- 
2.53.0


From 71cb1c4131d00da973142fce2511636f033c43c1 Mon Sep 17 00:00:00 2001
From: Geramy Loveless <gloveless@jqluv.com>
Date: Fri, 28 Aug 2026 21:11:55 +0000
Subject: [PATCH] PCI: pack oversized prefetchable windows by demoting small
 BARs at enumeration

Clear PREFETCH on small control BARs beside a large prefetchable BAR at
enumeration so they are placed below 4 GiB and the prefetchable window
packs to the large BAR alone. Bounded to small BARs (<=16 MiB) beside a
big one (>=64 MiB); pci=no_bar_demote disables it.
---
 .../admin-guide/kernel-parameters.txt         |  1 +
 drivers/pci/pci.c                             |  5 ++
 drivers/pci/pci.h                             |  1 +
 drivers/pci/quirks.c                          | 54 +++++++++++++++++++
 4 files changed, 61 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt
b/Documentation/admin-guide/kernel-parameters.txt
index 82d2b340fb98..7360407dd4b8 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5246,6 +5246,7 @@ Kernel parameters
 		hpbussize=nn	The minimum amount of additional bus numbers
 				reserved for buses below a hotplug bridge.
 				Default is 1.
+		no_bar_demote	Do not demote small prefetchable BARs below 4 GiB to
pack oversized prefetchable bridge windows.
 		no_rebar_reserve	Do not reserve prefetchable bridge-window
 				space for the maximum size of downstream Resizable
 				BARs. By default such space is reserved so a driver
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index c9901b15432a..0bb4ad570171 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -128,6 +128,9 @@ unsigned long pci_hotplug_bus_size =
DEFAULT_HOTPLUG_BUS_SIZE;
  */
 bool pci_rebar_no_reserve;

+/* pci=no_bar_demote disables demoting small prefetchable BARs below 4
GiB. */
+bool pci_bar_demote_disabled;
+

 /* PCIe MPS/MRRS strategy; can be overridden by kernel command-line
param */
 enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
@@ -6850,6 +6853,8 @@ static int __init pci_setup(char *str)
 					pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
 			} else if (!strncmp(str, "no_rebar_reserve", 16)) {
 				pci_rebar_no_reserve = true;
+			} else if (!strncmp(str, "no_bar_demote", 13)) {
+				pci_bar_demote_disabled = true;
 			} else if (!strncmp(str, "pcie_bus_tune_off", 17)) {
 				pcie_bus_config = PCIE_BUS_TUNE_OFF;
 			} else if (!strncmp(str, "pcie_bus_safe", 13)) {
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 6eddcb492425..0f21972b674a 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -410,6 +410,7 @@ extern unsigned long pci_hotplug_mmio_size;
 extern unsigned long pci_hotplug_mmio_pref_size;
 extern unsigned long pci_hotplug_bus_size;
 extern bool pci_rebar_no_reserve;
+extern bool pci_bar_demote_disabled;

 static inline bool pci_is_cardbus_bridge(struct pci_dev *dev)
 {
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index de9bbccda21f..57a5926a273c 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -6423,3 +6423,57 @@ static void pci_mask_replay_timer_timeout(struct
pci_dev *pdev)
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750,
pci_mask_replay_timer_timeout);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755,
pci_mask_replay_timer_timeout);
 #endif
+
+/*
+ * Pack oversized prefetchable windows.
+ *
+ * A small prefetchable BAR (e.g. a 2 MiB doorbell) sharing a bridge's
single
+ * prefetchable window with a much larger one (e.g. a 32 GiB
framebuffer) pushes
+ * the window past the large BAR's alignment, so siblings behind a
switch waste
+ * an alignment each.  A prefetchable BAR may legally be assigned from the
+ * non-prefetchable window (PCI-to-PCI Bridge spec r1.2 sec 3.2.5);
clear PREFETCH
+ * on such small BARs so they are placed below 4 GiB and the
prefetchable window
+ * is sized to the large BAR alone.  Run at enumeration -- before
resources are
+ * sized -- so the reservation and assignment see the final layout.
Only small
+ * "control" BARs beside a large one are touched, bounding the extra
below-4G use.
+ * pci=no_bar_demote disables this.
+ */
+#define PCI_BAR_PACK_MIN	SZ_64M	/* device must have a pref BAR at least
this big */
+#define PCI_BAR_PACK_CAP	SZ_16M	/* only demote small pref BARs up to
this size */
+
+static void quirk_pci_pack_prefetch(struct pci_dev *dev)
+{
+	resource_size_t max_align = 0;
+	struct resource *r;
+	int i;
+
+	if (pci_bar_demote_disabled)
+		return;
+
+	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
+		r = pci_resource_n(dev, i);
+		if ((r->flags & (IORESOURCE_MEM | IORESOURCE_PREFETCH)) ==
+		    (IORESOURCE_MEM | IORESOURCE_PREFETCH))
+			max_align = max(max_align, pci_resource_alignment(dev, r));
+	}
+	if (max_align < PCI_BAR_PACK_MIN)
+		return;
+
+	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
+		r = pci_resource_n(dev, i);
+		if ((r->flags & (IORESOURCE_MEM | IORESOURCE_PREFETCH)) !=
+		    (IORESOURCE_MEM | IORESOURCE_PREFETCH))
+			continue;
+		if (r->flags & IORESOURCE_PCI_FIXED)
+			continue;
+		if (pci_resource_alignment(dev, r) >= max_align)
+			continue;		/* an alignment-setting BAR, keep it */
+		if (resource_size(r) > PCI_BAR_PACK_CAP)
+			continue;		/* not a small control BAR */
+
+		r->flags &= ~IORESOURCE_PREFETCH;
+		pci_info(dev, "%pR: assigned from the non-prefetchable window to pack
the prefetchable window\n",
+			 r);
+	}
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_ANY_ID, PCI_ANY_ID, quirk_pci_pack_prefetch);

base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229
prerequisite-patch-id: 73634e017e2770ecd47fe4fe434973e280b5d3d8
prerequisite-patch-id: c397a97535c26763ed6104deba168b0f5e456250
prerequisite-patch-id: 24c5bab9f84760d15064b6ea7d73720ce33eb5bc
prerequisite-patch-id: df6cdb5d432ed50ffffc5b31e3df32711c0b98c2
-- 
2.53.0

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Christian König]

On 8/28/26 23:37, Geramy Loveless wrote:
> Firmware typically sizes prefetchable bridge windows for the boot-time
> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
> no room for a driver to grow a Resizable BAR afterwards: every window
> from the leaf up to the root is sized for the small BAR, so
> pci_resize_resource() fails with -ENOSPC. 
> 
> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
> VRAM BAR) pushes the required window size past the large BAR's alignment. 
> Because bridge windows round up to a power of two, this forces a massive 
> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
> wasting ~32 GiB per GPU).
> 
> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
> 
> 1. Reserve Headroom: 
> Reserve prefetchable window headroom for the maximum size of each 
> downstream Resizable BAR during the bridge sizing pass. The device BAR 
> and hardware ReBAR are left at their boot size to prevent tearing down 
> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.

Yeah that was suggested before but that is clearly not something you can do in common code.

The ReBAR fields often doesn't reflect the actual needed space but rather the maximum the HW address logic can resolve.

So what you end up with is allocating multiple TiB for a window which just needs few GiB, sometimes even completely overflowing the 64bit address space made available by the root complex.
> 2. Pack Oversized Windows (BAR Demotion): 
> The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR 
> to be assigned from the non-prefetchable window. By clearing the PREFETCH 
> flag on small control BARs during enumeration, they are placed below 4 GiB. 
> The prefetchable window is then cleanly sized to the large BAR alone.

Interesting hack, shouldn't really matter for AMD GPUs but that is something more on the heuristic side.

A perfectly valid trick which is done by BIOS implementations but what the Linux PCI subsystem still hasn't learned are back to back allocations, e.g. something like this:

BAR0 32 GiB of GPU #1
BAR1 2	MiB of GPU #1

BAR1 2	MiB of GPU #2
BAR0 32 GiB of GPU #2

BAR0 32 GiB of GPU #3
...

I strongly suggest to implement that one first and see if it helps with your use case.

Regards,
Christian.

> 
> Why this is safe for 32-bit (< 4G) MMIO space:
> To prevent exhausting legacy < 4G space, demotion is strictly bounded. It 
> only triggers beside a massive prefetchable BAR (>= 64 MiB), and the demoted 
> BAR is strictly capped at 16 MiB (PCI_BAR_PACK_CAP). Even on an 8x GPU 
> system, this consumes at most 128 MiB of < 4G space. 
> 
> Fallback / Alignment logic:
> If the 32-bit space is exhausted, small BARs remain in the prefetchable 
> window. The alignment logic gracefully handles this by rounding the bridge 
> window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
> BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
> properly aligned slots.
> 
> Reserving is safe by default: __assign_resources_sorted() satisfies all
> required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
> to restore previous behaviours.
> 
> Testing Context:
> - ASUS K14PG-D24
> - 2x EPYC 9354
> - 8x Radeon AI PRO R9700 (Navi 48, 1002:7551)
> - Broadcom PEX890xx switches
> - Kernel base: 45c13f3f9e3b (tested with SEV-SNP passthrough=1)
> 
> Signed-off-by: Geramy Loveless <gloveless@jqluv.com>
> ---
>  .../admin-guide/kernel-parameters.txt         |   8 ++
>  drivers/pci/pci.c                             |  10 ++
>  drivers/pci/pci.h                             |   1 +
>  drivers/pci/setup-bus.c                       | 118 ++++++++++++++++++
>  4 files changed, 137 insertions(+)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt
> b/Documentation/admin-guide/kernel-parameters.txt
> index 37006fc3eac3..82d2b340fb98 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -5246,6 +5246,14 @@ Kernel parameters
>  		hpbussize=nn	The minimum amount of additional bus numbers
>  				reserved for buses below a hotplug bridge.
>  				Default is 1.
> +		no_rebar_reserve	Do not reserve prefetchable bridge-window
> +				space for the maximum size of downstream Resizable
> +				BARs. By default such space is reserved so a driver
> +				can grow a BAR later (e.g. GPU VRAM BAR) even behind
> +				fixed PCIe switch fabrics that firmware sized for the
> +				boot-time BAR; the device BAR is left at its boot size
> +				for the driver to resize and only bridge windows are
> +				enlarged. Use this to restore the old behaviour.
>  		realloc=	Enable/disable reallocating PCI bridge resources
>  				if allocations done by BIOS are too small to
>  				accommodate resources required by all child
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index b2879a6be5f8..c9901b15432a 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -120,6 +120,14 @@ unsigned long pci_hotplug_mmio_pref_size =
> DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
>  #define DEFAULT_HOTPLUG_BUS_SIZE	1
>  unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
> 
> +/*
> + * Prefetchable bridge-window headroom is reserved for the maximum size of
> + * downstream Resizable BARs by default, so a driver can grow one later
> + * (e.g. amdgpu VRAM BAR) even behind fixed PCIe switch fabrics.
> + * pci=no_rebar_reserve disables this.
> + */
> +bool pci_rebar_no_reserve;
> +
> 
>  /* PCIe MPS/MRRS strategy; can be overridden by kernel command-line
> param */
>  enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
> @@ -6840,6 +6848,8 @@ static int __init pci_setup(char *str)
>  					simple_strtoul(str + 10, &str, 0);
>  				if (pci_hotplug_bus_size > 0xff)
>  					pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
> +			} else if (!strncmp(str, "no_rebar_reserve", 16)) {
> +				pci_rebar_no_reserve = true;
>  			} else if (!strncmp(str, "pcie_bus_tune_off", 17)) {
>  				pcie_bus_config = PCIE_BUS_TUNE_OFF;
>  			} else if (!strncmp(str, "pcie_bus_safe", 13)) {
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index ba3c3fddddc2..6eddcb492425 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -409,6 +409,7 @@ extern unsigned long pci_hotplug_io_size;
>  extern unsigned long pci_hotplug_mmio_size;
>  extern unsigned long pci_hotplug_mmio_pref_size;
>  extern unsigned long pci_hotplug_bus_size;
> +extern bool pci_rebar_no_reserve;
> 
>  static inline bool pci_is_cardbus_bridge(struct pci_dev *dev)
>  {
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index e8c94aa1d3c1..aa02b8454269 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -19,6 +19,7 @@
>  #include <linux/bug.h>
>  #include <linux/init.h>
>  #include <linux/kernel.h>
> +#include <linux/log2.h>
>  #include <linux/minmax.h>
>  #include <linux/module.h>
>  #include <linux/pci.h>
> @@ -1258,6 +1259,53 @@ static bool pbus_size_mem_optional(struct pci_dev
> *dev, int resno,
>  	return true;
>  }
> 
> +/*
> + * pci_rebar_reserve_size - prefetchable window headroom to reserve for
> a BAR
> + *
> + * Reserve enough prefetchable window space to later grow @resno of @dev to
> + * its maximum Resizable BAR size. This is done for every ReBAR-capable
> device
> + * by default (disable with pci=no_rebar_reserve). Only the optional
> (add_size)
> + * part of the enclosing window is inflated here; the device BAR
> resource and
> + * the hardware ReBAR are left at their boot size, so the driver still
> performs
> + * the actual resize into the reserved window. Reserving the window up
> front
> + * avoids -ENOSPC in pci_resize_resource() on fixed switch fabrics, where a
> + * window shared with a sibling's still-assigned BAR cannot be released and
> + * regrown at resize time.
> + *
> + * The reservation is strictly optional: __assign_resources_sorted()
> satisfies
> + * all required resources first and only then places add_size on leftover
> + * space, so reserving by default cannot make a required BAR or window
> fail.
> + *
> + * Return the extra bytes to reserve and raise *add_align to the BAR's
> + * alignment so the reserved space can actually hold the grown BAR.
> + */
> +static resource_size_t pci_rebar_reserve_size(struct pci_dev *dev, int
> resno,
> +					      resource_size_t *add_align)
> +{
> +	struct resource *res = pci_resource_n(dev, resno);
> +	resource_size_t max_size, cur_size;
> +	int max;
> +
> +	if (pci_rebar_no_reserve || resno >= PCI_STD_NUM_BARS)
> +		return 0;
> +
> +	if ((res->flags & (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH)) !=
> +	    (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH))
> +		return 0;
> +
> +	max = pci_rebar_get_max_size(dev, resno);
> +	if (max < 0)
> +		return 0;
> +
> +	max_size = pci_rebar_size_to_bytes(max);
> +	cur_size = resource_size(res);
> +	if (max_size <= cur_size)
> +		return 0;
> +
> +	*add_align = max(*add_align, max_size);
> +	return max_size - cur_size;
> +}
> +
>  /**
>   * pbus_size_mem() - Size the memory window of a given bus
>   *
> @@ -1285,6 +1333,7 @@ static void pbus_size_mem(struct pci_bus *bus,
> struct resource *b_res,
>  	int order, max_order;
>  	resource_size_t children_add_size = 0;
>  	resource_size_t add_align = 0;
> +	resource_size_t rebar_align = 0;
> 
>  	if (!b_res)
>  		return;
> @@ -1343,15 +1392,29 @@ static void pbus_size_mem(struct pci_bus *bus,
> struct resource *b_res,
>  				aligns[order] += align;
>  			if (order > max_order)
>  				max_order = order;
> +
> +			size += pci_rebar_reserve_size(dev, i, &rebar_align);
>  		}
>  	}
> 
>  	win_align = pci_min_window_alignment(bus, b_res->flags);
>  	min_align = calculate_head_align(aligns, max_order);
>  	min_align = max(min_align, win_align);
> +	min_align = max(min_align, rebar_align);
>  	size0 = calculate_memsize(size, realloc_head ? 0 : add_size,
>  				  0, win_align);
> 
> +	/*
> +	 * A window that reserves ReBAR headroom is larger than its own
> +	 * alignment (e.g. 32 GiB BAR + doorbell => 33 GiB > 32 GiB). The
> +	 * parent only reserves alignment padding for a child window when
> +	 * its size <= its alignment, so round such a window's alignment up
> +	 * to a power of two >= its size; sibling 32 GiB BARs behind one
> +	 * switch then each land in a properly aligned slot.
> +	 */
> +	if (rebar_align && size0)
> +		min_align = max(min_align, roundup_pow_of_two(size0));
> +
>  	if (size0) {
>  		resource_set_range(b_res, min_align, size0);
>  		b_res->flags &= ~IORESOURCE_DISABLED;
> @@ -2179,6 +2242,52 @@ static void pci_prepare_next_assign_round(struct
> list_head *fail_head,
>   * Second and later try will clear small leaf bridge res.
>   * Will stop till to the max depth if can not find good one.
>   */
> +/*
> + * Release the 64-bit prefetchable BARs of resizable-BAR display
> devices so the
> + * windows enclosing them empty out and can be re-sized. The BAR *size*
> is left
> + * untouched (only released and re-placed), so the hardware ReBAR is never
> + * programmed here - the driver still owns the actual resize.
> + */
> +static int pci_release_rebar_bars_cb(struct pci_dev *dev, void *data)
> +{
> +	struct resource *r;
> +	unsigned int i;
> +
> +	if ((dev->class >> 16) != PCI_BASE_CLASS_DISPLAY)
> +		return 0;
> +	if (pci_rebar_get_max_size(dev, 0) <= pci_rebar_get_current_size(dev, 0))
> +		return 0;
> +	pci_dev_for_each_resource(dev, r, i) {
> +		if (i >= PCI_BRIDGE_RESOURCES)
> +			break;
> +		if (r->parent && (r->flags & IORESOURCE_MEM_64) &&
> +		    (r->flags & IORESOURCE_PREFETCH))
> +			pci_release_resource(dev, i);
> +	}
> +	return 0;
> +}
> +
> +/*
> + * Release the now-empty prefetchable bridge windows bottom-up so the
> sizing
> + * pass re-sizes them (with pci_rebar_pref_reserve() reservation) to
> fit the
> + * BARs a driver will later grow.
> + */
> +static void pci_release_rebar_windows(struct pci_bus *bus)
> +{
> +	struct pci_dev *dev;
> +	struct resource *w;
> +
> +	list_for_each_entry(dev, &bus->devices, bus_list)
> +		if (dev->subordinate)
> +			pci_release_rebar_windows(dev->subordinate);
> +
> +	if (!bus->self)
> +		return;
> +	w = &bus->self->resource[PCI_BRIDGE_PREF_MEM_WINDOW];
> +	if (w->parent && (w->flags & IORESOURCE_MEM_64) && !w->child)
> +		pci_release_resource(bus->self, PCI_BRIDGE_PREF_MEM_WINDOW);
> +}
> +
>  void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>  {
>  	LIST_HEAD(realloc_head);
> @@ -2190,6 +2299,15 @@ void
> pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>  	int pci_try_num = 1;
>  	enum enable_type enable_local;
> 
> +	/*
> +	 * Release resizable device BARs and their prefetchable windows so the
> +	 * sizing pass re-sizes those windows large enough
> (pci_rebar_pref_reserve)
> +	 * for the BARs a driver will later grow. Only released and re-placed
> - the
> +	 * BAR size is left untouched, so the hardware ReBAR is never programmed.
> +	 */
> +	pci_walk_bus(bus, pci_release_rebar_bars_cb, NULL);
> +	pci_release_rebar_windows(bus);
> +
>  	/* Don't realloc if asked to do so */
>  	enable_local = pci_realloc_detect(bus, pci_realloc_enable);
>  	if (pci_realloc_enabled(enable_local)) {
> 
> base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Ilpo Järvinen]

On Fri, 28 Aug 2026, Geramy Loveless wrote:

> Firmware typically sizes prefetchable bridge windows for the boot-time
> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
> no room for a driver to grow a Resizable BAR afterwards: every window
> from the leaf up to the root is sized for the small BAR, so
> pci_resize_resource() fails with -ENOSPC. 
> 
> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
> VRAM BAR) pushes the required window size past the large BAR's alignment. 

> Because bridge windows round up to a power of two,

Do they??? In which kernel version I must ask??

> this forces a massive 
> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
> wasting ~32 GiB per GPU).

The largest I've seen is 32GB + half of that. And that's not with the 
latest kernel.

> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
> 
> 1. Reserve Headroom: 
> Reserve prefetchable window headroom for the maximum size of each 
> downstream Resizable BAR during the bridge sizing pass. The device BAR 
> and hardware ReBAR are left at their boot size to prevent tearing down 
> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.

... reorganizing ...

> Fallback / Alignment logic:
> If the 32-bit space is exhausted, small BARs remain in the prefetchable 
> window. The alignment logic gracefully handles this by rounding the bridge 
> window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
> BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
> properly aligned slots.
> 
> Reserving is safe by default: __assign_resources_sorted() satisfies all
> required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
> to restore previous behaviours.

And how does this fallback, if the largest size doesn't fit? What are the 
failure modes?

It's correct that the sizing decision should be done in pbus_size_mem() 
but it must not result in breaking things which a naive approach like 
this will surely do.

Without proper fallbacks done, there will be too much breakage it's a 
showstopper for a naive patch like this.

Basically, what we'd want to do is try with the largest size, then reduce 
it one step at a time (because it's 2^n -> 2^(n-1) even one step saves 
quite much so one might not need to reduce that many steps). But also, in 
some IOV enabled cases the oversubscription of the space is very 
substantial and you'd still want to maximize the BAR sizes to what is 
allowed by the available space.

Fallback is hard to implement, because sizing and assignment are quite far 
from each other codewise and there are no structures to carry information 
over to the next retry phase.

> 2. Pack Oversized Windows (BAR Demotion): 
> The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR 
> to be assigned from the non-prefetchable window. By clearing the PREFETCH 
> flag on small control BARs during enumeration, they are placed below 4 GiB. 
> The prefetchable window is then cleanly sized to the large BAR alone.
>
> Why this is safe for 32-bit (< 4G) MMIO space:
> To prevent exhausting legacy < 4G space, demotion is strictly bounded. It 
> only triggers beside a massive prefetchable BAR (>= 64 MiB), and the demoted 
> BAR is strictly capped at 16 MiB (PCI_BAR_PACK_CAP). Even on an 8x GPU 
> system, this consumes at most 128 MiB of < 4G space.  

I'm not entirely sure I agree with this, at least it shouldn't be the 
default configuration.

This would also break setups where the GPU has large BARs that do not 
share the alignment.

It might be workable as very targetted instruction to put a particular BAR 
into non-prefetchable window. For that, one needs to write a parser for 
giving them on cmdline though but that might end up being generally 
useful for avoiding some hotplug cannot-predict-the-future problems 
(the current memsize controls are too coarse-grained to be useful in many 
hotplug scenarios).


A somewhat related problem currently is that more and more devices are 
taking the liberty of not setting PREFETCH because recent PCIe spec made 
the bridge window difference be 32/64-bit based. This results in very 
suboptimal placement for such BARs.


BTW, why you're not sending the patches as a proper series but merging 
them into a single email??? It makes things harder for the reviewers. :-(


Also, I note you sent multiple versions within a day. Please give some 
time for commenting the first version. You should also list the changes 
you've made in each version (the patch history).

-- 
 i.

> Testing Context:
> - ASUS K14PG-D24
> - 2x EPYC 9354
> - 8x Radeon AI PRO R9700 (Navi 48, 1002:7551)
> - Broadcom PEX890xx switches
> - Kernel base: 45c13f3f9e3b (tested with SEV-SNP passthrough=1)
> 
> Signed-off-by: Geramy Loveless <gloveless@jqluv.com>
> ---
>  .../admin-guide/kernel-parameters.txt         |   8 ++
>  drivers/pci/pci.c                             |  10 ++
>  drivers/pci/pci.h                             |   1 +
>  drivers/pci/setup-bus.c                       | 118 ++++++++++++++++++
>  4 files changed, 137 insertions(+)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt
> b/Documentation/admin-guide/kernel-parameters.txt
> index 37006fc3eac3..82d2b340fb98 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -5246,6 +5246,14 @@ Kernel parameters
>  		hpbussize=nn	The minimum amount of additional bus numbers
>  				reserved for buses below a hotplug bridge.
>  				Default is 1.
> +		no_rebar_reserve	Do not reserve prefetchable bridge-window
> +				space for the maximum size of downstream Resizable
> +				BARs. By default such space is reserved so a driver
> +				can grow a BAR later (e.g. GPU VRAM BAR) even behind
> +				fixed PCIe switch fabrics that firmware sized for the
> +				boot-time BAR; the device BAR is left at its boot size
> +				for the driver to resize and only bridge windows are
> +				enlarged. Use this to restore the old behaviour.
>  		realloc=	Enable/disable reallocating PCI bridge resources
>  				if allocations done by BIOS are too small to
>  				accommodate resources required by all child
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index b2879a6be5f8..c9901b15432a 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -120,6 +120,14 @@ unsigned long pci_hotplug_mmio_pref_size =
> DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
>  #define DEFAULT_HOTPLUG_BUS_SIZE	1
>  unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
> 
> +/*
> + * Prefetchable bridge-window headroom is reserved for the maximum size of
> + * downstream Resizable BARs by default, so a driver can grow one later
> + * (e.g. amdgpu VRAM BAR) even behind fixed PCIe switch fabrics.
> + * pci=no_rebar_reserve disables this.
> + */
> +bool pci_rebar_no_reserve;
> +
> 
>  /* PCIe MPS/MRRS strategy; can be overridden by kernel command-line
> param */
>  enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
> @@ -6840,6 +6848,8 @@ static int __init pci_setup(char *str)
>  					simple_strtoul(str + 10, &str, 0);
>  				if (pci_hotplug_bus_size > 0xff)
>  					pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
> +			} else if (!strncmp(str, "no_rebar_reserve", 16)) {
> +				pci_rebar_no_reserve = true;
>  			} else if (!strncmp(str, "pcie_bus_tune_off", 17)) {
>  				pcie_bus_config = PCIE_BUS_TUNE_OFF;
>  			} else if (!strncmp(str, "pcie_bus_safe", 13)) {
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index ba3c3fddddc2..6eddcb492425 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -409,6 +409,7 @@ extern unsigned long pci_hotplug_io_size;
>  extern unsigned long pci_hotplug_mmio_size;
>  extern unsigned long pci_hotplug_mmio_pref_size;
>  extern unsigned long pci_hotplug_bus_size;
> +extern bool pci_rebar_no_reserve;
> 
>  static inline bool pci_is_cardbus_bridge(struct pci_dev *dev)
>  {
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index e8c94aa1d3c1..aa02b8454269 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -19,6 +19,7 @@
>  #include <linux/bug.h>
>  #include <linux/init.h>
>  #include <linux/kernel.h>
> +#include <linux/log2.h>
>  #include <linux/minmax.h>
>  #include <linux/module.h>
>  #include <linux/pci.h>
> @@ -1258,6 +1259,53 @@ static bool pbus_size_mem_optional(struct pci_dev
> *dev, int resno,
>  	return true;
>  }
> 
> +/*
> + * pci_rebar_reserve_size - prefetchable window headroom to reserve for
> a BAR
> + *
> + * Reserve enough prefetchable window space to later grow @resno of @dev to
> + * its maximum Resizable BAR size. This is done for every ReBAR-capable
> device
> + * by default (disable with pci=no_rebar_reserve). Only the optional
> (add_size)
> + * part of the enclosing window is inflated here; the device BAR
> resource and
> + * the hardware ReBAR are left at their boot size, so the driver still
> performs
> + * the actual resize into the reserved window. Reserving the window up
> front
> + * avoids -ENOSPC in pci_resize_resource() on fixed switch fabrics, where a
> + * window shared with a sibling's still-assigned BAR cannot be released and
> + * regrown at resize time.
> + *
> + * The reservation is strictly optional: __assign_resources_sorted()
> satisfies
> + * all required resources first and only then places add_size on leftover
> + * space, so reserving by default cannot make a required BAR or window
> fail.
> + *
> + * Return the extra bytes to reserve and raise *add_align to the BAR's
> + * alignment so the reserved space can actually hold the grown BAR.
> + */
> +static resource_size_t pci_rebar_reserve_size(struct pci_dev *dev, int
> resno,
> +					      resource_size_t *add_align)
> +{
> +	struct resource *res = pci_resource_n(dev, resno);
> +	resource_size_t max_size, cur_size;
> +	int max;
> +
> +	if (pci_rebar_no_reserve || resno >= PCI_STD_NUM_BARS)
> +		return 0;
> +
> +	if ((res->flags & (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH)) !=
> +	    (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH))
> +		return 0;
> +
> +	max = pci_rebar_get_max_size(dev, resno);
> +	if (max < 0)
> +		return 0;
> +
> +	max_size = pci_rebar_size_to_bytes(max);
> +	cur_size = resource_size(res);
> +	if (max_size <= cur_size)
> +		return 0;
> +
> +	*add_align = max(*add_align, max_size);
> +	return max_size - cur_size;
> +}
> +
>  /**
>   * pbus_size_mem() - Size the memory window of a given bus
>   *
> @@ -1285,6 +1333,7 @@ static void pbus_size_mem(struct pci_bus *bus,
> struct resource *b_res,
>  	int order, max_order;
>  	resource_size_t children_add_size = 0;
>  	resource_size_t add_align = 0;
> +	resource_size_t rebar_align = 0;
> 
>  	if (!b_res)
>  		return;
> @@ -1343,15 +1392,29 @@ static void pbus_size_mem(struct pci_bus *bus,
> struct resource *b_res,
>  				aligns[order] += align;
>  			if (order > max_order)
>  				max_order = order;
> +
> +			size += pci_rebar_reserve_size(dev, i, &rebar_align);
>  		}
>  	}
> 
>  	win_align = pci_min_window_alignment(bus, b_res->flags);
>  	min_align = calculate_head_align(aligns, max_order);
>  	min_align = max(min_align, win_align);
> +	min_align = max(min_align, rebar_align);
>  	size0 = calculate_memsize(size, realloc_head ? 0 : add_size,
>  				  0, win_align);
> 
> +	/*
> +	 * A window that reserves ReBAR headroom is larger than its own
> +	 * alignment (e.g. 32 GiB BAR + doorbell => 33 GiB > 32 GiB). The
> +	 * parent only reserves alignment padding for a child window when
> +	 * its size <= its alignment, so round such a window's alignment up
> +	 * to a power of two >= its size; sibling 32 GiB BARs behind one
> +	 * switch then each land in a properly aligned slot.
> +	 */
> +	if (rebar_align && size0)
> +		min_align = max(min_align, roundup_pow_of_two(size0));
> +
>  	if (size0) {
>  		resource_set_range(b_res, min_align, size0);
>  		b_res->flags &= ~IORESOURCE_DISABLED;
> @@ -2179,6 +2242,52 @@ static void pci_prepare_next_assign_round(struct
> list_head *fail_head,
>   * Second and later try will clear small leaf bridge res.
>   * Will stop till to the max depth if can not find good one.
>   */
> +/*
> + * Release the 64-bit prefetchable BARs of resizable-BAR display
> devices so the
> + * windows enclosing them empty out and can be re-sized. The BAR *size*
> is left
> + * untouched (only released and re-placed), so the hardware ReBAR is never
> + * programmed here - the driver still owns the actual resize.
> + */
> +static int pci_release_rebar_bars_cb(struct pci_dev *dev, void *data)
> +{
> +	struct resource *r;
> +	unsigned int i;
> +
> +	if ((dev->class >> 16) != PCI_BASE_CLASS_DISPLAY)
> +		return 0;
> +	if (pci_rebar_get_max_size(dev, 0) <= pci_rebar_get_current_size(dev, 0))
> +		return 0;
> +	pci_dev_for_each_resource(dev, r, i) {
> +		if (i >= PCI_BRIDGE_RESOURCES)
> +			break;
> +		if (r->parent && (r->flags & IORESOURCE_MEM_64) &&
> +		    (r->flags & IORESOURCE_PREFETCH))
> +			pci_release_resource(dev, i);
> +	}
> +	return 0;
> +}
> +/*
> + * Release the now-empty prefetchable bridge windows bottom-up so the
> sizing
> + * pass re-sizes them (with pci_rebar_pref_reserve() reservation) to
> fit the
> + * BARs a driver will later grow.
> + */
> +static void pci_release_rebar_windows(struct pci_bus *bus)
> +{
> +	struct pci_dev *dev;
> +	struct resource *w;
> +
> +	list_for_each_entry(dev, &bus->devices, bus_list)
> +		if (dev->subordinate)
> +			pci_release_rebar_windows(dev->subordinate);
> +
> +	if (!bus->self)
> +		return;
> +	w = &bus->self->resource[PCI_BRIDGE_PREF_MEM_WINDOW];
> +	if (w->parent && (w->flags & IORESOURCE_MEM_64) && !w->child)
> +		pci_release_resource(bus->self, PCI_BRIDGE_PREF_MEM_WINDOW);
> +}
> +
>  void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>  {
>  	LIST_HEAD(realloc_head);
> @@ -2190,6 +2299,15 @@ void
> pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>  	int pci_try_num = 1;
>  	enum enable_type enable_local;
> 
> +	/*
> +	 * Release resizable device BARs and their prefetchable windows so the
> +	 * sizing pass re-sizes those windows large enough
> (pci_rebar_pref_reserve)
> +	 * for the BARs a driver will later grow. Only released and re-placed
> - the
> +	 * BAR size is left untouched, so the hardware ReBAR is never programmed.
> +	 */
> +	pci_walk_bus(bus, pci_release_rebar_bars_cb, NULL);
> +	pci_release_rebar_windows(bus);
> +
>  	/* Don't realloc if asked to do so */
>  	enable_local = pci_realloc_detect(bus, pci_realloc_enable);
>  	if (pci_realloc_enabled(enable_local)) {
> 
> base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229
> 

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 14958 bytes --]

On Mon, 31 Aug 2026, Christian König wrote:

> On 8/28/26 23:37, Geramy Loveless wrote:
> > Firmware typically sizes prefetchable bridge windows for the boot-time
> > BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
> > no room for a driver to grow a Resizable BAR afterwards: every window
> > from the leaf up to the root is sized for the small BAR, so
> > pci_resize_resource() fails with -ENOSPC. 
> > 
> > Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
> > bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
> > VRAM BAR) pushes the required window size past the large BAR's alignment. 
> > Because bridge windows round up to a power of two, this forces a massive 
> > alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
> > wasting ~32 GiB per GPU).
> > 
> > This patch solves both issues to enable ReBAR on cascaded switch fabrics:
> > 
> > 1. Reserve Headroom: 
> > Reserve prefetchable window headroom for the maximum size of each 
> > downstream Resizable BAR during the bridge sizing pass. The device BAR 
> > and hardware ReBAR are left at their boot size to prevent tearing down 
> > firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
> 
> Yeah that was suggested before but that is clearly not something you can 
> do in common code.
> 
> The ReBAR fields often doesn't reflect the actual needed space but 
> rather the maximum the HW address logic can resolve.
> 
> So what you end up with is allocating multiple TiB for a window which 
> just needs few GiB, sometimes even completely overflowing the 64bit 
> address space made available by the root complex.

Yes. A naive approach to (only) go to the max ReBAR allows just doesn't 
work well enough to be usable in general case.

A general algorithm to pick the largests fitting sizes for ReBARs is still 
under works, a few steps away still, and more if I discover more 
challenges I need address before I can take that final step.

> > 2. Pack Oversized Windows (BAR Demotion): 
> > The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR 
> > to be assigned from the non-prefetchable window. By clearing the PREFETCH 
> > flag on small control BARs during enumeration, they are placed below 4 GiB. 
> > The prefetchable window is then cleanly sized to the large BAR alone.
> 
> Interesting hack, shouldn't really matter for AMD GPUs but that is 
> something more on the heuristic side. 
> 
> A perfectly valid trick which is done by BIOS implementations but what 
> the Linux PCI subsystem still hasn't learned are back to back 
> allocations, e.g. something like this:
> 
> BAR0 32 GiB of GPU #1
> BAR1 2	MiB of GPU #1
> 
> BAR1 2	MiB of GPU #2
> BAR0 32 GiB of GPU #2

Assuming you're oversimplifying the layout above, this was added by the commit 
9036bd0efcb6 ("PCI: Align head space better").

--
 i.

> BAR0 32 GiB of GPU #3
> ...
> 
> I strongly suggest to implement that one first and see if it helps with your use case.
> 
> Regards,
> Christian.
> 
> > 
> > Why this is safe for 32-bit (< 4G) MMIO space:
> > To prevent exhausting legacy < 4G space, demotion is strictly bounded. It 
> > only triggers beside a massive prefetchable BAR (>= 64 MiB), and the demoted 
> > BAR is strictly capped at 16 MiB (PCI_BAR_PACK_CAP). Even on an 8x GPU 
> > system, this consumes at most 128 MiB of < 4G space. 
> > 
> > Fallback / Alignment logic:
> > If the 32-bit space is exhausted, small BARs remain in the prefetchable 
> > window. The alignment logic gracefully handles this by rounding the bridge 
> > window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
> > BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
> > properly aligned slots.
> > 
> > Reserving is safe by default: __assign_resources_sorted() satisfies all
> > required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
> > to restore previous behaviours.
> > 
> > Testing Context:
> > - ASUS K14PG-D24
> > - 2x EPYC 9354
> > - 8x Radeon AI PRO R9700 (Navi 48, 1002:7551)
> > - Broadcom PEX890xx switches
> > - Kernel base: 45c13f3f9e3b (tested with SEV-SNP passthrough=1)
> > 
> > Signed-off-by: Geramy Loveless <gloveless@jqluv.com>
> > ---
> >  .../admin-guide/kernel-parameters.txt         |   8 ++
> >  drivers/pci/pci.c                             |  10 ++
> >  drivers/pci/pci.h                             |   1 +
> >  drivers/pci/setup-bus.c                       | 118 ++++++++++++++++++
> >  4 files changed, 137 insertions(+)
> > 
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt
> > b/Documentation/admin-guide/kernel-parameters.txt
> > index 37006fc3eac3..82d2b340fb98 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -5246,6 +5246,14 @@ Kernel parameters
> >  		hpbussize=nn	The minimum amount of additional bus numbers
> >  				reserved for buses below a hotplug bridge.
> >  				Default is 1.
> > +		no_rebar_reserve	Do not reserve prefetchable bridge-window
> > +				space for the maximum size of downstream Resizable
> > +				BARs. By default such space is reserved so a driver
> > +				can grow a BAR later (e.g. GPU VRAM BAR) even behind
> > +				fixed PCIe switch fabrics that firmware sized for the
> > +				boot-time BAR; the device BAR is left at its boot size
> > +				for the driver to resize and only bridge windows are
> > +				enlarged. Use this to restore the old behaviour.
> >  		realloc=	Enable/disable reallocating PCI bridge resources
> >  				if allocations done by BIOS are too small to
> >  				accommodate resources required by all child
> > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > index b2879a6be5f8..c9901b15432a 100644
> > --- a/drivers/pci/pci.c
> > +++ b/drivers/pci/pci.c
> > @@ -120,6 +120,14 @@ unsigned long pci_hotplug_mmio_pref_size =
> > DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
> >  #define DEFAULT_HOTPLUG_BUS_SIZE	1
> >  unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
> > 
> > +/*
> > + * Prefetchable bridge-window headroom is reserved for the maximum size of
> > + * downstream Resizable BARs by default, so a driver can grow one later
> > + * (e.g. amdgpu VRAM BAR) even behind fixed PCIe switch fabrics.
> > + * pci=no_rebar_reserve disables this.
> > + */
> > +bool pci_rebar_no_reserve;
> > +
> > 
> >  /* PCIe MPS/MRRS strategy; can be overridden by kernel command-line
> > param */
> >  enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
> > @@ -6840,6 +6848,8 @@ static int __init pci_setup(char *str)
> >  					simple_strtoul(str + 10, &str, 0);
> >  				if (pci_hotplug_bus_size > 0xff)
> >  					pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
> > +			} else if (!strncmp(str, "no_rebar_reserve", 16)) {
> > +				pci_rebar_no_reserve = true;
> >  			} else if (!strncmp(str, "pcie_bus_tune_off", 17)) {
> >  				pcie_bus_config = PCIE_BUS_TUNE_OFF;
> >  			} else if (!strncmp(str, "pcie_bus_safe", 13)) {
> > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > index ba3c3fddddc2..6eddcb492425 100644
> > --- a/drivers/pci/pci.h
> > +++ b/drivers/pci/pci.h
> > @@ -409,6 +409,7 @@ extern unsigned long pci_hotplug_io_size;
> >  extern unsigned long pci_hotplug_mmio_size;
> >  extern unsigned long pci_hotplug_mmio_pref_size;
> >  extern unsigned long pci_hotplug_bus_size;
> > +extern bool pci_rebar_no_reserve;
> > 
> >  static inline bool pci_is_cardbus_bridge(struct pci_dev *dev)
> >  {
> > diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> > index e8c94aa1d3c1..aa02b8454269 100644
> > --- a/drivers/pci/setup-bus.c
> > +++ b/drivers/pci/setup-bus.c
> > @@ -19,6 +19,7 @@
> >  #include <linux/bug.h>
> >  #include <linux/init.h>
> >  #include <linux/kernel.h>
> > +#include <linux/log2.h>
> >  #include <linux/minmax.h>
> >  #include <linux/module.h>
> >  #include <linux/pci.h>
> > @@ -1258,6 +1259,53 @@ static bool pbus_size_mem_optional(struct pci_dev
> > *dev, int resno,
> >  	return true;
> >  }
> > 
> > +/*
> > + * pci_rebar_reserve_size - prefetchable window headroom to reserve for
> > a BAR
> > + *
> > + * Reserve enough prefetchable window space to later grow @resno of @dev to
> > + * its maximum Resizable BAR size. This is done for every ReBAR-capable
> > device
> > + * by default (disable with pci=no_rebar_reserve). Only the optional
> > (add_size)
> > + * part of the enclosing window is inflated here; the device BAR
> > resource and
> > + * the hardware ReBAR are left at their boot size, so the driver still
> > performs
> > + * the actual resize into the reserved window. Reserving the window up
> > front
> > + * avoids -ENOSPC in pci_resize_resource() on fixed switch fabrics, where a
> > + * window shared with a sibling's still-assigned BAR cannot be released and
> > + * regrown at resize time.
> > + *
> > + * The reservation is strictly optional: __assign_resources_sorted()
> > satisfies
> > + * all required resources first and only then places add_size on leftover
> > + * space, so reserving by default cannot make a required BAR or window
> > fail.
> > + *
> > + * Return the extra bytes to reserve and raise *add_align to the BAR's
> > + * alignment so the reserved space can actually hold the grown BAR.
> > + */
> > +static resource_size_t pci_rebar_reserve_size(struct pci_dev *dev, int
> > resno,
> > +					      resource_size_t *add_align)
> > +{
> > +	struct resource *res = pci_resource_n(dev, resno);
> > +	resource_size_t max_size, cur_size;
> > +	int max;
> > +
> > +	if (pci_rebar_no_reserve || resno >= PCI_STD_NUM_BARS)
> > +		return 0;
> > +
> > +	if ((res->flags & (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH)) !=
> > +	    (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH))
> > +		return 0;
> > +
> > +	max = pci_rebar_get_max_size(dev, resno);
> > +	if (max < 0)
> > +		return 0;
> > +
> > +	max_size = pci_rebar_size_to_bytes(max);
> > +	cur_size = resource_size(res);
> > +	if (max_size <= cur_size)
> > +		return 0;
> > +
> > +	*add_align = max(*add_align, max_size);
> > +	return max_size - cur_size;
> > +}
> > +
> >  /**
> >   * pbus_size_mem() - Size the memory window of a given bus
> >   *
> > @@ -1285,6 +1333,7 @@ static void pbus_size_mem(struct pci_bus *bus,
> > struct resource *b_res,
> >  	int order, max_order;
> >  	resource_size_t children_add_size = 0;
> >  	resource_size_t add_align = 0;
> > +	resource_size_t rebar_align = 0;
> > 
> >  	if (!b_res)
> >  		return;
> > @@ -1343,15 +1392,29 @@ static void pbus_size_mem(struct pci_bus *bus,
> > struct resource *b_res,
> >  				aligns[order] += align;
> >  			if (order > max_order)
> >  				max_order = order;
> > +
> > +			size += pci_rebar_reserve_size(dev, i, &rebar_align);
> >  		}
> >  	}
> > 
> >  	win_align = pci_min_window_alignment(bus, b_res->flags);
> >  	min_align = calculate_head_align(aligns, max_order);
> >  	min_align = max(min_align, win_align);
> > +	min_align = max(min_align, rebar_align);
> >  	size0 = calculate_memsize(size, realloc_head ? 0 : add_size,
> >  				  0, win_align);
> > 
> > +	/*
> > +	 * A window that reserves ReBAR headroom is larger than its own
> > +	 * alignment (e.g. 32 GiB BAR + doorbell => 33 GiB > 32 GiB). The
> > +	 * parent only reserves alignment padding for a child window when
> > +	 * its size <= its alignment, so round such a window's alignment up
> > +	 * to a power of two >= its size; sibling 32 GiB BARs behind one
> > +	 * switch then each land in a properly aligned slot.
> > +	 */
> > +	if (rebar_align && size0)
> > +		min_align = max(min_align, roundup_pow_of_two(size0));
> > +
> >  	if (size0) {
> >  		resource_set_range(b_res, min_align, size0);
> >  		b_res->flags &= ~IORESOURCE_DISABLED;
> > @@ -2179,6 +2242,52 @@ static void pci_prepare_next_assign_round(struct
> > list_head *fail_head,
> >   * Second and later try will clear small leaf bridge res.
> >   * Will stop till to the max depth if can not find good one.
> >   */
> > +/*
> > + * Release the 64-bit prefetchable BARs of resizable-BAR display
> > devices so the
> > + * windows enclosing them empty out and can be re-sized. The BAR *size*
> > is left
> > + * untouched (only released and re-placed), so the hardware ReBAR is never
> > + * programmed here - the driver still owns the actual resize.
> > + */
> > +static int pci_release_rebar_bars_cb(struct pci_dev *dev, void *data)
> > +{
> > +	struct resource *r;
> > +	unsigned int i;
> > +
> > +	if ((dev->class >> 16) != PCI_BASE_CLASS_DISPLAY)
> > +		return 0;
> > +	if (pci_rebar_get_max_size(dev, 0) <= pci_rebar_get_current_size(dev, 0))
> > +		return 0;
> > +	pci_dev_for_each_resource(dev, r, i) {
> > +		if (i >= PCI_BRIDGE_RESOURCES)
> > +			break;
> > +		if (r->parent && (r->flags & IORESOURCE_MEM_64) &&
> > +		    (r->flags & IORESOURCE_PREFETCH))
> > +			pci_release_resource(dev, i);
> > +	}
> > +	return 0;
> > +}
> > +
> > +/*
> > + * Release the now-empty prefetchable bridge windows bottom-up so the
> > sizing
> > + * pass re-sizes them (with pci_rebar_pref_reserve() reservation) to
> > fit the
> > + * BARs a driver will later grow.
> > + */
> > +static void pci_release_rebar_windows(struct pci_bus *bus)
> > +{
> > +	struct pci_dev *dev;
> > +	struct resource *w;
> > +
> > +	list_for_each_entry(dev, &bus->devices, bus_list)
> > +		if (dev->subordinate)
> > +			pci_release_rebar_windows(dev->subordinate);
> > +
> > +	if (!bus->self)
> > +		return;
> > +	w = &bus->self->resource[PCI_BRIDGE_PREF_MEM_WINDOW];
> > +	if (w->parent && (w->flags & IORESOURCE_MEM_64) && !w->child)
> > +		pci_release_resource(bus->self, PCI_BRIDGE_PREF_MEM_WINDOW);
> > +}
> > +
> >  void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
> >  {
> >  	LIST_HEAD(realloc_head);
> > @@ -2190,6 +2299,15 @@ void
> > pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
> >  	int pci_try_num = 1;
> >  	enum enable_type enable_local;
> > 
> > +	/*
> > +	 * Release resizable device BARs and their prefetchable windows so the
> > +	 * sizing pass re-sizes those windows large enough
> > (pci_rebar_pref_reserve)
> > +	 * for the BARs a driver will later grow. Only released and re-placed
> > - the
> > +	 * BAR size is left untouched, so the hardware ReBAR is never programmed.
> > +	 */
> > +	pci_walk_bus(bus, pci_release_rebar_bars_cb, NULL);
> > +	pci_release_rebar_windows(bus);
> > +
> >  	/* Don't realloc if asked to do so */
> >  	enable_local = pci_realloc_detect(bus, pci_realloc_enable);
> >  	if (pci_realloc_enabled(enable_local)) {
> > 
> > base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229
> 

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Geramy Loveless]

On 8/31/26 9:31 AM, Ilpo Järvinen wrote:
> On Fri, 28 Aug 2026, Geramy Loveless wrote:
>
>> Firmware typically sizes prefetchable bridge windows for the boot-time
>> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
>> no room for a driver to grow a Resizable BAR afterwards: every window
>> from the leaf up to the root is sized for the small BAR, so
>> pci_resize_resource() fails with -ENOSPC. 
>>
>> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
>> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
>> VRAM BAR) pushes the required window size past the large BAR's alignment. 
>> Because bridge windows round up to a power of two,
> Do they??? In which kernel version I must ask??
I'm sorry this was in reference to a prior patch I made, **ignore**
>
>> this forces a massive 
>> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
>> wasting ~32 GiB per GPU).
> The largest I've seen is 32GB + half of that. And that's not with the 
> latest kernel.

This is in reference to when there are two R9700s on one switch basically.
As far as my understanding currently how it works is that when there are two GPUs they both have a BAR0, BAR2, and BAR5 the BAR5 goes onto the 32-bit window.
Bar 0 and 2 don't go onto the second window by default as they could overfill it of course, bar 0 would not fit period, and 2 might which is the case I took. 
This allows BAR 0 on (GPU 0) in leaf 0 on window 0 to align as a multiple of 32G like it I guess has to? This is how I understand it, so if R9700 has 32G on, bar0, leaf 0, window 0, and share the same window as GPU 1 then we have to pad the space between GPU0 BARs on window 0 by 32G, meaning BAR2 really is also using 32G, its not possible or at least right now its not possible to move BAR2 to the end of all the allocations it doesnt fit the standards? Keep in mind this is not my specialty here I am trying my best to understand and solve the issue at hand.

>
>> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
>>
>> 1. Reserve Headroom: 
>> Reserve prefetchable window headroom for the maximum size of each 
>> downstream Resizable BAR during the bridge sizing pass. The device BAR 
>> and hardware ReBAR are left at their boot size to prevent tearing down 
>> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
> ... reorganizing ...
>
>> Fallback / Alignment logic:
>> If the 32-bit space is exhausted, small BARs remain in the prefetchable 
>> window. The alignment logic gracefully handles this by rounding the bridge 
>> window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
>> BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
>> properly aligned slots.
>>
>> Reserving is safe by default: __assign_resources_sorted() satisfies all
>> required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
>> to restore previous behaviours.
> And how does this fallback, if the largest size doesn't fit? What are the 
> failure modes?
So if the smaller bars under 16M don't fit in the 32-bit addressable space the fallback mode is to do as I said above to your prior comment, basically put.
Don't move the 2M BAR2 over to the 32-bit addressable space / window and instead fallback to aligning the prior window by the size of the GPUs memory space as I understand it, when you have gpu0 and gpu1 both requiring 32GB their address has to be a multiple of their allocation size? meaning we get 32G for BAR0 and 32G for BAR2 even though BAR2 only needs 2M because its not aligned to allow GPU1 to be on a aligned space?
>
> It's correct that the sizing decision should be done in pbus_size_mem() 
> but it must not result in breaking things which a naive approach like 
> this will surely do.
>
> Without proper fallbacks done, there will be too much breakage it's a 
> showstopper for a naive patch like this.
>
> Basically, what we'd want to do is try with the largest size, then reduce 
> it one step at a time (because it's 2^n -> 2^(n-1) even one step saves 
> quite much so one might not need to reduce that many steps). But also, in 
> some IOV enabled cases the oversubscription of the space is very 
> substantial and you'd still want to maximize the BAR sizes to what is 
> allowed by the available space.
>
> Fallback is hard to implement, because sizing and assignment are quite far 
> from each other codewise and there are no structures to carry information 
> over to the next retry phase.
>
>> 2. Pack Oversized Windows (BAR Demotion): 
>> The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR 
>> to be assigned from the non-prefetchable window. By clearing the PREFETCH 
>> flag on small control BARs during enumeration, they are placed below 4 GiB. 
>> The prefetchable window is then cleanly sized to the large BAR alone.
>>
>> Why this is safe for 32-bit (< 4G) MMIO space:
>> To prevent exhausting legacy < 4G space, demotion is strictly bounded. It 
>> only triggers beside a massive prefetchable BAR (>= 64 MiB), and the demoted 
>> BAR is strictly capped at 16 MiB (PCI_BAR_PACK_CAP). Even on an 8x GPU 
>> system, this consumes at most 128 MiB of < 4G space.  
> I'm not entirely sure I agree with this, at least it shouldn't be the 
> default configuration.
That's okay I believe, it probably is not the default case anyhow.
>
> This would also break setups where the GPU has large BARs that do not 
> share the alignment.
>
> It might be workable as very targetted instruction to put a particular BAR 
> into non-prefetchable window. For that, one needs to write a parser for 
> giving them on cmdline though but that might end up being generally 
> useful for avoiding some hotplug cannot-predict-the-future problems 
> (the current memsize controls are too coarse-grained to be useful in many 
> hotplug scenarios).
>
>
> A somewhat related problem currently is that more and more devices are 
> taking the liberty of not setting PREFETCH because recent PCIe spec made 
> the bridge window difference be 32/64-bit based. This results in very 
> suboptimal placement for such BARs.
>
>
> BTW, why you're not sending the patches as a proper series but merging 
> them into a single email??? It makes things harder for the reviewers. :-(

Yes that was an accident actually, I was using command line tools to build the patch file and had done that by accident.
Technically these patches where fully related but my tree is split in the fixes, I had to rebase it this is why I submitted multiple versions.
I do have a new version which resolves a few problems / use cases I appreciate you taking the time to review this work.

>
>
> Also, I note you sent multiple versions within a day. Please give some 
> time for commenting the first version. You should also list the changes 
> you've made in each version (the patch history).
I had noticed my first patch was split and realized how difficult that would of been to review. So I resubmitted the patch as one instead of two because they both technically depend on each other and thats not how it would get merged anyways because then the window compression feature would be missing, ie. not a huge fan of resource waste. Then I submitted another one generalizing the patch a bit more so it fit use cases without breaking other machines. I will in the future try to do patch submission better and appreciate your patience, I will also make sure follow-up patch versions are summarizing the changes made between the prior one and next one, thank you!
>

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Geramy Loveless]

On 8/31/26 9:58 AM, Ilpo Järvinen wrote:
> On Mon, 31 Aug 2026, Christian König wrote:
>
>> On 8/28/26 23:37, Geramy Loveless wrote:
>>> Firmware typically sizes prefetchable bridge windows for the boot-time
>>> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
>>> no room for a driver to grow a Resizable BAR afterwards: every window
>>> from the leaf up to the root is sized for the small BAR, so
>>> pci_resize_resource() fails with -ENOSPC. 
>>>
>>> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
>>> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
>>> VRAM BAR) pushes the required window size past the large BAR's alignment. 
>>> Because bridge windows round up to a power of two, this forces a massive 
>>> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
>>> wasting ~32 GiB per GPU).
>>>
>>> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
>>>
>>> 1. Reserve Headroom: 
>>> Reserve prefetchable window headroom for the maximum size of each 
>>> downstream Resizable BAR during the bridge sizing pass. The device BAR 
>>> and hardware ReBAR are left at their boot size to prevent tearing down 
>>> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
>> Yeah that was suggested before but that is clearly not something you can 
>> do in common code.
>>
>> The ReBAR fields often doesn't reflect the actual needed space but 
>> rather the maximum the HW address logic can resolve.
>>
>> So what you end up with is allocating multiple TiB for a window which 
>> just needs few GiB, sometimes even completely overflowing the 64bit 
>> address space made available by the root complex.

Yeah I could imagine that would be bad, hence I tried to compact the 64-bit space as well. As far as I understand PCI/PCIe standards its making this patch difficult.
Christian I'm not sure if you could share these reference documents but that probably would be a better start for me to look at before I update the patch or make changes, also I need to wait for a review on the last patch I submitted too, so that leaves me with some time to review some standards if you know specifically where to look, if not thats fine too.

> Yes. A naive approach to (only) go to the max ReBAR allows just doesn't 
> work well enough to be usable in general case.
I did not know the max would report above the amount is actually needed because of vendors decisions in the cards, that's interesting.
>
> A general algorithm to pick the largests fitting sizes for ReBARs is still 
> under works, a few steps away still, and more if I discover more 
> challenges I need address before I can take that final step.
So then I would assume instead of doing the "hack" i'm having to use I would change it to your new algorithm?
>
>>> 2. Pack Oversized Windows (BAR Demotion): 
>>> The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR 
>>> to be assigned from the non-prefetchable window. By clearing the PREFETCH 
>>> flag on small control BARs during enumeration, they are placed below 4 GiB. 
>>> The prefetchable window is then cleanly sized to the large BAR alone.
>> Interesting hack, shouldn't really matter for AMD GPUs but that is 
>> something more on the heuristic side. 
>>
>> A perfectly valid trick which is done by BIOS implementations but what 
>> the Linux PCI subsystem still hasn't learned are back to back 
>> allocations, e.g. something like this:
>>
>> BAR0 32 GiB of GPU #1
>> BAR1 2	MiB of GPU #1
>>
>> BAR1 2	MiB of GPU #2
>> BAR0 32 GiB of GPU #2
This does work in practice actually, but it only works for 2 GPUs or so I thought, instead of doing that I opted it out to look for a real multi-purpose solution that fits more than my use case of one switch and two GPUs because I figured the chance of that getting implemented over a patch like this would be more likely.
> Assuming you're oversimplifying the layout above, this was added by the commit 
> 9036bd0efcb6 ("PCI: Align head space better").
>
> --
>  i.
>
>> BAR0 32 GiB of GPU #3
>> ...
>>
>> I strongly suggest to implement that one first and see if it helps with your use case.
>>
>> Regards,
>> Christian.
It does, but I didn't think it was upstream-able at that time, I am new to linux kernel patching so I am learning what is acceptable and what is not haha :)
Thanks for the honest review.
>>
>>> Why this is safe for 32-bit (< 4G) MMIO space:
>>> To prevent exhausting legacy < 4G space, demotion is strictly bounded. It 
>>> only triggers beside a massive prefetchable BAR (>= 64 MiB), and the demoted 
>>> BAR is strictly capped at 16 MiB (PCI_BAR_PACK_CAP). Even on an 8x GPU 
>>> system, this consumes at most 128 MiB of < 4G space. 
>>>
>>> Fallback / Alignment logic:
>>> If the 32-bit space is exhausted, small BARs remain in the prefetchable 
>>> window. The alignment logic gracefully handles this by rounding the bridge 
>>> window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
>>> BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
>>> properly aligned slots.
>>>
>>> Reserving is safe by default: __assign_resources_sorted() satisfies all
>>> required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
>>> to restore previous behaviours.
>>>
>>> Testing Context:
>>> - ASUS K14PG-D24
>>> - 2x EPYC 9354
>>> - 8x Radeon AI PRO R9700 (Navi 48, 1002:7551)
>>> - Broadcom PEX890xx switches
>>> - Kernel base: 45c13f3f9e3b (tested with SEV-SNP passthrough=1)
>>>
>>> Signed-off-by: Geramy Loveless <gloveless@jqluv.com>
>>> ---
>>>  .../admin-guide/kernel-parameters.txt         |   8 ++
>>>  drivers/pci/pci.c                             |  10 ++
>>>  drivers/pci/pci.h                             |   1 +
>>>  drivers/pci/setup-bus.c                       | 118 ++++++++++++++++++
>>>  4 files changed, 137 insertions(+)
>>>
>>> diff --git a/Documentation/admin-guide/kernel-parameters.txt
>>> b/Documentation/admin-guide/kernel-parameters.txt
>>> index 37006fc3eac3..82d2b340fb98 100644
>>> --- a/Documentation/admin-guide/kernel-parameters.txt
>>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>>> @@ -5246,6 +5246,14 @@ Kernel parameters
>>>  		hpbussize=nn	The minimum amount of additional bus numbers
>>>  				reserved for buses below a hotplug bridge.
>>>  				Default is 1.
>>> +		no_rebar_reserve	Do not reserve prefetchable bridge-window
>>> +				space for the maximum size of downstream Resizable
>>> +				BARs. By default such space is reserved so a driver
>>> +				can grow a BAR later (e.g. GPU VRAM BAR) even behind
>>> +				fixed PCIe switch fabrics that firmware sized for the
>>> +				boot-time BAR; the device BAR is left at its boot size
>>> +				for the driver to resize and only bridge windows are
>>> +				enlarged. Use this to restore the old behaviour.
>>>  		realloc=	Enable/disable reallocating PCI bridge resources
>>>  				if allocations done by BIOS are too small to
>>>  				accommodate resources required by all child
>>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>>> index b2879a6be5f8..c9901b15432a 100644
>>> --- a/drivers/pci/pci.c
>>> +++ b/drivers/pci/pci.c
>>> @@ -120,6 +120,14 @@ unsigned long pci_hotplug_mmio_pref_size =
>>> DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
>>>  #define DEFAULT_HOTPLUG_BUS_SIZE	1
>>>  unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
>>>
>>> +/*
>>> + * Prefetchable bridge-window headroom is reserved for the maximum size of
>>> + * downstream Resizable BARs by default, so a driver can grow one later
>>> + * (e.g. amdgpu VRAM BAR) even behind fixed PCIe switch fabrics.
>>> + * pci=no_rebar_reserve disables this.
>>> + */
>>> +bool pci_rebar_no_reserve;
>>> +
>>>
>>>  /* PCIe MPS/MRRS strategy; can be overridden by kernel command-line
>>> param */
>>>  enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
>>> @@ -6840,6 +6848,8 @@ static int __init pci_setup(char *str)
>>>  					simple_strtoul(str + 10, &str, 0);
>>>  				if (pci_hotplug_bus_size > 0xff)
>>>  					pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
>>> +			} else if (!strncmp(str, "no_rebar_reserve", 16)) {
>>> +				pci_rebar_no_reserve = true;
>>>  			} else if (!strncmp(str, "pcie_bus_tune_off", 17)) {
>>>  				pcie_bus_config = PCIE_BUS_TUNE_OFF;
>>>  			} else if (!strncmp(str, "pcie_bus_safe", 13)) {
>>> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
>>> index ba3c3fddddc2..6eddcb492425 100644
>>> --- a/drivers/pci/pci.h
>>> +++ b/drivers/pci/pci.h
>>> @@ -409,6 +409,7 @@ extern unsigned long pci_hotplug_io_size;
>>>  extern unsigned long pci_hotplug_mmio_size;
>>>  extern unsigned long pci_hotplug_mmio_pref_size;
>>>  extern unsigned long pci_hotplug_bus_size;
>>> +extern bool pci_rebar_no_reserve;
>>>
>>>  static inline bool pci_is_cardbus_bridge(struct pci_dev *dev)
>>>  {
>>> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
>>> index e8c94aa1d3c1..aa02b8454269 100644
>>> --- a/drivers/pci/setup-bus.c
>>> +++ b/drivers/pci/setup-bus.c
>>> @@ -19,6 +19,7 @@
>>>  #include <linux/bug.h>
>>>  #include <linux/init.h>
>>>  #include <linux/kernel.h>
>>> +#include <linux/log2.h>
>>>  #include <linux/minmax.h>
>>>  #include <linux/module.h>
>>>  #include <linux/pci.h>
>>> @@ -1258,6 +1259,53 @@ static bool pbus_size_mem_optional(struct pci_dev
>>> *dev, int resno,
>>>  	return true;
>>>  }
>>>
>>> +/*
>>> + * pci_rebar_reserve_size - prefetchable window headroom to reserve for
>>> a BAR
>>> + *
>>> + * Reserve enough prefetchable window space to later grow @resno of @dev to
>>> + * its maximum Resizable BAR size. This is done for every ReBAR-capable
>>> device
>>> + * by default (disable with pci=no_rebar_reserve). Only the optional
>>> (add_size)
>>> + * part of the enclosing window is inflated here; the device BAR
>>> resource and
>>> + * the hardware ReBAR are left at their boot size, so the driver still
>>> performs
>>> + * the actual resize into the reserved window. Reserving the window up
>>> front
>>> + * avoids -ENOSPC in pci_resize_resource() on fixed switch fabrics, where a
>>> + * window shared with a sibling's still-assigned BAR cannot be released and
>>> + * regrown at resize time.
>>> + *
>>> + * The reservation is strictly optional: __assign_resources_sorted()
>>> satisfies
>>> + * all required resources first and only then places add_size on leftover
>>> + * space, so reserving by default cannot make a required BAR or window
>>> fail.
>>> + *
>>> + * Return the extra bytes to reserve and raise *add_align to the BAR's
>>> + * alignment so the reserved space can actually hold the grown BAR.
>>> + */
>>> +static resource_size_t pci_rebar_reserve_size(struct pci_dev *dev, int
>>> resno,
>>> +					      resource_size_t *add_align)
>>> +{
>>> +	struct resource *res = pci_resource_n(dev, resno);
>>> +	resource_size_t max_size, cur_size;
>>> +	int max;
>>> +
>>> +	if (pci_rebar_no_reserve || resno >= PCI_STD_NUM_BARS)
>>> +		return 0;
>>> +
>>> +	if ((res->flags & (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH)) !=
>>> +	    (IORESOURCE_MEM_64 | IORESOURCE_PREFETCH))
>>> +		return 0;
>>> +
>>> +	max = pci_rebar_get_max_size(dev, resno);
>>> +	if (max < 0)
>>> +		return 0;
>>> +
>>> +	max_size = pci_rebar_size_to_bytes(max);
>>> +	cur_size = resource_size(res);
>>> +	if (max_size <= cur_size)
>>> +		return 0;
>>> +
>>> +	*add_align = max(*add_align, max_size);
>>> +	return max_size - cur_size;
>>> +}
>>> +
>>>  /**
>>>   * pbus_size_mem() - Size the memory window of a given bus
>>>   *
>>> @@ -1285,6 +1333,7 @@ static void pbus_size_mem(struct pci_bus *bus,
>>> struct resource *b_res,
>>>  	int order, max_order;
>>>  	resource_size_t children_add_size = 0;
>>>  	resource_size_t add_align = 0;
>>> +	resource_size_t rebar_align = 0;
>>>
>>>  	if (!b_res)
>>>  		return;
>>> @@ -1343,15 +1392,29 @@ static void pbus_size_mem(struct pci_bus *bus,
>>> struct resource *b_res,
>>>  				aligns[order] += align;
>>>  			if (order > max_order)
>>>  				max_order = order;
>>> +
>>> +			size += pci_rebar_reserve_size(dev, i, &rebar_align);
>>>  		}
>>>  	}
>>>
>>>  	win_align = pci_min_window_alignment(bus, b_res->flags);
>>>  	min_align = calculate_head_align(aligns, max_order);
>>>  	min_align = max(min_align, win_align);
>>> +	min_align = max(min_align, rebar_align);
>>>  	size0 = calculate_memsize(size, realloc_head ? 0 : add_size,
>>>  				  0, win_align);
>>>
>>> +	/*
>>> +	 * A window that reserves ReBAR headroom is larger than its own
>>> +	 * alignment (e.g. 32 GiB BAR + doorbell => 33 GiB > 32 GiB). The
>>> +	 * parent only reserves alignment padding for a child window when
>>> +	 * its size <= its alignment, so round such a window's alignment up
>>> +	 * to a power of two >= its size; sibling 32 GiB BARs behind one
>>> +	 * switch then each land in a properly aligned slot.
>>> +	 */
>>> +	if (rebar_align && size0)
>>> +		min_align = max(min_align, roundup_pow_of_two(size0));
>>> +
>>>  	if (size0) {
>>>  		resource_set_range(b_res, min_align, size0);
>>>  		b_res->flags &= ~IORESOURCE_DISABLED;
>>> @@ -2179,6 +2242,52 @@ static void pci_prepare_next_assign_round(struct
>>> list_head *fail_head,
>>>   * Second and later try will clear small leaf bridge res.
>>>   * Will stop till to the max depth if can not find good one.
>>>   */
>>> +/*
>>> + * Release the 64-bit prefetchable BARs of resizable-BAR display
>>> devices so the
>>> + * windows enclosing them empty out and can be re-sized. The BAR *size*
>>> is left
>>> + * untouched (only released and re-placed), so the hardware ReBAR is never
>>> + * programmed here - the driver still owns the actual resize.
>>> + */
>>> +static int pci_release_rebar_bars_cb(struct pci_dev *dev, void *data)
>>> +{
>>> +	struct resource *r;
>>> +	unsigned int i;
>>> +
>>> +	if ((dev->class >> 16) != PCI_BASE_CLASS_DISPLAY)
>>> +		return 0;
>>> +	if (pci_rebar_get_max_size(dev, 0) <= pci_rebar_get_current_size(dev, 0))
>>> +		return 0;
>>> +	pci_dev_for_each_resource(dev, r, i) {
>>> +		if (i >= PCI_BRIDGE_RESOURCES)
>>> +			break;
>>> +		if (r->parent && (r->flags & IORESOURCE_MEM_64) &&
>>> +		    (r->flags & IORESOURCE_PREFETCH))
>>> +			pci_release_resource(dev, i);
>>> +	}
>>> +	return 0;
>>> +}
>>> +
>>> +/*
>>> + * Release the now-empty prefetchable bridge windows bottom-up so the
>>> sizing
>>> + * pass re-sizes them (with pci_rebar_pref_reserve() reservation) to
>>> fit the
>>> + * BARs a driver will later grow.
>>> + */
>>> +static void pci_release_rebar_windows(struct pci_bus *bus)
>>> +{
>>> +	struct pci_dev *dev;
>>> +	struct resource *w;
>>> +
>>> +	list_for_each_entry(dev, &bus->devices, bus_list)
>>> +		if (dev->subordinate)
>>> +			pci_release_rebar_windows(dev->subordinate);
>>> +
>>> +	if (!bus->self)
>>> +		return;
>>> +	w = &bus->self->resource[PCI_BRIDGE_PREF_MEM_WINDOW];
>>> +	if (w->parent && (w->flags & IORESOURCE_MEM_64) && !w->child)
>>> +		pci_release_resource(bus->self, PCI_BRIDGE_PREF_MEM_WINDOW);
>>> +}
>>> +
>>>  void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>>>  {
>>>  	LIST_HEAD(realloc_head);
>>> @@ -2190,6 +2299,15 @@ void
>>> pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>>>  	int pci_try_num = 1;
>>>  	enum enable_type enable_local;
>>>
>>> +	/*
>>> +	 * Release resizable device BARs and their prefetchable windows so the
>>> +	 * sizing pass re-sizes those windows large enough
>>> (pci_rebar_pref_reserve)
>>> +	 * for the BARs a driver will later grow. Only released and re-placed
>>> - the
>>> +	 * BAR size is left untouched, so the hardware ReBAR is never programmed.
>>> +	 */
>>> +	pci_walk_bus(bus, pci_release_rebar_bars_cb, NULL);
>>> +	pci_release_rebar_windows(bus);
>>> +
>>>  	/* Don't realloc if asked to do so */
>>>  	enable_local = pci_realloc_detect(bus, pci_realloc_enable);
>>>  	if (pci_realloc_enabled(enable_local)) {
>>>
>>> base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Christian König]

On 8/31/26 18:58, Ilpo Järvinen wrote:
> On Mon, 31 Aug 2026, Christian König wrote:
...
>>> 2. Pack Oversized Windows (BAR Demotion): 
>>> The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR 
>>> to be assigned from the non-prefetchable window. By clearing the PREFETCH 
>>> flag on small control BARs during enumeration, they are placed below 4 GiB. 
>>> The prefetchable window is then cleanly sized to the large BAR alone.
>>
>> Interesting hack, shouldn't really matter for AMD GPUs but that is 
>> something more on the heuristic side. 
>>
>> A perfectly valid trick which is done by BIOS implementations but what 
>> the Linux PCI subsystem still hasn't learned are back to back 
>> allocations, e.g. something like this:
>>
>> BAR0 32 GiB of GPU #1
>> BAR1 2	MiB of GPU #1
>>
>> BAR1 2	MiB of GPU #2
>> BAR0 32 GiB of GPU #2
> 
> Assuming you're oversimplifying the layout above, this was added by the commit 
> 9036bd0efcb6 ("PCI: Align head space better").

Oh nice! Yeah we had quite a number of bug reports because of that.

Usually hot unplugging some PCIe HW and then re-plugging it caused surprises when the BIOS assigned the resources initially and then the Linux kernel on re-plug.

I need to dig up those bug reports and add a reference to the commit to let people retest. 

Thanks for that pointer,
Christian.

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Mario Limonciello]

On 8/31/26 12:52, Christian König wrote:
> 
> 
> On 8/31/26 18:58, Ilpo Järvinen wrote:
>> On Mon, 31 Aug 2026, Christian König wrote:
> ...
>>>> 2. Pack Oversized Windows (BAR Demotion):
>>>> The PCI-to-PCI Bridge spec (r1.2, sec 3.2.5) permits a prefetchable BAR
>>>> to be assigned from the non-prefetchable window. By clearing the PREFETCH
>>>> flag on small control BARs during enumeration, they are placed below 4 GiB.
>>>> The prefetchable window is then cleanly sized to the large BAR alone.
>>>
>>> Interesting hack, shouldn't really matter for AMD GPUs but that is
>>> something more on the heuristic side.
>>>
>>> A perfectly valid trick which is done by BIOS implementations but what
>>> the Linux PCI subsystem still hasn't learned are back to back
>>> allocations, e.g. something like this:
>>>
>>> BAR0 32 GiB of GPU #1
>>> BAR1 2	MiB of GPU #1
>>>
>>> BAR1 2	MiB of GPU #2
>>> BAR0 32 GiB of GPU #2
>>
>> Assuming you're oversimplifying the layout above, this was added by the commit
>> 9036bd0efcb6 ("PCI: Align head space better").
> 
> Oh nice! Yeah we had quite a number of bug reports because of that.
> 
> Usually hot unplugging some PCIe HW and then re-plugging it caused surprises when the BIOS assigned the resources initially and then the Linux kernel on re-plug.
> 
> I need to dig up those bug reports and add a reference to the commit to let people retest.
> 
That's actually "part" of why we have a behavior in thunderbolt.ko that 
we do a USB4 router reset at module load.

It gives you a clean break to let the resources get assigned by the kernel.

The other part is consistency with other OSes.

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Christian König]

On 8/31/26 19:49, Geramy Loveless wrote:
> On 8/31/26 9:58 AM, Ilpo Järvinen wrote:
>> On Mon, 31 Aug 2026, Christian König wrote:
>>
>>> On 8/28/26 23:37, Geramy Loveless wrote:
>>>> Firmware typically sizes prefetchable bridge windows for the boot-time
>>>> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
>>>> no room for a driver to grow a Resizable BAR afterwards: every window
>>>> from the leaf up to the root is sized for the small BAR, so
>>>> pci_resize_resource() fails with -ENOSPC. 
>>>>
>>>> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
>>>> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
>>>> VRAM BAR) pushes the required window size past the large BAR's alignment. 
>>>> Because bridge windows round up to a power of two, this forces a massive 
>>>> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
>>>> wasting ~32 GiB per GPU).
>>>>
>>>> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
>>>>
>>>> 1. Reserve Headroom: 
>>>> Reserve prefetchable window headroom for the maximum size of each 
>>>> downstream Resizable BAR during the bridge sizing pass. The device BAR 
>>>> and hardware ReBAR are left at their boot size to prevent tearing down 
>>>> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
>>> Yeah that was suggested before but that is clearly not something you can 
>>> do in common code.
>>>
>>> The ReBAR fields often doesn't reflect the actual needed space but 
>>> rather the maximum the HW address logic can resolve.
>>>
>>> So what you end up with is allocating multiple TiB for a window which 
>>> just needs few GiB, sometimes even completely overflowing the 64bit 
>>> address space made available by the root complex.
> 
> Yeah I could imagine that would be bad, hence I tried to compact the 64-bit space as well. As far as I understand PCI/PCIe standards its making this patch difficult.
> Christian I'm not sure if you could share these reference documents but that probably would be a better start for me to look at before I update the patch or make changes, also I need to wait for a review on the last patch I submitted too, so that leaves me with some time to review some standards if you know specifically where to look, if not thats fine too.
> 
>> Yes. A naive approach to (only) go to the max ReBAR allows just doesn't 
>> work well enough to be usable in general case.
> I did not know the max would report above the amount is actually needed because of vendors decisions in the cards, that's interesting.

Yeah as I said before to actually figure out how large your BAR needs to be for an AMD GPU you have to load quite a bit of FW and probe the connected RAM chips.

If the GPU is your initial display device that is not much of a problem because that happens quite early with the VBIOS load, but if you have non display HW the driver actually needs to do this ASIC init/reset.

The bus interface just gives out the possible BAR sizes which could potentially make sense on the board.

At least AMD has also produced quite a number of GPUs with bugs in the ReBAR registers, so for example Navi 1x reports possible BAR sizes of only between 1 - 4 MiB while it should be 8GiB at least.

I clearly don't recommend using that register as source for the resource estimation for upstream bridges.

Regards,
Christian.

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Mario Limonciello]

On 8/31/26 13:07, Christian König wrote:
> On 8/31/26 19:49, Geramy Loveless wrote:
>> On 8/31/26 9:58 AM, Ilpo Järvinen wrote:
>>> On Mon, 31 Aug 2026, Christian König wrote:
>>>
>>>> On 8/28/26 23:37, Geramy Loveless wrote:
>>>>> Firmware typically sizes prefetchable bridge windows for the boot-time
>>>>> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
>>>>> no room for a driver to grow a Resizable BAR afterwards: every window
>>>>> from the leaf up to the root is sized for the small BAR, so
>>>>> pci_resize_resource() fails with -ENOSPC.
>>>>>
>>>>> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a
>>>>> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB
>>>>> VRAM BAR) pushes the required window size past the large BAR's alignment.
>>>>> Because bridge windows round up to a power of two, this forces a massive
>>>>> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window,
>>>>> wasting ~32 GiB per GPU).
>>>>>
>>>>> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
>>>>>
>>>>> 1. Reserve Headroom:
>>>>> Reserve prefetchable window headroom for the maximum size of each
>>>>> downstream Resizable BAR during the bridge sizing pass. The device BAR
>>>>> and hardware ReBAR are left at their boot size to prevent tearing down
>>>>> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
>>>> Yeah that was suggested before but that is clearly not something you can
>>>> do in common code.
>>>>
>>>> The ReBAR fields often doesn't reflect the actual needed space but
>>>> rather the maximum the HW address logic can resolve.
>>>>
>>>> So what you end up with is allocating multiple TiB for a window which
>>>> just needs few GiB, sometimes even completely overflowing the 64bit
>>>> address space made available by the root complex.
>>
>> Yeah I could imagine that would be bad, hence I tried to compact the 64-bit space as well. As far as I understand PCI/PCIe standards its making this patch difficult.
>> Christian I'm not sure if you could share these reference documents but that probably would be a better start for me to look at before I update the patch or make changes, also I need to wait for a review on the last patch I submitted too, so that leaves me with some time to review some standards if you know specifically where to look, if not thats fine too.
>>
>>> Yes. A naive approach to (only) go to the max ReBAR allows just doesn't
>>> work well enough to be usable in general case.
>> I did not know the max would report above the amount is actually needed because of vendors decisions in the cards, that's interesting.
> 
> Yeah as I said before to actually figure out how large your BAR needs to be for an AMD GPU you have to load quite a bit of FW and probe the connected RAM chips.
> 
> If the GPU is your initial display device that is not much of a problem because that happens quite early with the VBIOS load, but if you have non display HW the driver actually needs to do this ASIC init/reset.
> 
> The bus interface just gives out the possible BAR sizes which could potentially make sense on the board.
> 
> At least AMD has also produced quite a number of GPUs with bugs in the ReBAR registers, so for example Navi 1x reports possible BAR sizes of only between 1 - 4 MiB while it should be 8GiB at least.
> 
> I clearly don't recommend using that register as source for the resource estimation for upstream bridges.
> 
> Regards,
> Christian.

Generally speaking besides eGPUs people don't "frequently" change PCIe 
topologies.

Another way to approach this could be to cache some of this information 
somewhere stateful the kernel can optionally get.  Something like this:

1. Add a new sysfs knob to "snapshot" the topology layout and the ideal 
size of specific devices connected.
2. This would be saved to an EFI variable.
3. During the next bootup the EFI variable can be fetched and compare if 
the topology layout changed.
4. If the topology is identical take into account the requested size for 
specific devices.
5. If the topology is different, ignore the EFI variable.

There would be a lot of discussion on the nuts and bolts of how to judge 
it's identical (is VID/DID/RID and BDF enough?).

Both at runtime and boot time I think it would have to exclude anything 
that deemed external too.

Thoughts on the idea otherwise?

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 4006 bytes --]

On Mon, 31 Aug 2026, Geramy Loveless wrote:
> On 8/31/26 9:58 AM, Ilpo Järvinen wrote:
> > On Mon, 31 Aug 2026, Christian König wrote:
> >
> >> On 8/28/26 23:37, Geramy Loveless wrote:
> >>> Firmware typically sizes prefetchable bridge windows for the boot-time
> >>> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
> >>> no room for a driver to grow a Resizable BAR afterwards: every window
> >>> from the leaf up to the root is sized for the small BAR, so
> >>> pci_resize_resource() fails with -ENOSPC. 
> >>>
> >>> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
> >>> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
> >>> VRAM BAR) pushes the required window size past the large BAR's alignment. 
> >>> Because bridge windows round up to a power of two, this forces a massive 
> >>> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
> >>> wasting ~32 GiB per GPU).
> >>>
> >>> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
> >>>
> >>> 1. Reserve Headroom: 
> >>> Reserve prefetchable window headroom for the maximum size of each 
> >>> downstream Resizable BAR during the bridge sizing pass. The device BAR 
> >>> and hardware ReBAR are left at their boot size to prevent tearing down 
> >>> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
> >> Yeah that was suggested before but that is clearly not something you can 
> >> do in common code.
> >>
> >> The ReBAR fields often doesn't reflect the actual needed space but 
> >> rather the maximum the HW address logic can resolve.
> >>
> >> So what you end up with is allocating multiple TiB for a window which 
> >> just needs few GiB, sometimes even completely overflowing the 64bit 
> >> address space made available by the root complex.
> 
> Yeah I could imagine that would be bad, hence I tried to compact the 
> 64-bit space as well. As far as I understand PCI/PCIe standards its 
> making this patch difficult. 

No compacting will save you in some cases. There are setups with many GPUs 
and with VF BARs, the space requirement to fit full-sized ReBARs is way 
beyond what the root bus resource can hold. Where's your fallback for such 
cases? What are the failure modes of that fallback?

> Christian I'm not sure if you could share these reference documents but 
> that probably would be a better start for me to look at before I update 
> the patch or make changes, also I need to wait for a review on the last 
> patch I submitted too, so that leaves me with some time to review some 
> standards if you know specifically where to look, if not thats fine too.
> 
> > Yes. A naive approach to (only) go to the max ReBAR allows just doesn't 
> > work well enough to be usable in general case.
>
> I did not know the max would report above the amount is actually needed 
> because of vendors decisions in the cards, that's interesting. 

Not just that, using too large size has ripple effects on other resources 
in the system, and those failures are hard to correct (your code doesn't 
even try to do that).

> > A general algorithm to pick the largests fitting sizes for ReBARs is still 
> > under works, a few steps away still, and more if I discover more 
> > challenges I need address before I can take that final step.
>
> So then I would assume instead of doing the "hack" i'm having to use I 
> would change it to your new algorithm?

Assuming you meant we apply this kind of naive approach first and later 
changing to the new algorithm, the problem with that approach is this
"hack" must not break working setups either, but it surely would. If it 
does break things, the change will just end up reverted. And that end 
result is same as not doing the "hack" at all.

It's not enough it doesn't break your setup. It generally must not break 
any setup, which is unfortunately very very high bar.


-- 
 i.

Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 5828 bytes --]

On Mon, 31 Aug 2026, Geramy Loveless wrote:

> 
> On 8/31/26 9:31 AM, Ilpo Järvinen wrote:
> > On Fri, 28 Aug 2026, Geramy Loveless wrote:
> >
> >> Firmware typically sizes prefetchable bridge windows for the boot-time
> >> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
> >> no room for a driver to grow a Resizable BAR afterwards: every window
> >> from the leaf up to the root is sized for the small BAR, so
> >> pci_resize_resource() fails with -ENOSPC. 
> >>
> >> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
> >> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
> >> VRAM BAR) pushes the required window size past the large BAR's alignment. 
> >> Because bridge windows round up to a power of two,
> > Do they??? In which kernel version I must ask??
> I'm sorry this was in reference to a prior patch I made, **ignore**
> >
> >> this forces a massive 
> >> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
> >> wasting ~32 GiB per GPU).
> > The largest I've seen is 32GB + half of that. And that's not with the 
> > latest kernel.
> 
> This is in reference to when there are two R9700s on one switch basically.
> As far as my understanding currently how it works is that when there are two GPUs they both have a BAR0, BAR2, and BAR5 the BAR5 goes onto the 32-bit window.
> Bar 0 and 2 don't go onto the second window by default as they could overfill it of course, bar 0 would not fit period, and 2 might which is the case I took. 
> This allows BAR 0 on (GPU 0) in leaf 0 on window 0 to align as a multiple of 32G like it I guess has to? This is how I understand it, so if R9700 has 32G on, bar0, leaf 0, window 0, and share the same window as GPU 1 then we have to pad the space between GPU0 BARs on window 0 by 32G, meaning BAR2 really is also using 32G, its not possible or at least right now its not possible to move BAR2 to the end of all the allocations it doesnt fit the standards? Keep in mind this is not my specialty here I am trying my best to understand and solve the issue at hand.

With the empty space in between, your 64GB calculation would indeed be 
true, but that wasn't what you said because you made an inaccurate claim 
about window sizes being power of two in case of 32G+2M.

(I suppose we're generally in agreement here but just ended up discussion 
on finer details and the interpretation of those details.)

> >> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
> >>
> >> 1. Reserve Headroom: 
> >> Reserve prefetchable window headroom for the maximum size of each 
> >> downstream Resizable BAR during the bridge sizing pass. The device BAR 
> >> and hardware ReBAR are left at their boot size to prevent tearing down 
> >> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
> > ... reorganizing ...
> >
> >> Fallback / Alignment logic:
> >> If the 32-bit space is exhausted, small BARs remain in the prefetchable 
> >> window. The alignment logic gracefully handles this by rounding the bridge 
> >> window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
> >> BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
> >> properly aligned slots.
> >>
> >> Reserving is safe by default: __assign_resources_sorted() satisfies all
> >> required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
> >> to restore previous behaviours.
> >
> > And how does this fallback, if the largest size doesn't fit? What are the 
> > failure modes?
>
> So if the smaller bars under 16M don't fit in the 32-bit addressable 
> space the fallback mode is to do as I said above to your prior comment, 
> basically put. 
> Don't move the 2M BAR2 over to the 32-bit addressable space / window and 
> instead fallback to aligning the prior window by the size of the GPUs 
> memory space as I understand it, when you have gpu0 and gpu1 both 
> requiring 32GB their address has to be a multiple of their allocation 
> size? meaning we get 32G for BAR0 and 32G for BAR2 even though BAR2 only 
> needs 2M because its not aligned to allow GPU1 to be on a aligned space?

I'm more referring to the case where the big ReBARs cannot be assigned as 
space runs out or some other BAR fails to assign because ReBAR took "too 
much", and the numerous ripple effects from that. When that happens in 
some system for something important breaking something, you'd have to 
solve that problem too or this change gets reverted...

> > It's correct that the sizing decision should be done in pbus_size_mem() 
> > but it must not result in breaking things which a naive approach like 
> > this will surely do.
> >
> > Without proper fallbacks done, there will be too much breakage it's a 
> > showstopper for a naive patch like this.
> >
> > Basically, what we'd want to do is try with the largest size, then reduce 
> > it one step at a time (because it's 2^n -> 2^(n-1) even one step saves 
> > quite much so one might not need to reduce that many steps). But also, in 
> > some IOV enabled cases the oversubscription of the space is very 
> > substantial and you'd still want to maximize the BAR sizes to what is 
> > allowed by the available space.
> >
> > Fallback is hard to implement, because sizing and assignment are quite far 
> > from each other codewise and there are no structures to carry information 
> > over to the next retry phase.

...And that's why there should be proper fallback so that failing one 
assignment somewhere middle will not result in failure in the end.

Basically, my own plan is for the algorithm to try smaller sizes until all 
things fit. And all that retry logic is far from simple.

-- 
 i.