On Mon, 21 Sep 2026, Grochowski, Maciej wrote: > Hi Ilpo, Nikolas, > > I have similar instance of what appears to be the same > multi-composite-resource case. > > The simplified topology is: > >     AST2720 root complex (0001:00:00.0) >       `-- PM50052 Switchtec (0001:01:00.0) >             `-- eight downstream bridges >                   `-- endpoints 0001:03:00.0 ... 0001:0a:00.0 > > Each endpoint has a 16 MiB BAR2 and a 4 MiB BAR0, both 64-bit > prefetchable. BAR4 is suppressed by platform policy. Each child bridge > window is therefore 20 MiB with 16 MiB alignment. > > pbus_size_mem() accounts for the eight children as 160 MiB. Their > observed placement uses 32 MiB strides, so the occupied span is 244 MiB. > Five endpoints receive their BARs; the remaining three do not. The host > has a 4 GiB prefetchable aperture. > > I currently carry the max() -> ALIGN() change locally. It grows the > parent window to 256 MiB and restores all eight endpoints. We use it only > to unblock this fixed topology and agree that it is not a general > solution. > > Ilpo, if you have any work-in-progress code for this, I would be happy > to test it on this N>=8 topology and return allocator traces, lspci output.  > I would much rather help validate the general fix > than carry the local max() -> ALIGN() workaround into deployment. Hi Maciej, Could you please try this series: https://lore.kernel.org/linux-pci/20261002113319.6652-1-ilpo.jarvinen@linux.intel.com/ -- i. > Regards, > Maciej > > From: Ilpo Järvinen > Date: Thursday, September 3, 2026 at 4:54 AM > To: Nikolas Joshua Britton > Cc: linux-pci@vger.kernel.org ; Bjorn Helgaas > ; regressions@lists.linux.dev ; > amd-gfx@lists.freedesktop.org > Subject: Re: PCI: bridge window undersized when child bridge windows are larger than > their alignment (regression since 3958bf16e2fe, v7.0) > > On Thu, 3 Sep 2026, Nikolas Joshua Britton wrote: > > > Hi, > > > > On a Mac Pro 7,1 with two Radeon Pro Vega II Duo cards, enabling 32 GB > > Resizable BARs leaves exactly half the GPU dies with no BAR at all. The > > shared root-port prefetchable window is sized as the plain sum of its two > > child bridge windows, but each child secretly requires a 32 GB-aligned > > start, so the window that gets allocated is ~32 GB smaller than the span > > that is actually needed. The second child of each pair loses, > > deterministically. > > > > There is 1 TiB of free space in the host bridge's _CRS window, so this is > > not address-space exhaustion. > > > > This is a regression. On the same machine, with the same script > > (resize-amdgpu-bars,https://urldefense.com/v3/__https://github.com/exabit-io/resize-amdgpu-bars__;!!HUFU > Ugx-IQ7VcAu3Ktk!DSmzn4BJhDHHjgz9X64_VKu7J2iJYBrJlaenlhDoLhvZyShd55JpgyzFkjP5BluDsdC > Nhw0gZivtlTPuwP6UyFZd2FbvPT1fdA$ , > > which performs the sequence under "Reproduction" below at boot) and the > > same sequence of operations, Ubuntu's 6.8.0-138 (upstream 6.8.12), > > 6.11.0-29, 6.14.0-37 and 6.17.0-42 kernels all size the shared window at > > 96 GiB (48 GiB per child) and all four dies get their 32 GiB BAR on the > > first attempt, every time (each kernel tested from a full power-off). > > 7.0.0-30 is based on upstream 7.0.12 and fails identically from a cold > > boot and from a warm reboot. I have traced it to commit 3958bf16e2fe > > ("PCI: Stop over-estimating bridge window size"), first shipped in v7.0; > > analysis and a proposed one-line fix are below. Ubuntu's setup-bus.c is > > byte-identical to v7.0.12, which already includes 8cb081667377 ("PCI: > > Fix alignment calculation for resource size larger than align") and > > dc4b4d04e1ca ("PCI: Prevent shrinking bridge window from its required > > size"), so those do not cover this case; current master has no further > > change to this logic. > > > > > > System > > ------ > > > >   Machine:  Apple Inc. MacPro7,1, BIOS 2103.160.2.0.0 > >   Fails:    7.0.0-30-generic (Ubuntu 24.04 HWE, 7.0.0-30.30~24.04.1, > >             upstream 7.0.12) > >             self-built upstream v7.0.12, unpatched, Ubuntu config trimmed > >             with localmodconfig (control; fails identically) > >   Works:    6.8.0-138-generic  (Ubuntu 24.04 GA, upstream 6.8.12) > >             6.11.0-29-generic  (Ubuntu 24.04, linux-generic-6.11) > >             6.14.0-37-generic  (Ubuntu 24.04, linux-generic-6.14) > >             6.17.0-42-generic  (Ubuntu 24.04, linux-generic-6.17) > >             (all four: identical 96 GiB window layout, 4/4 dies, one > >             4-node XGMI hive, no traces; verified 2026-09-02) > >             self-built v7.0.12 + the patch below (same config as the > >             control): 4/4 dies, 128 GiB root-port window, cold boot and > >             warm reboot, no traces > >   Cmdline:  ro log_buf_len=16M pci=realloc mitigations=off > >             (the 6.x and the self-built 7.0.12 boots also carried > >             intremap=no_x2apic_optout; it has no bearing on this) > >   GPUs:     4x Vega20 [1002:66a3], two dies per Vega II Duo card > >   BAR0 ReBAR capability: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB > >   BAR2 (doorbell): 2MB, fixed in practice > > > > > > Topology > > -------- > > > > Each Duo card presents two dies behind ONE root port, each die on its own > > sub-bridge chain: > > > >  +-[0000:06]-+-00.0-[07-0e]--00.0-[08-0e]--+-08.0-[09-0b]--00.0-[0a-0b]--00.0-[0b]--0 > 0.0  Vega20 > >              |                              > \-10.0-[0c-0e]--00.0-[0d-0e]--00.0-[0e]--00.0  Vega20 > >  +-[0000:16]-+-00.0-[17-1e]--00.0-[18-1e]--+-08.0-[19-1b]--00.0-[1a-1b]--00.0-[1b]--0 > 0.0  Vega20 > >                                             > \-10.0-[1c-1e]--00.0-[1d-1e]--00.0-[1e]--00.0  Vega20 > > > > So 08:08.0 and 08:10.0 are siblings sharing the prefetchable window of > > 07:00.0 / 06:00.0. Each subtree contains one Vega20 with BAR0 = 32 GB > > (alignment 32 GB) plus BAR2 = 2 MB, i.e. each child bridge window is > > 32 GB + 2 MB = 0x800200000. > > > > Both cards fail identically. Card 2 (16:00.0 / 18:08.0 / 18:10.0) is > > omitted below for brevity; its trace is byte-for-byte analogous. > > > > > > The arithmetic > > -------------- > > > >   host bridge _CRS window   0x90000000000-0x9ffffffffff   1 TiB free > >   06:00.0 / 07:00.0 window  0x90000000000-0x910003fffff   0x1000400000 > >                                                           (64 GiB + 4 MiB) > > > >   08:08.0 window assigned   0x90000000000-0x908001fffff   0x800200000 > >   next free address                       0x90800200000 > >   08:10.0 needs 32 GiB alignment, so its next legal start is > >                                           0x91000000000 > >   08:10.0 would then end at               0x91800200000 > >   but the parent window ends at           0x91000400000  <-- ~32 GiB short > > > >   span actually required    0x90000000000-0x91800200000   0x1800200000 > >                                                           (96 GiB + 2 MiB) > >   span allocated                                          0x1000400000 > >                                                           (64 GiB + 4 MiB) > >   shortfall                                               0x7ffe00000 > >                                                           (32 GiB - 2 MiB) > > > > The telling detail: 0x90800200000 + 0x800200000 = 0x91000400000, which is > > *exactly* the parent window's exclusive end. In other words, had the child > > bridge windows only needed ~1 MiB alignment, the two of them would have > > fit perfectly, to the byte. The sizing pass produced a window that is > > correct if and only if the children can be packed back-to-back, which > > they cannot, because the assignment pass then enforces the real 32 GiB > > alignment inherited from the BAR inside each child. > > > > > > Good kernel, for comparison > > --------------------------- > > > > Same hardware, same steps, 6.17.0-42 (6.8, 6.11 and 6.14 are identical). > > The parent is sized for the worst-case packing, and both children fit: > > > >   0000:06:00.0: 90000000000-917ffffffff [size=96G]      <- shared parent > >   0000:08:08.0: 90000000000-90bffffffff [size=48G] > >   0000:08:10.0: 90c00000000-917ffffffff [size=48G] > >   0000:0b:00.0: Region 0: Memory at 90000000000 (64-bit, prefetchable) [size=32G] > >   0000:0e:00.0: Region 0: Memory at 91000000000 (64-bit, prefetchable) [size=32G] > > > > i.e. 96 GiB = 3 x 32 GiB: each child gets (32 GiB + 2 MiB) rounded up to > > the next 32 GiB boundary plus slack, so the second child's aligned start > > is always inside the parent. That is the 3A + eps span from the arithmetic > > below, and 7.0 allocates 2A + 2 eps instead. > > > > > > dmesg (7.0) > > ----------- > > > > Sizing and assignment of the shared window, then the two children: > > > >   pci 0000:06:00.0: bridge window [mem 0x90000000000-0x910003fffff 64bit pref]: > assigned > >   pci 0000:07:00.0: bridge window [mem 0x90000000000-0x910003fffff 64bit pref]: > assigned > >   pci 0000:08:08.0: bridge window [mem 0x90000000000-0x908001fffff 64bit pref]: > assigned > >   pci 0000:08:10.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; > no space > >   pci 0000:08:10.0: bridge window [mem size 0x800200000 64bit pref]: failed to > assign > > > > The failure then cascades down the losing chain, and the endpoint is left > > with neither BAR0 nor BAR2: > > > >   pci 0000:0c:00.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; > no space > >   pci 0000:0d:00.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; > no space > >   pci 0000:0e:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no space > >   pci 0000:0e:00.0: BAR 0 [mem size 0x800000000 64bit pref]: failed to assign > >   pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't assign; no space > >   pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: failed to assign > >   pci 0000:0e:00.0: BAR 5 [mem 0x74600000-0x7467ffff]: assigned > > > > Note BAR5 (the 512 KB register aperture, non-prefetchable) still gets > > assigned. That matters for the downstream impact described below: the > > device is half-alive rather than obviously dead. > > > > Resulting state, from lspci -vv. The entire losing chain has no > > prefetchable window whatsoever: > > > >   0000:08:08.0: 90000000000-908001fffff [size=32770M]    <- winner > >   0000:09:00.0: 90000000000-908001fffff [size=32770M] > >   0000:0a:00.0: 90000000000-908001fffff [size=32770M] > >   0000:08:10.0: [disabled]                               <- loser > >   0000:0c:00.0: [disabled] > >   0000:0d:00.0: [disabled] > >   0000:06:00.0: 90000000000-910003fffff [size=65540M]    <- shared parent > > > > And /sys/bus/pci/devices/0000:0e:00.0/resource: > > > >   0x0000000000000000 0x0000000000000000 0x0000000000000000   BAR0 unassigned > >   0x0000000000000000 0x0000000000000000 0x0000000000000000 > >   0x0000000000000000 0x0000000000000000 0x0000000000000000   BAR2 unassigned > >   0x0000000000000000 0x0000000000000000 0x0000000000000000 > >   0x0000000000006000 0x00000000000060ff 0x0000000000040101 > >   0x0000000074600000 0x000000007467ffff 0x0000000000040200   BAR5 assigned > > > > > > Observation > > ----------- > > > > Observed, and I think not in dispute: > > > >   - the sizing pass produced a parent window exactly equal to the sum of > >     the two child window sizes (0x800200000 * 2 = 0x1000400000); > >   - the assignment pass refused to place the second child at > >     0x90800200000, which is 2 MiB-aligned and would have fit exactly; > >   - therefore assignment enforced an alignment that sizing did not budget > >     for. > > > > Where in the code: pbus_size_mem() sums the child bridge windows > > without regard to the alignment the assignment pass will enforce on > > them. The analysis, a small model that reproduces every number above, > > and a one-line fix that has been A/B tested on this machine are in > > "Root cause" and "Proposed fix" below, before the list of gaps. > > > > The shortfall is structural rather than specific to 32 GB. For N=2 > > siblings each needing (A + eps) at alignment A, the required span is > > 2A + (A + eps) = 3A + eps, while the sum is 2A + 2eps. I have confirmed > > this at the other end of the range: after one failed 32 GiB attempt, > > writing the ReBAR index back to 256 MiB and re-enumerating fails in > > exactly the same way, because the firmware's original 770 MiB windows > > (~3A + eps for A = 256 MiB) are gone and the kernel re-sizes the parent > > to 2A + 2 eps: > > > >   pci 0000:06:00.0: bridge window [mem 0x90000000000-0x900203fffff 64bit pref]: > assigned > >                                               (512 MiB + 4 MiB) > >   pci 0000:08:08.0: bridge window [mem 0x90000000000-0x900101fffff 64bit pref]: > assigned > >   pci 0000:08:10.0: bridge window [mem size 0x10200000 64bit pref]: can't assign; > no space > >   pci 0000:0e:00.0: BAR 0 [mem size 0x10000000 64bit pref]: can't assign; no space > >   pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't assign; no space > > > > So on 7.0 there is no in-place recovery once the kernel has re-sized the > > window: the only layouts that ever work are the ones the firmware left > > behind. 16 GB and 8 GB remain untested but I would not expect them to > > differ. > > > > > > Downstream impact: this is not a soft failure > > --------------------------------------------- > > > > The BAR-less die is not merely unusable. Because BAR5 is still assigned, > > amdgpu probes it, its register reads return garbage, and > > RCC_IOV_FUNC_IDENTIFIER comes back with bit 0 set. The driver concludes > > the device is an SR-IOV *virtual function*: > > > >   amdgpu 0000:0b:00.0: register mmio base: 0x74400000     <- healthy die > >   amdgpu 0000:0e:00.0: register mmio base: 0x74600000 > >   amdgpu 0000:0e:00.0: MCBP is enabled                    <- only set when > >                                                              amdgpu_sriov_vf() > > > > It then calls amdgpu_virt_request_full_gpu() -> > > xgpu_ai_request_full_gpu_access() and waits forever for a hypervisor > > mailbox that does not exist: > > > >   amdgpu 0000:0e:00.0: trn=2 ACK should not assert! wait again ! > >   (repeating roughly 2490 times per 5 seconds, indefinitely) > > > >   xgpu_ai_mailbox_trans_msg+0x1a9/0x1f0 [amdgpu] > >   xgpu_ai_send_access_requests+0x21/0xe0 [amdgpu] > >   xgpu_ai_request_full_gpu_access+0x1a/0x30 [amdgpu] > >   amdgpu_virt_request_full_gpu+0x2a/0x70 [amdgpu] > >   amdgpu_device_ip_early_init.constprop.0+0x173/0x780 [amdgpu] > >   amdgpu_device_init+0x83d/0x1180 [amdgpu] > >   amdgpu_driver_load_kms+0x1a/0xd0 [amdgpu] > >   amdgpu_pci_probe+0x1df/0x590 [amdgpu] > > > > modprobe wedges in uninterruptible D state holding the device mutex, > > blocks the AER IRQ thread, and never returns: > > > >   INFO: task irq/34-aerdrv:1554 blocked for more than 122 seconds. > >   INFO: task irq/34-aerdrv:1554 is blocked on a mutex likely owned by > >         task modprobe:1583. > > > > SIGKILL does not touch it; systemd's TimeoutStartSec fires and the unit > > goes to "failed" while the task stays in the cgroup. The machine needs a > > reboot. The remaining two dies are never probed at all. > > > > So the practical outcome of the sizing bug with a plain "modprobe amdgpu" > > is: 1 of 4 GPUs usable (the healthy die of the first card; its BAR-less > > sibling wedges the probe and the second card is never reached), no XGMI > > hive (kfd reports a single-node hive), and an unkillable task on every > > boot. Keeping the BAR-less dies away from the driver with > > driver_override, which resize-amdgpu-bars now does, gets 2 of 4 dies and > > a 2-node hive; that is the "2/4 dies" figure under "Proposed fix" below. > > Whether amdgpu should be more defensive about probing a device with > > an unassigned BAR0 is a separate question for amd-gfx, and I've cc'd them, > > but the PCI-side undersizing is the trigger. > > > > > > Reproduction > > ------------ > > > > With the four dies at their default 256 MB BAR0 (this is what > > resize-amdgpu-bars does at boot, done by hand): > > > >   1. Set BAR0 to 32 GB on all four dies (ReBAR control register at > >      capability offset 0x200, control at 0x208, size index in bits 8-13, > >      index 15 = 2^35): > > > >        setpci -s 0000:0b:00.0 0x208.l=00000f40 > >        (likewise for 0e:00.0, 1b:00.0, 1e:00.0) > > > >   2. Force full re-enumeration so the kernel re-sizes every bridge window > >      from scratch: > > > >        echo 1 > /sys/bus/pci/devices/0000:06:00.0/remove > >        echo 1 > /sys/bus/pci/devices/0000:16:00.0/remove > >        echo 1 > /sys/bus/pci/rescan > > > >   3. dmesg shows the "can't assign; no space" trace above; 0e:00.0 and > >      1e:00.0 have BAR0 and BAR2 unassigned, 0b:00.0 and 1b:00.0 are fine. > > > > Fully deterministic across many attempts: the first-enumerated die of > > each card always wins. > > > > > > Root cause > > ---------- > > > > Since 3958bf16e2fe, pbus_size_mem() sizes a bridge window as > > > >     size += max(r_size, align);        /* per child */ > >     size0 = ALIGN(size, win_align);    /* win_align = 1 MB */ > > > > with calculate_head_align() making only the window *start* satisfy the > > largest child alignment. The tight fit is gap-free only if, in the > > descending-alignment assignment order, every child's size is a multiple > > of the alignments of the children placed after it. That holds for BARs > > (size == alignment) but not for bridge windows: their size is the sum of > > what is below them, while their alignment (IORESOURCE_STARTALIGN) is that > > of the largest BAR below them. > > > > Here each sub-bridge window holds BAR0 (32 GiB) + BAR2 (2 MiB doorbell), > > so it is sized 32 GiB + 2 MiB with 32 GiB alignment. The root port sums > > two of them: 64 GiB + 4 MiB. Assignment then places the first child at > > offset 0 (ends at 32 GiB + 2 MiB) and must put the second at the next > > 32 GiB boundary, i.e. 64 GiB .. 96 GiB + 2 MiB. Needed 96 GiB + 2 MiB, > > sized 64 GiB + 4 MiB -> "can't assign; no space" for the second die. > > > >     pci 0000:06:00.0: bridge window [mem 0x90000000000-0x910003fffff 64bit pref]: > assigned > >     pci 0000:08:08.0: bridge window [mem 0x90000000000-0x908001fffff 64bit pref]: > assigned > >     pci 0000:0b:00.0: BAR 0 [mem 0x90000000000-0x907ffffffff 64bit pref]: assigned > >     pci 0000:0b:00.0: BAR 2 [mem 0x90800000000-0x908001fffff 64bit pref]: assigned > >     pci 0000:0e:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no > space > >     pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't assign; no > space > > > > The same arithmetic at the default 256 MiB BAR0 gives 512 MiB + 4 MiB > > sized vs 768 MiB + 2 MiB needed, which is why even reverting the BAR > > size does not recover (the firmware's original windows were larger). > > > > Up to v6.17, calculate_memsize() rounded each bridge window up to its own > > min_align (ALIGN(size, min_align); the old calculate_mem_align() gave > > 16 GiB here), so each sub-bridge window was 48 GiB and the root port > > 96 GiB, and siblings packed by accident. A small model of both versions > > of pbus_size_mem() reproduces every number seen on this machine > > (48G/96G on 6.8-6.17; 32770M/65540M and 258M/516M on 7.0). > > > > This is the tail-side sibling of the head-side under-estimation Guenter > > Roeck raised on 2026-03-05 for the same series (4M@4M + 3M@1M + 1M@1M > > needing 9 MiB in an 8 MiB window), which 8cb081667377 addressed for the > > head alignment bookkeeping only. > > > > Proposed fix > > ------------ > > > > Pad a child up to its alignment when its size is not a multiple of it: > > > >     -                 size += max(r_size, align); > >     +                 size += ALIGN(r_size, align); > > > > This is a no-op for BARs, so the v7.0 tight fit for leaf resources is > > kept; for bridge windows it restores the pre-v7.0 parent sizing. It > > over-estimates by up to one alignment unit (128 GiB here rather than the > > exact 96 GiB + 2 MiB); an exact version would have to walk the children > > in assignment order and simulate the offsets. The diff follows below the > > "---" line at the end of this mail; the full patch with changelog and > > Signed-off-by is ready and I will send it as a separate [PATCH] if that > > is preferred. > > > > Tested on the same machine, same script, same sequence (2026-09-02): > > > >   v7.0.12 unpatched (control): identical to 7.0.0-30. Root-port window > >     0x90000000000-0x900203fffff (512 MiB + 4 MiB at the 256 MiB baseline > >     the kernel falls back to), "can't assign; no space" on 0e:00.0 and > >     1e:00.0, 2/4 dies, 2-node XGMI hive. > >   v7.0.12 + patch: 4/4 dies with 32 GiB BAR0, 4-node XGMI hive, from a > >     cold boot and again from a warm reboot. Window layout for card 1: > > > >       pci 0000:06:00.0: bridge window [mem 0x90000000000-0x91fffffffff 64bit pref]: > assigned   (128 GiB) > >       pci 0000:07:00.0: bridge window [mem 0x90000000000-0x91fffffffff 64bit pref]: > assigned > >       pci 0000:08:08.0: bridge window [mem 0x90000000000-0x90fffffffff 64bit pref]: > assigned   (64 GiB) > >       pci 0000:09:00.0: bridge window [mem 0x90000000000-0x90fffffffff 64bit pref]: > assigned > >       pci 0000:0a:00.0: bridge window [mem 0x90000000000-0x908001fffff 64bit pref]: > assigned   (32 GiB + 2 MiB) > >       pci 0000:0b:00.0: BAR 0 [mem 0x90000000000-0x907ffffffff 64bit pref]: > assigned > >       pci 0000:0b:00.0: BAR 2 [mem 0x90800000000-0x908001fffff 64bit pref]: > assigned > >       pci 0000:08:10.0: bridge window [mem 0x91000000000-0x91fffffffff 64bit pref]: > assigned > >       pci 0000:0e:00.0: BAR 0 [mem 0x91000000000-0x917ffffffff 64bit pref]: > assigned > >       pci 0000:0e:00.0: BAR 2 [mem 0x91800000000-0x918001fffff 64bit pref]: > assigned > > > >     The padding is applied once, where 09:00.0 sums its 32 GiB + 2 MiB > >     child (alignment 32 GiB) to 64 GiB; the levels above are plain sums > >     of already-aligned children. Net cost 128 GiB per card instead of > >     the exact 96 GiB + 2 MiB, out of 1 TiB available. > > > > > > Why the repro does not use the sysfs resource0_resize interface > > ---------------------------------------------------------------- > > > > The repro above pokes the ReBAR control register with setpci and then > > forces a rescan instead of using the sanctioned interface > > (echo 15 > .../resource0_resize). That is deliberate: the sanctioned > > interface cannot grow a die that sits behind the card's own PCIe switch, > > on any kernel, and that is the reason resize-amdgpu-bars exists at all. > > It was the first thing I tried, and I re-measured it on 7.0.12 for this > > report so the failure is on record with the kernel's own lines. > > > >     Measured (7.0.12 vanilla, unit masked, amdgpu blacklisted, so all > >     four dies sat at the firmware 256 MB and nothing was bound; the > >     sibling die's BARs stay assigned whether or not a driver is bound): > > > >       # echo 15 > /sys/bus/pci/devices/0000:0b:00.0/resource0_resize > >       write error: No space left on device            (-ENOSPC) > > > >       pci 0000:0b:00.0: BAR 0 [mem 0x9ffe0000000-0x9ffefffffff 64bit pref]: > releasing > >       pcieport 0000:0a:00.0: bridge window [mem 0x9ffe0000000-0x9fff01fffff 64bit > pref]: releasing > >       pcieport 0000:09:00.0: bridge window [mem 0x9ffe0000000-0x9fff01fffff 64bit > pref]: releasing > >       pcieport 0000:08:08.0: bridge window [mem 0x9ffe0000000-0x9fff01fffff 64bit > pref]: releasing > >       pcieport 0000:07:00.0: bridge window [mem 0x9ffc0000000-0x9fff01fffff 64bit > pref]: was not released (still contains assigned resources) > >       pcieport 0000:06:00.0: bridge window [mem 0x9ffc0000000-0x9fff01fffff 64bit > pref]: was not released (still contains assigned resources) > >       pcieport 0000:08:08.0: bridge window [mem size 0x800200000 64bit pref]: can't > assign; no space > >       pci 0000:0b:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no > space > >       pci 0000:0b:00.0: BAR 0 [mem 0x9ffe0000000-0x9ffefffffff 64bit pref]: old > value restored > > > >     Parent window (06:00.0 and 07:00.0, shared by both dies) before and > >     after: 0x9ffc0000000-0x9fff01fffff, 770 MB, unchanged. Writing 8 back > >     returned 0 and every window came back byte-identical. The whole thing > >     took under half a second and nothing hung. > > > >     So the sysfs path fails the same way for the user (-ENOSPC, "old > >     value restored", BAR stays 256 MB) but for a reason one level below > >     the sizing bug: the shared window on the two bridges above the switch > >     is never released while the sibling die's BARs are assigned in it, so > >     the three bridge windows on the way down have to grow to 32 GB + 2 MB > >     inside a 770 MB parent, and cannot. The in-place path never gets to > >     re-size the shared window at all; it is limited to what fits in the > >     firmware layout. That is independent of this bug (it would fail the > >     same way on a fixed kernel and on 6.17), and it is why my tool grows a > >     dual-die module by removing the whole module and rescanning from the > >     root port instead. The sysfs write is not a workaround here and does > >     not exercise the sizing path this patch fixes; the setpci + rescan > >     repro and amdgpu's own resize at probe (below) are the two routes that > >     reach it, and both fail with "can't assign; no space" on unpatched 7.0. > > > > > > What I have not tested > > ---------------------- > > > > I want to be straight about the gaps: > > > >   - I have not tested 16 GB or 8 GB BARs (see the structural argument > >     above; 256 MB and 32 GB are the two data points). > > > >   - amdgpu's own resize at probe time (amdgpu_device_resize_fb_bar -> > >     pci_resize_resource) hits the same wall on 7.0. From an earlier boot > >     where amdgpu was allowed to autoload with the dies still at 256 MB: > > > >       amdgpu 0000:1b:00.0: BAR 2 [mem 0xbfff0000000-0xbfff01fffff 64bit pref]: old > value restored > >       amdgpu 0000:1b:00.0: BAR 0 [mem 0xbffe0000000-0xbffefffffff 64bit pref]: old > value restored > >       amdgpu 0000:1b:00.0: Not enough PCI address space for a large BAR. > >       amdgpu 0000:1b:00.0: [drm] Detected VRAM RAM=32752M, BAR=256M > > > >     That path fails closed (ReBAR index reverted, driver continues at > >     256 MB), so it does not trigger the hang described above, but it > >     does not get a large BAR either. > > > >   - I have not bisected by booting: the commit was identified by reading > >     setup-bus.c across the versions and confirmed by the A/B test of > >     v7.0.12 with and without the one-line patch (see "Proposed fix"). If > >     a real bisection would still help I can run it. > > > > Happy to test patches, gather more traces, or run with any debug options > > that would help. The machine is otherwise idle and I can reboot it freely. > > > > > > Available on request (not attached; the largest is 1.1 MB): > >   - full dmesg from the failing boot > >   - lspci -vvnn and lspci -tvnn > >   - the trimmed allocation trace for card 1 > >   - lspci -vv from the 6.17 boot (working layout) > >   - the sizing model (pci-window-sim.py, 130 lines) > >   - full dmesg from the v7.0.12 control boot and the v7.0.12 + patch boot > >   - lspci -vvnn from the v7.0.12 + patch boot > > > > > > #regzbot introduced: 3958bf16e2fe > > > > --- > >  drivers/pci/setup-bus.c | 12 +++++++++++- > >  1 file changed, 11 insertions(+), 1 deletion(-) > > > > --- a/drivers/pci/setup-bus.c > > +++ b/drivers/pci/setup-bus.c > > @@ -1329,7 +1329,17 @@ static void pbus_size_mem(struct pci_bus *bus, struct > resource *b_res, > >                                continue; > >  > >                        r_size = resource_size(r); > > -                     size += max(r_size, align); > > +                     /* > > +                      * Resources are assigned in descending alignment > > +                      * order, so a tight-fit sum is only gap-free if each > > +                      * size is a multiple of the alignments that follow. > > +                      * BARs always are (size == align); bridge windows > > +                      * are not (arbitrary size, IORESOURCE_STARTALIGN to > > +                      * their largest BAR), and two 32G+2M windows aligned > > +                      * to 32G need 96G+2M of span, not 64G+4M. Pad such > > +                      * resources up to their alignment. > > +                      */ > > +                     size += ALIGN(r_size, align); > > This is not required for case where there's just a single composite > resource so it will add again the space wastage back I've tried hard to > remove. And because of that wasted space, it will regress on some > systems/scenarios so it's, while simple looking "solution", a non-starter. > > I didn't want to read the long explanation which contained just snippets > but this likely is the same case Bjorn reported to me privately that > relates to two (or more) composite resources (resources whose size do > not align with the final align). > > I've been busy with dealing another set of resource problems caused by > 9036bd0efcb6 but I was going to write the fix to this problem soon as > well. > > The final head alignment is not calculated until later in pbus_size_mem() > and may be different for case with optional resources. The correction to > the window should be based on those real alignments, not on the resource's > own alignment, and only applied to the final size if there's more than one > composite resources in the first place. > > -- >  i. > > > >