From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CA3DCC624A4 for ; Thu, 3 Sep 2026 07:34:55 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D52B410F430; Thu, 3 Sep 2026 07:34:49 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=exabit-io.20251104.gappssmtp.com header.i=@exabit-io.20251104.gappssmtp.com header.b="DAOLBAM4"; dkim-atps=neutral Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3C1E710E03E for ; Thu, 3 Sep 2026 06:31:26 +0000 (UTC) Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-853c07a76adso2365991b3a.0 for ; Wed, 02 Sep 2026 23:31:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=exabit-io.20251104.gappssmtp.com; s=20251104; t=1788417086; x=1789021886; darn=lists.freedesktop.org; h=content-transfer-encoding:content-type:mime-version:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Oyn1eAlWhmjTg1gDpWtJ1K38m8614GuSxJp7RLTJwds=; b=DAOLBAM48BFzO3Bdr16BwP+ew6H/kyjMujbGCOvNbYJnYlGloMSEGXOWow/Zf4bS1S x6iV1COi3n/QtgFhgMHXR9XgCuhFpSltIJ4fkKeXQjxNq/mhvpfbGW3iaJkD9d+8ign7 HTAnYhgwhHfBZaGNAEojhabP8cEYwRqskFy75jSQwBlAUooa8Jw5ELZ1NyhxlmIEqc31 Um8753g7UF26OHaPtBlcHtfJbt1aYyr4g5I15A2PdfYVrVSur5h2nJL8DgLuU4SDmT4m oWVts6y2ItNXzKeIuEdPxwSnvWC2zALJ9mNMtfzAThYip1zNYnDvkXybUQ5KmqG95NSe kpYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788417086; x=1789021886; h=content-transfer-encoding:content-type:mime-version:message-id:date :subject:cc:to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Oyn1eAlWhmjTg1gDpWtJ1K38m8614GuSxJp7RLTJwds=; b=jzLADNU3BvNHeRBOQ7opk7yRb8Jxo+93wQ5Pg5owyqKOQS0g6MYYrjOWvKflnb1UDa TvZ5DfLFTiv4TiApX9LbdrOdbsQsxNzmi1tigU7Tpw1+snvsT9sUwTXPbtkTK9GTsaPh jLjbV29nYQv2t5i+Abz7wil+YVqlGnM59H0U0zcSHppRT2GNcWu4MTa0EuKWUcLjjHSb aKPzDHg3OIP43c4pqArnrXhtnBLVPF2CidKaLI3QmzF0c8YAFEl4cxC6bWrBCe2uYAIN R4AD5BY218id8B8jGDNfhFNS1rouOVEAU0h8zXgnfRZQUSz40jSXaiPy9iuNq99aRVW+ ChzQ== X-Forwarded-Encrypted: i=1; AKwUvBwHll7rYV3hdGyIdXifuWESKHIVeEd1oCoYFYePxCRGUCKm0ysDY04RIn2P0UgiMRuC3LAKxUjS@lists.freedesktop.org X-Gm-Message-State: AFuF++kgiGqC/KaWPvntmqsKbDf/kCc8vymoai5OfTfEwKUWeX5MkoPG GdThfktuAVNQpL6a6/I/kHGrpxyhwFwMAULs7gzP4fqYoz4qNoWTYGhvdvsqE+UDsbE= X-Gm-Gg: AYBFou2mdH6WO+e74IH3M9SBsrhCmUIsX8Hdxp1yM727gwNIw9WHZb8LOAjYd/oml/t etgjUHVfCGA7bdk6gt++NCO33r/cu4oP+L9wj4FOQZOLKmIp4n/lcORqZNsrV8lmfEUcnlqT9yA 4diAvFDhnKnXj1qfyqWb5+qdGryz2mRfX8TjzjMtAdzOKWr1vofn7oM2rJHRbimFTkXRas9+Zof TTk2DCssJUzS6waBk59v13NwUzGulL3BWgeE7AWluvX1gBpBor53n4GhqmOe9xZ6GgmjsO9OdG2 27EcC3VCxgNfTn9pePzKRf/8GWG6PNvrzUsId/Si/8nFUOnJKZ/AYP4ZNFHaPFf7ncviEM2Dv9+ MO3WrvD/l8Uynw2xp9RjHqrdufEptCFnSgFTiO79nO41mM9INP8m3ZDHMNfE5xxcKrVXQMEIJfN h2q/ZHtd5yM/AEJZVmiav1jL66hNItsYVbqVYeKMKbWKKbyABy7kx5sLwjblxy94gJMjkpKoq5Q fDcInNRcE4YtNy0SOAkMw/Bjr8= X-Received: by 2002:a05:6a00:c8d:b0:83e:b443:965e with SMTP id d2e1a72fcca58-85ed5ccca45mr14370109b3a.3.1788417085359; Wed, 02 Sep 2026 23:31:25 -0700 (PDT) Received: from macpro2019-01.. (c-71-236-167-8.hsd1.or.comcast.net. [71.236.167.8]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-85db23f8d8asm2536761b3a.12.2026.09.02.23.31.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Sep 2026 23:31:24 -0700 (PDT) From: Nikolas Joshua Britton To: linux-pci@vger.kernel.org Cc: Bjorn Helgaas , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , regressions@lists.linux.dev, amd-gfx@lists.freedesktop.org, Nikolas Joshua Britton Subject: PCI: bridge window undersized when child bridge windows are larger than their alignment (regression since 3958bf16e2fe, v7.0) Date: Thu, 3 Sep 2026 06:31:24 +0000 Message-ID: <20260903063124.9316-1-nbritton@exabit.io> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Thu, 03 Sep 2026 07:34:48 +0000 X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" Hi, On a Mac Pro 7,1 with two Radeon Pro Vega II Duo cards, enabling 32 GB Resizable BARs leaves exactly half the GPU dies with no BAR at all. The shared root-port prefetchable window is sized as the plain sum of its two child bridge windows, but each child secretly requires a 32 GB-aligned start, so the window that gets allocated is ~32 GB smaller than the span that is actually needed. The second child of each pair loses, deterministically. There is 1 TiB of free space in the host bridge's _CRS window, so this is not address-space exhaustion. This is a regression. On the same machine, with the same script (resize-amdgpu-bars, https://github.com/exabit-io/resize-amdgpu-bars, which performs the sequence under "Reproduction" below at boot) and the same sequence of operations, Ubuntu's 6.8.0-138 (upstream 6.8.12), 6.11.0-29, 6.14.0-37 and 6.17.0-42 kernels all size the shared window at 96 GiB (48 GiB per child) and all four dies get their 32 GiB BAR on the first attempt, every time (each kernel tested from a full power-off). 7.0.0-30 is based on upstream 7.0.12 and fails identically from a cold boot and from a warm reboot. I have traced it to commit 3958bf16e2fe ("PCI: Stop over-estimating bridge window size"), first shipped in v7.0; analysis and a proposed one-line fix are below. Ubuntu's setup-bus.c is byte-identical to v7.0.12, which already includes 8cb081667377 ("PCI: Fix alignment calculation for resource size larger than align") and dc4b4d04e1ca ("PCI: Prevent shrinking bridge window from its required size"), so those do not cover this case; current master has no further change to this logic. System ------ Machine: Apple Inc. MacPro7,1, BIOS 2103.160.2.0.0 Fails: 7.0.0-30-generic (Ubuntu 24.04 HWE, 7.0.0-30.30~24.04.1, upstream 7.0.12) self-built upstream v7.0.12, unpatched, Ubuntu config trimmed with localmodconfig (control; fails identically) Works: 6.8.0-138-generic (Ubuntu 24.04 GA, upstream 6.8.12) 6.11.0-29-generic (Ubuntu 24.04, linux-generic-6.11) 6.14.0-37-generic (Ubuntu 24.04, linux-generic-6.14) 6.17.0-42-generic (Ubuntu 24.04, linux-generic-6.17) (all four: identical 96 GiB window layout, 4/4 dies, one 4-node XGMI hive, no traces; verified 2026-09-02) self-built v7.0.12 + the patch below (same config as the control): 4/4 dies, 128 GiB root-port window, cold boot and warm reboot, no traces Cmdline: ro log_buf_len=16M pci=realloc mitigations=off (the 6.x and the self-built 7.0.12 boots also carried intremap=no_x2apic_optout; it has no bearing on this) GPUs: 4x Vega20 [1002:66a3], two dies per Vega II Duo card BAR0 ReBAR capability: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB BAR2 (doorbell): 2MB, fixed in practice Topology -------- Each Duo card presents two dies behind ONE root port, each die on its own sub-bridge chain: +-[0000:06]-+-00.0-[07-0e]--00.0-[08-0e]--+-08.0-[09-0b]--00.0-[0a-0b]--00.0-[0b]--00.0 Vega20 | \-10.0-[0c-0e]--00.0-[0d-0e]--00.0-[0e]--00.0 Vega20 +-[0000:16]-+-00.0-[17-1e]--00.0-[18-1e]--+-08.0-[19-1b]--00.0-[1a-1b]--00.0-[1b]--00.0 Vega20 \-10.0-[1c-1e]--00.0-[1d-1e]--00.0-[1e]--00.0 Vega20 So 08:08.0 and 08:10.0 are siblings sharing the prefetchable window of 07:00.0 / 06:00.0. Each subtree contains one Vega20 with BAR0 = 32 GB (alignment 32 GB) plus BAR2 = 2 MB, i.e. each child bridge window is 32 GB + 2 MB = 0x800200000. Both cards fail identically. Card 2 (16:00.0 / 18:08.0 / 18:10.0) is omitted below for brevity; its trace is byte-for-byte analogous. The arithmetic -------------- host bridge _CRS window 0x90000000000-0x9ffffffffff 1 TiB free 06:00.0 / 07:00.0 window 0x90000000000-0x910003fffff 0x1000400000 (64 GiB + 4 MiB) 08:08.0 window assigned 0x90000000000-0x908001fffff 0x800200000 next free address 0x90800200000 08:10.0 needs 32 GiB alignment, so its next legal start is 0x91000000000 08:10.0 would then end at 0x91800200000 but the parent window ends at 0x91000400000 <-- ~32 GiB short span actually required 0x90000000000-0x91800200000 0x1800200000 (96 GiB + 2 MiB) span allocated 0x1000400000 (64 GiB + 4 MiB) shortfall 0x7ffe00000 (32 GiB - 2 MiB) The telling detail: 0x90800200000 + 0x800200000 = 0x91000400000, which is *exactly* the parent window's exclusive end. In other words, had the child bridge windows only needed ~1 MiB alignment, the two of them would have fit perfectly, to the byte. The sizing pass produced a window that is correct if and only if the children can be packed back-to-back, which they cannot, because the assignment pass then enforces the real 32 GiB alignment inherited from the BAR inside each child. Good kernel, for comparison --------------------------- Same hardware, same steps, 6.17.0-42 (6.8, 6.11 and 6.14 are identical). The parent is sized for the worst-case packing, and both children fit: 0000:06:00.0: 90000000000-917ffffffff [size=96G] <- shared parent 0000:08:08.0: 90000000000-90bffffffff [size=48G] 0000:08:10.0: 90c00000000-917ffffffff [size=48G] 0000:0b:00.0: Region 0: Memory at 90000000000 (64-bit, prefetchable) [size=32G] 0000:0e:00.0: Region 0: Memory at 91000000000 (64-bit, prefetchable) [size=32G] i.e. 96 GiB = 3 x 32 GiB: each child gets (32 GiB + 2 MiB) rounded up to the next 32 GiB boundary plus slack, so the second child's aligned start is always inside the parent. That is the 3A + eps span from the arithmetic below, and 7.0 allocates 2A + 2 eps instead. dmesg (7.0) ----------- Sizing and assignment of the shared window, then the two children: pci 0000:06:00.0: bridge window [mem 0x90000000000-0x910003fffff 64bit pref]: assigned pci 0000:07:00.0: bridge window [mem 0x90000000000-0x910003fffff 64bit pref]: assigned pci 0000:08:08.0: bridge window [mem 0x90000000000-0x908001fffff 64bit pref]: assigned pci 0000:08:10.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; no space pci 0000:08:10.0: bridge window [mem size 0x800200000 64bit pref]: failed to assign The failure then cascades down the losing chain, and the endpoint is left with neither BAR0 nor BAR2: pci 0000:0c:00.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; no space pci 0000:0d:00.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; no space pci 0000:0e:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no space pci 0000:0e:00.0: BAR 0 [mem size 0x800000000 64bit pref]: failed to assign pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't assign; no space pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: failed to assign pci 0000:0e:00.0: BAR 5 [mem 0x74600000-0x7467ffff]: assigned Note BAR5 (the 512 KB register aperture, non-prefetchable) still gets assigned. That matters for the downstream impact described below: the device is half-alive rather than obviously dead. Resulting state, from lspci -vv. The entire losing chain has no prefetchable window whatsoever: 0000:08:08.0: 90000000000-908001fffff [size=32770M] <- winner 0000:09:00.0: 90000000000-908001fffff [size=32770M] 0000:0a:00.0: 90000000000-908001fffff [size=32770M] 0000:08:10.0: [disabled] <- loser 0000:0c:00.0: [disabled] 0000:0d:00.0: [disabled] 0000:06:00.0: 90000000000-910003fffff [size=65540M] <- shared parent And /sys/bus/pci/devices/0000:0e:00.0/resource: 0x0000000000000000 0x0000000000000000 0x0000000000000000 BAR0 unassigned 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 BAR2 unassigned 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000006000 0x00000000000060ff 0x0000000000040101 0x0000000074600000 0x000000007467ffff 0x0000000000040200 BAR5 assigned Observation ----------- Observed, and I think not in dispute: - the sizing pass produced a parent window exactly equal to the sum of the two child window sizes (0x800200000 * 2 = 0x1000400000); - the assignment pass refused to place the second child at 0x90800200000, which is 2 MiB-aligned and would have fit exactly; - therefore assignment enforced an alignment that sizing did not budget for. Where in the code: pbus_size_mem() sums the child bridge windows without regard to the alignment the assignment pass will enforce on them. The analysis, a small model that reproduces every number above, and a one-line fix that has been A/B tested on this machine are in "Root cause" and "Proposed fix" below, before the list of gaps. The shortfall is structural rather than specific to 32 GB. For N=2 siblings each needing (A + eps) at alignment A, the required span is 2A + (A + eps) = 3A + eps, while the sum is 2A + 2eps. I have confirmed this at the other end of the range: after one failed 32 GiB attempt, writing the ReBAR index back to 256 MiB and re-enumerating fails in exactly the same way, because the firmware's original 770 MiB windows (~3A + eps for A = 256 MiB) are gone and the kernel re-sizes the parent to 2A + 2 eps: pci 0000:06:00.0: bridge window [mem 0x90000000000-0x900203fffff 64bit pref]: assigned (512 MiB + 4 MiB) pci 0000:08:08.0: bridge window [mem 0x90000000000-0x900101fffff 64bit pref]: assigned pci 0000:08:10.0: bridge window [mem size 0x10200000 64bit pref]: can't assign; no space pci 0000:0e:00.0: BAR 0 [mem size 0x10000000 64bit pref]: can't assign; no space pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't assign; no space So on 7.0 there is no in-place recovery once the kernel has re-sized the window: the only layouts that ever work are the ones the firmware left behind. 16 GB and 8 GB remain untested but I would not expect them to differ. Downstream impact: this is not a soft failure --------------------------------------------- The BAR-less die is not merely unusable. Because BAR5 is still assigned, amdgpu probes it, its register reads return garbage, and RCC_IOV_FUNC_IDENTIFIER comes back with bit 0 set. The driver concludes the device is an SR-IOV *virtual function*: amdgpu 0000:0b:00.0: register mmio base: 0x74400000 <- healthy die amdgpu 0000:0e:00.0: register mmio base: 0x74600000 amdgpu 0000:0e:00.0: MCBP is enabled <- only set when amdgpu_sriov_vf() It then calls amdgpu_virt_request_full_gpu() -> xgpu_ai_request_full_gpu_access() and waits forever for a hypervisor mailbox that does not exist: amdgpu 0000:0e:00.0: trn=2 ACK should not assert! wait again ! (repeating roughly 2490 times per 5 seconds, indefinitely) xgpu_ai_mailbox_trans_msg+0x1a9/0x1f0 [amdgpu] xgpu_ai_send_access_requests+0x21/0xe0 [amdgpu] xgpu_ai_request_full_gpu_access+0x1a/0x30 [amdgpu] amdgpu_virt_request_full_gpu+0x2a/0x70 [amdgpu] amdgpu_device_ip_early_init.constprop.0+0x173/0x780 [amdgpu] amdgpu_device_init+0x83d/0x1180 [amdgpu] amdgpu_driver_load_kms+0x1a/0xd0 [amdgpu] amdgpu_pci_probe+0x1df/0x590 [amdgpu] modprobe wedges in uninterruptible D state holding the device mutex, blocks the AER IRQ thread, and never returns: INFO: task irq/34-aerdrv:1554 blocked for more than 122 seconds. INFO: task irq/34-aerdrv:1554 is blocked on a mutex likely owned by task modprobe:1583. SIGKILL does not touch it; systemd's TimeoutStartSec fires and the unit goes to "failed" while the task stays in the cgroup. The machine needs a reboot. The remaining two dies are never probed at all. So the practical outcome of the sizing bug with a plain "modprobe amdgpu" is: 1 of 4 GPUs usable (the healthy die of the first card; its BAR-less sibling wedges the probe and the second card is never reached), no XGMI hive (kfd reports a single-node hive), and an unkillable task on every boot. Keeping the BAR-less dies away from the driver with driver_override, which resize-amdgpu-bars now does, gets 2 of 4 dies and a 2-node hive; that is the "2/4 dies" figure under "Proposed fix" below. Whether amdgpu should be more defensive about probing a device with an unassigned BAR0 is a separate question for amd-gfx, and I've cc'd them, but the PCI-side undersizing is the trigger. Reproduction ------------ With the four dies at their default 256 MB BAR0 (this is what resize-amdgpu-bars does at boot, done by hand): 1. Set BAR0 to 32 GB on all four dies (ReBAR control register at capability offset 0x200, control at 0x208, size index in bits 8-13, index 15 = 2^35): setpci -s 0000:0b:00.0 0x208.l=00000f40 (likewise for 0e:00.0, 1b:00.0, 1e:00.0) 2. Force full re-enumeration so the kernel re-sizes every bridge window from scratch: echo 1 > /sys/bus/pci/devices/0000:06:00.0/remove echo 1 > /sys/bus/pci/devices/0000:16:00.0/remove echo 1 > /sys/bus/pci/rescan 3. dmesg shows the "can't assign; no space" trace above; 0e:00.0 and 1e:00.0 have BAR0 and BAR2 unassigned, 0b:00.0 and 1b:00.0 are fine. Fully deterministic across many attempts: the first-enumerated die of each card always wins. Root cause ---------- Since 3958bf16e2fe, pbus_size_mem() sizes a bridge window as size += max(r_size, align); /* per child */ size0 = ALIGN(size, win_align); /* win_align = 1 MB */ with calculate_head_align() making only the window *start* satisfy the largest child alignment. The tight fit is gap-free only if, in the descending-alignment assignment order, every child's size is a multiple of the alignments of the children placed after it. That holds for BARs (size == alignment) but not for bridge windows: their size is the sum of what is below them, while their alignment (IORESOURCE_STARTALIGN) is that of the largest BAR below them. Here each sub-bridge window holds BAR0 (32 GiB) + BAR2 (2 MiB doorbell), so it is sized 32 GiB + 2 MiB with 32 GiB alignment. The root port sums two of them: 64 GiB + 4 MiB. Assignment then places the first child at offset 0 (ends at 32 GiB + 2 MiB) and must put the second at the next 32 GiB boundary, i.e. 64 GiB .. 96 GiB + 2 MiB. Needed 96 GiB + 2 MiB, sized 64 GiB + 4 MiB -> "can't assign; no space" for the second die. pci 0000:06:00.0: bridge window [mem 0x90000000000-0x910003fffff 64bit pref]: assigned pci 0000:08:08.0: bridge window [mem 0x90000000000-0x908001fffff 64bit pref]: assigned pci 0000:0b:00.0: BAR 0 [mem 0x90000000000-0x907ffffffff 64bit pref]: assigned pci 0000:0b:00.0: BAR 2 [mem 0x90800000000-0x908001fffff 64bit pref]: assigned pci 0000:0e:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no space pci 0000:0e:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't assign; no space The same arithmetic at the default 256 MiB BAR0 gives 512 MiB + 4 MiB sized vs 768 MiB + 2 MiB needed, which is why even reverting the BAR size does not recover (the firmware's original windows were larger). Up to v6.17, calculate_memsize() rounded each bridge window up to its own min_align (ALIGN(size, min_align); the old calculate_mem_align() gave 16 GiB here), so each sub-bridge window was 48 GiB and the root port 96 GiB, and siblings packed by accident. A small model of both versions of pbus_size_mem() reproduces every number seen on this machine (48G/96G on 6.8-6.17; 32770M/65540M and 258M/516M on 7.0). This is the tail-side sibling of the head-side under-estimation Guenter Roeck raised on 2026-03-05 for the same series (4M@4M + 3M@1M + 1M@1M needing 9 MiB in an 8 MiB window), which 8cb081667377 addressed for the head alignment bookkeeping only. Proposed fix ------------ Pad a child up to its alignment when its size is not a multiple of it: - size += max(r_size, align); + size += ALIGN(r_size, align); This is a no-op for BARs, so the v7.0 tight fit for leaf resources is kept; for bridge windows it restores the pre-v7.0 parent sizing. It over-estimates by up to one alignment unit (128 GiB here rather than the exact 96 GiB + 2 MiB); an exact version would have to walk the children in assignment order and simulate the offsets. The diff follows below the "---" line at the end of this mail; the full patch with changelog and Signed-off-by is ready and I will send it as a separate [PATCH] if that is preferred. Tested on the same machine, same script, same sequence (2026-09-02): v7.0.12 unpatched (control): identical to 7.0.0-30. Root-port window 0x90000000000-0x900203fffff (512 MiB + 4 MiB at the 256 MiB baseline the kernel falls back to), "can't assign; no space" on 0e:00.0 and 1e:00.0, 2/4 dies, 2-node XGMI hive. v7.0.12 + patch: 4/4 dies with 32 GiB BAR0, 4-node XGMI hive, from a cold boot and again from a warm reboot. Window layout for card 1: pci 0000:06:00.0: bridge window [mem 0x90000000000-0x91fffffffff 64bit pref]: assigned (128 GiB) pci 0000:07:00.0: bridge window [mem 0x90000000000-0x91fffffffff 64bit pref]: assigned pci 0000:08:08.0: bridge window [mem 0x90000000000-0x90fffffffff 64bit pref]: assigned (64 GiB) pci 0000:09:00.0: bridge window [mem 0x90000000000-0x90fffffffff 64bit pref]: assigned pci 0000:0a:00.0: bridge window [mem 0x90000000000-0x908001fffff 64bit pref]: assigned (32 GiB + 2 MiB) pci 0000:0b:00.0: BAR 0 [mem 0x90000000000-0x907ffffffff 64bit pref]: assigned pci 0000:0b:00.0: BAR 2 [mem 0x90800000000-0x908001fffff 64bit pref]: assigned pci 0000:08:10.0: bridge window [mem 0x91000000000-0x91fffffffff 64bit pref]: assigned pci 0000:0e:00.0: BAR 0 [mem 0x91000000000-0x917ffffffff 64bit pref]: assigned pci 0000:0e:00.0: BAR 2 [mem 0x91800000000-0x918001fffff 64bit pref]: assigned The padding is applied once, where 09:00.0 sums its 32 GiB + 2 MiB child (alignment 32 GiB) to 64 GiB; the levels above are plain sums of already-aligned children. Net cost 128 GiB per card instead of the exact 96 GiB + 2 MiB, out of 1 TiB available. Why the repro does not use the sysfs resource0_resize interface ---------------------------------------------------------------- The repro above pokes the ReBAR control register with setpci and then forces a rescan instead of using the sanctioned interface (echo 15 > .../resource0_resize). That is deliberate: the sanctioned interface cannot grow a die that sits behind the card's own PCIe switch, on any kernel, and that is the reason resize-amdgpu-bars exists at all. It was the first thing I tried, and I re-measured it on 7.0.12 for this report so the failure is on record with the kernel's own lines. Measured (7.0.12 vanilla, unit masked, amdgpu blacklisted, so all four dies sat at the firmware 256 MB and nothing was bound; the sibling die's BARs stay assigned whether or not a driver is bound): # echo 15 > /sys/bus/pci/devices/0000:0b:00.0/resource0_resize write error: No space left on device (-ENOSPC) pci 0000:0b:00.0: BAR 0 [mem 0x9ffe0000000-0x9ffefffffff 64bit pref]: releasing pcieport 0000:0a:00.0: bridge window [mem 0x9ffe0000000-0x9fff01fffff 64bit pref]: releasing pcieport 0000:09:00.0: bridge window [mem 0x9ffe0000000-0x9fff01fffff 64bit pref]: releasing pcieport 0000:08:08.0: bridge window [mem 0x9ffe0000000-0x9fff01fffff 64bit pref]: releasing pcieport 0000:07:00.0: bridge window [mem 0x9ffc0000000-0x9fff01fffff 64bit pref]: was not released (still contains assigned resources) pcieport 0000:06:00.0: bridge window [mem 0x9ffc0000000-0x9fff01fffff 64bit pref]: was not released (still contains assigned resources) pcieport 0000:08:08.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; no space pci 0000:0b:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no space pci 0000:0b:00.0: BAR 0 [mem 0x9ffe0000000-0x9ffefffffff 64bit pref]: old value restored Parent window (06:00.0 and 07:00.0, shared by both dies) before and after: 0x9ffc0000000-0x9fff01fffff, 770 MB, unchanged. Writing 8 back returned 0 and every window came back byte-identical. The whole thing took under half a second and nothing hung. So the sysfs path fails the same way for the user (-ENOSPC, "old value restored", BAR stays 256 MB) but for a reason one level below the sizing bug: the shared window on the two bridges above the switch is never released while the sibling die's BARs are assigned in it, so the three bridge windows on the way down have to grow to 32 GB + 2 MB inside a 770 MB parent, and cannot. The in-place path never gets to re-size the shared window at all; it is limited to what fits in the firmware layout. That is independent of this bug (it would fail the same way on a fixed kernel and on 6.17), and it is why my tool grows a dual-die module by removing the whole module and rescanning from the root port instead. The sysfs write is not a workaround here and does not exercise the sizing path this patch fixes; the setpci + rescan repro and amdgpu's own resize at probe (below) are the two routes that reach it, and both fail with "can't assign; no space" on unpatched 7.0. What I have not tested ---------------------- I want to be straight about the gaps: - I have not tested 16 GB or 8 GB BARs (see the structural argument above; 256 MB and 32 GB are the two data points). - amdgpu's own resize at probe time (amdgpu_device_resize_fb_bar -> pci_resize_resource) hits the same wall on 7.0. From an earlier boot where amdgpu was allowed to autoload with the dies still at 256 MB: amdgpu 0000:1b:00.0: BAR 2 [mem 0xbfff0000000-0xbfff01fffff 64bit pref]: old value restored amdgpu 0000:1b:00.0: BAR 0 [mem 0xbffe0000000-0xbffefffffff 64bit pref]: old value restored amdgpu 0000:1b:00.0: Not enough PCI address space for a large BAR. amdgpu 0000:1b:00.0: [drm] Detected VRAM RAM=32752M, BAR=256M That path fails closed (ReBAR index reverted, driver continues at 256 MB), so it does not trigger the hang described above, but it does not get a large BAR either. - I have not bisected by booting: the commit was identified by reading setup-bus.c across the versions and confirmed by the A/B test of v7.0.12 with and without the one-line patch (see "Proposed fix"). If a real bisection would still help I can run it. Happy to test patches, gather more traces, or run with any debug options that would help. The machine is otherwise idle and I can reboot it freely. Available on request (not attached; the largest is 1.1 MB): - full dmesg from the failing boot - lspci -vvnn and lspci -tvnn - the trimmed allocation trace for card 1 - lspci -vv from the 6.17 boot (working layout) - the sizing model (pci-window-sim.py, 130 lines) - full dmesg from the v7.0.12 control boot and the v7.0.12 + patch boot - lspci -vvnn from the v7.0.12 + patch boot #regzbot introduced: 3958bf16e2fe --- drivers/pci/setup-bus.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1329,7 +1329,17 @@ static void pbus_size_mem(struct pci_bus *bus, struct resource *b_res, continue; r_size = resource_size(r); - size += max(r_size, align); + /* + * Resources are assigned in descending alignment + * order, so a tight-fit sum is only gap-free if each + * size is a multiple of the alignments that follow. + * BARs always are (size == align); bridge windows + * are not (arbitrary size, IORESOURCE_STARTALIGN to + * their largest BAR), and two 32G+2M windows aligned + * to 32G need 96G+2M of span, not 64G+4M. Pad such + * resources up to their alignment. + */ + size += ALIGN(r_size, align); /* * If resource's size is larger than its alignment,