Re: [BUG] PCIe bridge resource allocation creates invalid limit addresses after Secondary Bus Reset recovery

[Bjorn Helgaas <helgaas@kernel.org>]

[+cc Ilpo]

On Wed, Mar 11, 2026 at 10:00:39PM +0000, Shawn Jin wrote:
> Hello,
> 
> I'm reporting a potential critical bug in the Linux kernel's PCIe
> resource allocation code that creates invalid bridge window limit
> addresses during hotplug re-enumeration after Secondary Bus Reset
> (SBR) recovery.

Thanks for the report and the repro and debugging information!

> ## AFFECTED KERNEL VERSIONS
> - Confirmed: 5.15.0, 6.8.0 (Ubuntu 6.8.0-88-generic, 6.8.0-90-generic)
> - Likely affected: All recent kernels including 6.19

Do you know of any kernels that are *not* affected?  If you do, we
could bisect.

> ## HARDWARE CONFIGURATION
> Intel Ice Lake server with PCIe Gen5 switches and endpoints:
> 
> Topology 1:
>   Root Port 96:01.0 → 98:00.0 → 99:01.0 → 9b:00.0 (NVIDIA L20 GPU)
> 
> Kernel parameter: pci=realloc=on
> 
> ## PROBLEM DESCRIPTION
> 
> After performing Secondary Bus Reset on a PCIe switch port and
> clearing the reset bit, the kernel re-enumerates devices and assigns
> bridge window resources. However, the assigned memory window limit
> addresses are INVALID according to PCIe specification.
> 
> ### Evidence from dmesg (Topology 1):
> 
> **Before SBR (correct allocation):**
> ```
> [    6.636493] pci 0000:98:00.0: PCI bridge to [bus 99-9c]
> [    6.636539] pci 0000:98:00.0:   bridge window [mem 0xe9600000-0xe96fffff]
> [    6.636645] pci 0000:98:00.0:   bridge window [mem 0x13b000000000-0x13b7ffffffff 64bit pref]
> 
> [    6.644429] pci 0000:99:01.0: PCI bridge to [bus 9b]
> [    6.644476] pci 0000:99:01.0:   bridge window [mem 0xe9600000-0xe96fffff]
> [    6.644656] pci 0000:99:01.0:   bridge window [mem 0x13b000000000-0x13b7ffffffff 64bit pref]
> 
> [    6.654203] pci 0000:9b:00.0: [1e3e:0002] type 00 class 0x120000 PCIe Endpoint
> [    6.654652] pci 0000:9b:00.0: BAR 0 [mem 0x13b000000000-0x13b7ffffffff 64bit pref]
> [    6.654666] pci 0000:9b:00.0: BAR 2 [mem 0xe9600000-0xe963ffff]
> ```
> 
> **After SBR clear (INVALID allocation):**
> ```
> [  656.644184] pci 0000:98:00.0: bridge window [mem 0x13b000000000-0x13b7ffffffff 64bit pref]: assigned
> [  656.644186] pci 0000:98:00.0: bridge window [mem 0xe9600000-0xe96fffff]: assigned
> [  656.644188] pci 0000:99:01.0: bridge window [mem 0x13b000000000-0x13b7fffffffe 64bit pref]: assigned
> [  656.644189] pci 0000:99:01.0: bridge window [mem 0xe9600000-0xe96ffffe]: assigned
> 
> [  656.644830] pci 0000:9b:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no space
> [  656.644831] pci 0000:9b:00.0: BAR 0 [mem size 0x800000000 64bit pref]: failed to assign
> // BAR2 can still be assigned because the size is only 256KB, while the min window in the bridge is 1MB
> [  656.644832] pci 0000:9b:00.0: BAR 2 [mem 0xe9600000-0xe963ffff]: assigned
> 
> ```
> 
> ### Invalid Addresses Created by Kernel:
> - `0x13b7ffffffff` (ends in 0xFFFE - **2 bytes short**)
> - `0xe96ffffe`  (ends in 0xFFFE - **2 bytes short**)
> 
> ## IMPACT
> 
> 1. **Device initialization failure**: Endpoints cannot allocate required BARs
>    ```
> [  656.644830] pci 0000:9b:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't assign; no space
> [  656.644831] pci 0000:9b:00.0: BAR 0 [mem size 0x800000000 64bit pref]: failed to assign
>    ```
> 
> 2. **Consistent across multiple hierarchies**: Affects different PCIe topologies independently
> 
> ## REPRODUCTION
> 
> The attached script test_rc_sbr.sh.txt issues a SBR to the root port.
> 
> ## SUSPECTED ROOT CAUSE
> 
> The bug appears to be in `drivers/pci/setup-bus.c`, likely in:
> - `pci_bus_distribute_available_resources()`
> - `adjust_bridge_window()`
> - `pci_assign_unassigned_bridge_resources()`
> 
> The resource end address calculation appears to perform multiple subtractions:
> 1. Initial calculation: `res->end = res->start + size - 1` (correct)
> 2. During redistribution: Another subtraction occurs, creating `res->end = ... - 2`
> 
> ## WORKAROUND ATTEMPTS
> 
> - `pci=realloc=on`: Does NOT fix the issue
> - Manual remove/rescan from root: Does NOT fix the issue
> - Initial boot allocation: Works correctly (bug only occurs during hotplug re-enumeration)
> 
> ## REQUEST
> 
> I want to track how the bridge windows are allocated. Is there a way
> to enable additional kernel messages to show the path? Please
> investigate if this is a real kernel bug.
> 
> Thank you,
> Shawn

> #!/bin/bash
> 
> # Function to display usage
> usage() {
>     echo "Usage: $0 -rp <ROOT_PORT_BDF> -usp <USP_BDF>"
>     echo "Example: $0 -rp c6:01.0 -usp c7:00.0"
>     exit 1
> }
> 
> # Initialize variables
> ROOT_PORT_BDF=""
> USP_BDF=""
> 
> # Parse command-line arguments
> while [[ $# -gt 0 ]]; do
>     case $1 in
>         -rp)
>             ROOT_PORT_BDF="$2"
>             shift 2
>             ;;
>         -usp)
>             USP_BDF="$2"
>             shift 2
>             ;;
>         -h|--help)
>             usage
>             ;;
>         *)
>             echo "Unknown option: $1"
>             usage
>             ;;
>     esac
> done
> 
> # Validate that both arguments are provided
> if [ -z "$ROOT_PORT_BDF" ] || [ -z "$USP_BDF" ]; then
>     echo "Error: Both -rp and -usp arguments are required"
>     usage
> fi
> 
> echo "Root Port BDF: $ROOT_PORT_BDF"
> echo "USP BDF: $USP_BDF"
> echo ""
> 
> # Remove the USP device
> echo 1 | sudo tee /sys/bus/pci/devices/0000:${USP_BDF}/remove
> 
> # Trigger SBR via Bridge Control register
> BRIDGE_CTL=$(sudo setpci -s ${ROOT_PORT_BDF} 0x3E.w)
> BRIDGE_CTL_RESET=$(printf "0x%04x" $((0x$BRIDGE_CTL | 0x0040)))
> 
> echo "Asserting Secondary Bus Reset..."
> sudo setpci -s ${ROOT_PORT_BDF} 0x3E.w=$BRIDGE_CTL_RESET
> sleep 1
> 
> echo "De-asserting Secondary Bus Reset..."
> sudo setpci -s ${ROOT_PORT_BDF} 0x3E.w=$BRIDGE_CTL
> sleep 2
> 
> # Rescan PCI bus
> echo 1 | sudo tee /sys/bus/pci/rescan