* [PATCH v2] PCI: Use downstream bridges for distributing resources
@ 2024-12-04 2:24 Kai-Heng Feng
2024-12-04 5:48 ` Mika Westerberg
2025-02-13 21:21 ` Bjorn Helgaas
0 siblings, 2 replies; 6+ messages in thread
From: Kai-Heng Feng @ 2024-12-04 2:24 UTC (permalink / raw)
To: bhelgaas, mika.westerberg
Cc: linux-pci, linux-kernel, Kai-Heng Feng, Carol Soto,
Jonathan Cameron, Chris Chiu, Chia-Lin Kao
Commit 7180c1d08639 ("PCI: Distribute available resources for root
buses, too") breaks BAR assignment on some devcies:
[ 10.021193] pci 0006:03:00.0: BAR 0 [mem 0x6300c0000000-0x6300c1ffffff 64bit pref]: assigned
[ 10.029880] pci 0006:03:00.1: BAR 0 [mem 0x6300c2000000-0x6300c3ffffff 64bit pref]: assigned
[ 10.038561] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: can't assign; no space
[ 10.047191] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: failed to assign
[ 10.055285] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
[ 10.064180] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
[ 10.072543] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
[ 10.081437] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
The apertures of domain 0006 before the commit:
6300c0000000-63ffffffffff : PCI Bus 0006:00
6300c0000000-6300c9ffffff : PCI Bus 0006:01
6300c0000000-6300c9ffffff : PCI Bus 0006:02
6300c0000000-6300c8ffffff : PCI Bus 0006:03
6300c0000000-6300c1ffffff : 0006:03:00.0
6300c0000000-6300c1ffffff : mlx5_core
6300c2000000-6300c3ffffff : 0006:03:00.1
6300c2000000-6300c3ffffff : mlx5_core
6300c4000000-6300c47fffff : 0006:03:00.2
6300c4800000-6300c67fffff : 0006:03:00.0
6300c6800000-6300c87fffff : 0006:03:00.1
6300c9000000-6300c9bfffff : PCI Bus 0006:04
6300c9000000-6300c9bfffff : PCI Bus 0006:05
6300c9000000-6300c91fffff : PCI Bus 0006:06
6300c9200000-6300c93fffff : PCI Bus 0006:07
6300c9400000-6300c95fffff : PCI Bus 0006:08
6300c9600000-6300c97fffff : PCI Bus 0006:09
After the commit:
6300c0000000-63ffffffffff : PCI Bus 0006:00
6300c0000000-6300c9ffffff : PCI Bus 0006:01
6300c0000000-6300c9ffffff : PCI Bus 0006:02
6300c0000000-6300c43fffff : PCI Bus 0006:03
6300c0000000-6300c1ffffff : 0006:03:00.0
6300c0000000-6300c1ffffff : mlx5_core
6300c2000000-6300c3ffffff : 0006:03:00.1
6300c2000000-6300c3ffffff : mlx5_core
6300c4400000-6300c4dfffff : PCI Bus 0006:04
6300c4400000-6300c4dfffff : PCI Bus 0006:05
6300c4400000-6300c45fffff : PCI Bus 0006:06
6300c4600000-6300c47fffff : PCI Bus 0006:07
6300c4800000-6300c49fffff : PCI Bus 0006:08
6300c4a00000-6300c4bfffff : PCI Bus 0006:09
We can see that the window of 0006:03 gets shrunken too much and 0006:04
eats away the window for 0006:03:00.2.
The offending commit distributes the upstream bridge's resources
multiple times to every downstream bridges, hence makes the aperture
smaller than desired because calculation of io_per_b, mmio_per_b and
mmio_pref_per_b becomes incorrect.
Instead, distributing downstream bridges' own resources to resolve the
issue.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=219540
Cc: Carol Soto <csoto@nvidia.com>
Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Cc: Chris Chiu <chris.chiu@canonical.com>
Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
Tested-by: Chia-Lin Kao (AceLan) <acelan.kao@canonical.com>
Fixes: 7180c1d08639 ("PCI: Distribute available resources for root buses, too")
Signed-off-by: Kai-Heng Feng <kaihengf@nvidia.com>
---
drivers/pci/setup-bus.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 23082bc0ca37..a6e653a4f5b1 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -2105,8 +2105,7 @@ pci_root_bus_distribute_available_resources(struct pci_bus *bus,
* in case of root bus.
*/
if (bridge && pci_bridge_resources_not_assigned(dev))
- pci_bridge_distribute_available_resources(bridge,
- add_list);
+ pci_bridge_distribute_available_resources(dev, add_list);
else
pci_root_bus_distribute_available_resources(b, add_list);
}
--
2.47.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v2] PCI: Use downstream bridges for distributing resources
2024-12-04 2:24 [PATCH v2] PCI: Use downstream bridges for distributing resources Kai-Heng Feng
@ 2024-12-04 5:48 ` Mika Westerberg
2024-12-18 11:12 ` Kai-Heng Feng
2025-02-13 21:21 ` Bjorn Helgaas
1 sibling, 1 reply; 6+ messages in thread
From: Mika Westerberg @ 2024-12-04 5:48 UTC (permalink / raw)
To: Kai-Heng Feng
Cc: bhelgaas, linux-pci, linux-kernel, Carol Soto, Jonathan Cameron,
Chris Chiu, Chia-Lin Kao
On Wed, Dec 04, 2024 at 10:24:57AM +0800, Kai-Heng Feng wrote:
> Commit 7180c1d08639 ("PCI: Distribute available resources for root
> buses, too") breaks BAR assignment on some devcies:
> [ 10.021193] pci 0006:03:00.0: BAR 0 [mem 0x6300c0000000-0x6300c1ffffff 64bit pref]: assigned
> [ 10.029880] pci 0006:03:00.1: BAR 0 [mem 0x6300c2000000-0x6300c3ffffff 64bit pref]: assigned
> [ 10.038561] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: can't assign; no space
> [ 10.047191] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: failed to assign
> [ 10.055285] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
> [ 10.064180] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
> [ 10.072543] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
> [ 10.081437] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
>
> The apertures of domain 0006 before the commit:
> 6300c0000000-63ffffffffff : PCI Bus 0006:00
> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
> 6300c0000000-6300c8ffffff : PCI Bus 0006:03
> 6300c0000000-6300c1ffffff : 0006:03:00.0
> 6300c0000000-6300c1ffffff : mlx5_core
> 6300c2000000-6300c3ffffff : 0006:03:00.1
> 6300c2000000-6300c3ffffff : mlx5_core
> 6300c4000000-6300c47fffff : 0006:03:00.2
> 6300c4800000-6300c67fffff : 0006:03:00.0
> 6300c6800000-6300c87fffff : 0006:03:00.1
> 6300c9000000-6300c9bfffff : PCI Bus 0006:04
> 6300c9000000-6300c9bfffff : PCI Bus 0006:05
> 6300c9000000-6300c91fffff : PCI Bus 0006:06
> 6300c9200000-6300c93fffff : PCI Bus 0006:07
> 6300c9400000-6300c95fffff : PCI Bus 0006:08
> 6300c9600000-6300c97fffff : PCI Bus 0006:09
>
> After the commit:
> 6300c0000000-63ffffffffff : PCI Bus 0006:00
> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
> 6300c0000000-6300c43fffff : PCI Bus 0006:03
> 6300c0000000-6300c1ffffff : 0006:03:00.0
> 6300c0000000-6300c1ffffff : mlx5_core
> 6300c2000000-6300c3ffffff : 0006:03:00.1
> 6300c2000000-6300c3ffffff : mlx5_core
> 6300c4400000-6300c4dfffff : PCI Bus 0006:04
> 6300c4400000-6300c4dfffff : PCI Bus 0006:05
> 6300c4400000-6300c45fffff : PCI Bus 0006:06
> 6300c4600000-6300c47fffff : PCI Bus 0006:07
> 6300c4800000-6300c49fffff : PCI Bus 0006:08
> 6300c4a00000-6300c4bfffff : PCI Bus 0006:09
>
> We can see that the window of 0006:03 gets shrunken too much and 0006:04
> eats away the window for 0006:03:00.2.
>
> The offending commit distributes the upstream bridge's resources
> multiple times to every downstream bridges, hence makes the aperture
> smaller than desired because calculation of io_per_b, mmio_per_b and
> mmio_pref_per_b becomes incorrect.
>
> Instead, distributing downstream bridges' own resources to resolve the
> issue.
>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=219540
> Cc: Carol Soto <csoto@nvidia.com>
> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Cc: Chris Chiu <chris.chiu@canonical.com>
> Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH v2] PCI: Use downstream bridges for distributing resources
2024-12-04 5:48 ` Mika Westerberg
@ 2024-12-18 11:12 ` Kai-Heng Feng
2025-01-20 6:39 ` Kai-Heng Feng
0 siblings, 1 reply; 6+ messages in thread
From: Kai-Heng Feng @ 2024-12-18 11:12 UTC (permalink / raw)
To: bhelgaas
Cc: Mika Westerberg, linux-pci, linux-kernel, Carol Soto,
Jonathan Cameron, Chris Chiu, Chia-Lin Kao
Hi Bjorn,
On 2024/12/4 1:48 PM, Mika Westerberg wrote:
> External email: Use caution opening links or attachments
>
>
> On Wed, Dec 04, 2024 at 10:24:57AM +0800, Kai-Heng Feng wrote:
>> Commit 7180c1d08639 ("PCI: Distribute available resources for root
>> buses, too") breaks BAR assignment on some devcies:
>> [ 10.021193] pci 0006:03:00.0: BAR 0 [mem 0x6300c0000000-0x6300c1ffffff 64bit pref]: assigned
>> [ 10.029880] pci 0006:03:00.1: BAR 0 [mem 0x6300c2000000-0x6300c3ffffff 64bit pref]: assigned
>> [ 10.038561] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: can't assign; no space
>> [ 10.047191] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: failed to assign
>> [ 10.055285] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
>> [ 10.064180] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
>> [ 10.072543] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
>> [ 10.081437] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
>>
>> The apertures of domain 0006 before the commit:
>> 6300c0000000-63ffffffffff : PCI Bus 0006:00
>> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
>> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
>> 6300c0000000-6300c8ffffff : PCI Bus 0006:03
>> 6300c0000000-6300c1ffffff : 0006:03:00.0
>> 6300c0000000-6300c1ffffff : mlx5_core
>> 6300c2000000-6300c3ffffff : 0006:03:00.1
>> 6300c2000000-6300c3ffffff : mlx5_core
>> 6300c4000000-6300c47fffff : 0006:03:00.2
>> 6300c4800000-6300c67fffff : 0006:03:00.0
>> 6300c6800000-6300c87fffff : 0006:03:00.1
>> 6300c9000000-6300c9bfffff : PCI Bus 0006:04
>> 6300c9000000-6300c9bfffff : PCI Bus 0006:05
>> 6300c9000000-6300c91fffff : PCI Bus 0006:06
>> 6300c9200000-6300c93fffff : PCI Bus 0006:07
>> 6300c9400000-6300c95fffff : PCI Bus 0006:08
>> 6300c9600000-6300c97fffff : PCI Bus 0006:09
>>
>> After the commit:
>> 6300c0000000-63ffffffffff : PCI Bus 0006:00
>> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
>> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
>> 6300c0000000-6300c43fffff : PCI Bus 0006:03
>> 6300c0000000-6300c1ffffff : 0006:03:00.0
>> 6300c0000000-6300c1ffffff : mlx5_core
>> 6300c2000000-6300c3ffffff : 0006:03:00.1
>> 6300c2000000-6300c3ffffff : mlx5_core
>> 6300c4400000-6300c4dfffff : PCI Bus 0006:04
>> 6300c4400000-6300c4dfffff : PCI Bus 0006:05
>> 6300c4400000-6300c45fffff : PCI Bus 0006:06
>> 6300c4600000-6300c47fffff : PCI Bus 0006:07
>> 6300c4800000-6300c49fffff : PCI Bus 0006:08
>> 6300c4a00000-6300c4bfffff : PCI Bus 0006:09
>>
>> We can see that the window of 0006:03 gets shrunken too much and 0006:04
>> eats away the window for 0006:03:00.2.
>>
>> The offending commit distributes the upstream bridge's resources
>> multiple times to every downstream bridges, hence makes the aperture
>> smaller than desired because calculation of io_per_b, mmio_per_b and
>> mmio_pref_per_b becomes incorrect.
>>
>> Instead, distributing downstream bridges' own resources to resolve the
>> issue.
>>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=219540
>> Cc: Carol Soto <csoto@nvidia.com>
>> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> Cc: Chris Chiu <chris.chiu@canonical.com>
>> Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
>
> Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Is it possible to pickup this up in 6.13 window?
Kai-Heng
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH v2] PCI: Use downstream bridges for distributing resources
2024-12-18 11:12 ` Kai-Heng Feng
@ 2025-01-20 6:39 ` Kai-Heng Feng
2025-02-13 5:08 ` Kai-Heng Feng
0 siblings, 1 reply; 6+ messages in thread
From: Kai-Heng Feng @ 2025-01-20 6:39 UTC (permalink / raw)
To: bhelgaas
Cc: Mika Westerberg, linux-pci, linux-kernel, Carol Soto,
Jonathan Cameron, Chris Chiu, Chia-Lin Kao
On 2024/12/18 7:12 PM, Kai-Heng Feng wrote:
> Hi Bjorn,
>
> On 2024/12/4 1:48 PM, Mika Westerberg wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> On Wed, Dec 04, 2024 at 10:24:57AM +0800, Kai-Heng Feng wrote:
>>> Commit 7180c1d08639 ("PCI: Distribute available resources for root
>>> buses, too") breaks BAR assignment on some devcies:
>>> [ 10.021193] pci 0006:03:00.0: BAR 0 [mem 0x6300c0000000-0x6300c1ffffff
>>> 64bit pref]: assigned
>>> [ 10.029880] pci 0006:03:00.1: BAR 0 [mem 0x6300c2000000-0x6300c3ffffff
>>> 64bit pref]: assigned
>>> [ 10.038561] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]:
>>> can't assign; no space
>>> [ 10.047191] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]:
>>> failed to assign
>>> [ 10.055285] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>> can't assign; no space
>>> [ 10.064180] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>> failed to assign
>>> [ 10.072543] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>> can't assign; no space
>>> [ 10.081437] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>> failed to assign
>>>
>>> The apertures of domain 0006 before the commit:
>>> 6300c0000000-63ffffffffff : PCI Bus 0006:00
>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
>>> 6300c0000000-6300c8ffffff : PCI Bus 0006:03
>>> 6300c0000000-6300c1ffffff : 0006:03:00.0
>>> 6300c0000000-6300c1ffffff : mlx5_core
>>> 6300c2000000-6300c3ffffff : 0006:03:00.1
>>> 6300c2000000-6300c3ffffff : mlx5_core
>>> 6300c4000000-6300c47fffff : 0006:03:00.2
>>> 6300c4800000-6300c67fffff : 0006:03:00.0
>>> 6300c6800000-6300c87fffff : 0006:03:00.1
>>> 6300c9000000-6300c9bfffff : PCI Bus 0006:04
>>> 6300c9000000-6300c9bfffff : PCI Bus 0006:05
>>> 6300c9000000-6300c91fffff : PCI Bus 0006:06
>>> 6300c9200000-6300c93fffff : PCI Bus 0006:07
>>> 6300c9400000-6300c95fffff : PCI Bus 0006:08
>>> 6300c9600000-6300c97fffff : PCI Bus 0006:09
>>>
>>> After the commit:
>>> 6300c0000000-63ffffffffff : PCI Bus 0006:00
>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
>>> 6300c0000000-6300c43fffff : PCI Bus 0006:03
>>> 6300c0000000-6300c1ffffff : 0006:03:00.0
>>> 6300c0000000-6300c1ffffff : mlx5_core
>>> 6300c2000000-6300c3ffffff : 0006:03:00.1
>>> 6300c2000000-6300c3ffffff : mlx5_core
>>> 6300c4400000-6300c4dfffff : PCI Bus 0006:04
>>> 6300c4400000-6300c4dfffff : PCI Bus 0006:05
>>> 6300c4400000-6300c45fffff : PCI Bus 0006:06
>>> 6300c4600000-6300c47fffff : PCI Bus 0006:07
>>> 6300c4800000-6300c49fffff : PCI Bus 0006:08
>>> 6300c4a00000-6300c4bfffff : PCI Bus 0006:09
>>>
>>> We can see that the window of 0006:03 gets shrunken too much and 0006:04
>>> eats away the window for 0006:03:00.2.
>>>
>>> The offending commit distributes the upstream bridge's resources
>>> multiple times to every downstream bridges, hence makes the aperture
>>> smaller than desired because calculation of io_per_b, mmio_per_b and
>>> mmio_pref_per_b becomes incorrect.
>>>
>>> Instead, distributing downstream bridges' own resources to resolve the
>>> issue.
>>>
>>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=219540
>>> Cc: Carol Soto <csoto@nvidia.com>
>>> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>> Cc: Chris Chiu <chris.chiu@canonical.com>
>>> Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
>>
>> Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>
> Is it possible to pickup this up in 6.13 window?
Now 6.14 window is open, is it possible to include this fix?
Kai-Heng
>
> Kai-Heng
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH v2] PCI: Use downstream bridges for distributing resources
2025-01-20 6:39 ` Kai-Heng Feng
@ 2025-02-13 5:08 ` Kai-Heng Feng
0 siblings, 0 replies; 6+ messages in thread
From: Kai-Heng Feng @ 2025-02-13 5:08 UTC (permalink / raw)
To: bhelgaas
Cc: Mika Westerberg, linux-pci, linux-kernel, Carol Soto,
Jonathan Cameron, Chris Chiu, Chia-Lin Kao, Bjorn Helgaas
On 2025/1/20 2:39 PM, Kai-Heng Feng wrote:
>
>
> On 2024/12/18 7:12 PM, Kai-Heng Feng wrote:
>> Hi Bjorn,
>>
>> On 2024/12/4 1:48 PM, Mika Westerberg wrote:
>>> External email: Use caution opening links or attachments
>>>
>>>
>>> On Wed, Dec 04, 2024 at 10:24:57AM +0800, Kai-Heng Feng wrote:
>>>> Commit 7180c1d08639 ("PCI: Distribute available resources for root
>>>> buses, too") breaks BAR assignment on some devcies:
>>>> [ 10.021193] pci 0006:03:00.0: BAR 0 [mem 0x6300c0000000-0x6300c1ffffff
>>>> 64bit pref]: assigned
>>>> [ 10.029880] pci 0006:03:00.1: BAR 0 [mem 0x6300c2000000-0x6300c3ffffff
>>>> 64bit pref]: assigned
>>>> [ 10.038561] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]:
>>>> can't assign; no space
>>>> [ 10.047191] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]:
>>>> failed to assign
>>>> [ 10.055285] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>>> can't assign; no space
>>>> [ 10.064180] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>>> failed to assign
>>>> [ 10.072543] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>>> can't assign; no space
>>>> [ 10.081437] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]:
>>>> failed to assign
>>>>
>>>> The apertures of domain 0006 before the commit:
>>>> 6300c0000000-63ffffffffff : PCI Bus 0006:00
>>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
>>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
>>>> 6300c0000000-6300c8ffffff : PCI Bus 0006:03
>>>> 6300c0000000-6300c1ffffff : 0006:03:00.0
>>>> 6300c0000000-6300c1ffffff : mlx5_core
>>>> 6300c2000000-6300c3ffffff : 0006:03:00.1
>>>> 6300c2000000-6300c3ffffff : mlx5_core
>>>> 6300c4000000-6300c47fffff : 0006:03:00.2
>>>> 6300c4800000-6300c67fffff : 0006:03:00.0
>>>> 6300c6800000-6300c87fffff : 0006:03:00.1
>>>> 6300c9000000-6300c9bfffff : PCI Bus 0006:04
>>>> 6300c9000000-6300c9bfffff : PCI Bus 0006:05
>>>> 6300c9000000-6300c91fffff : PCI Bus 0006:06
>>>> 6300c9200000-6300c93fffff : PCI Bus 0006:07
>>>> 6300c9400000-6300c95fffff : PCI Bus 0006:08
>>>> 6300c9600000-6300c97fffff : PCI Bus 0006:09
>>>>
>>>> After the commit:
>>>> 6300c0000000-63ffffffffff : PCI Bus 0006:00
>>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
>>>> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
>>>> 6300c0000000-6300c43fffff : PCI Bus 0006:03
>>>> 6300c0000000-6300c1ffffff : 0006:03:00.0
>>>> 6300c0000000-6300c1ffffff : mlx5_core
>>>> 6300c2000000-6300c3ffffff : 0006:03:00.1
>>>> 6300c2000000-6300c3ffffff : mlx5_core
>>>> 6300c4400000-6300c4dfffff : PCI Bus 0006:04
>>>> 6300c4400000-6300c4dfffff : PCI Bus 0006:05
>>>> 6300c4400000-6300c45fffff : PCI Bus 0006:06
>>>> 6300c4600000-6300c47fffff : PCI Bus 0006:07
>>>> 6300c4800000-6300c49fffff : PCI Bus 0006:08
>>>> 6300c4a00000-6300c4bfffff : PCI Bus 0006:09
>>>>
>>>> We can see that the window of 0006:03 gets shrunken too much and 0006:04
>>>> eats away the window for 0006:03:00.2.
>>>>
>>>> The offending commit distributes the upstream bridge's resources
>>>> multiple times to every downstream bridges, hence makes the aperture
>>>> smaller than desired because calculation of io_per_b, mmio_per_b and
>>>> mmio_pref_per_b becomes incorrect.
>>>>
>>>> Instead, distributing downstream bridges' own resources to resolve the
>>>> issue.
>>>>
>>>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=219540
>>>> Cc: Carol Soto <csoto@nvidia.com>
>>>> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>>> Cc: Chris Chiu <chris.chiu@canonical.com>
>>>> Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
>>>
>>> Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>>
>> Is it possible to pickup this up in 6.13 window?
>
> Now 6.14 window is open, is it possible to include this fix?
Another gentle ping...
>
> Kai-Heng
>
>>
>> Kai-Heng
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2] PCI: Use downstream bridges for distributing resources
2024-12-04 2:24 [PATCH v2] PCI: Use downstream bridges for distributing resources Kai-Heng Feng
2024-12-04 5:48 ` Mika Westerberg
@ 2025-02-13 21:21 ` Bjorn Helgaas
1 sibling, 0 replies; 6+ messages in thread
From: Bjorn Helgaas @ 2025-02-13 21:21 UTC (permalink / raw)
To: Kai-Heng Feng
Cc: bhelgaas, mika.westerberg, linux-pci, linux-kernel, Carol Soto,
Jonathan Cameron, Chris Chiu, Chia-Lin Kao
On Wed, Dec 04, 2024 at 10:24:57AM +0800, Kai-Heng Feng wrote:
> Commit 7180c1d08639 ("PCI: Distribute available resources for root
> buses, too") breaks BAR assignment on some devcies:
> [ 10.021193] pci 0006:03:00.0: BAR 0 [mem 0x6300c0000000-0x6300c1ffffff 64bit pref]: assigned
> [ 10.029880] pci 0006:03:00.1: BAR 0 [mem 0x6300c2000000-0x6300c3ffffff 64bit pref]: assigned
> [ 10.038561] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: can't assign; no space
> [ 10.047191] pci 0006:03:00.2: BAR 0 [mem size 0x00800000 64bit pref]: failed to assign
> [ 10.055285] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
> [ 10.064180] pci 0006:03:00.0: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
> [ 10.072543] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: can't assign; no space
> [ 10.081437] pci 0006:03:00.1: VF BAR 0 [mem size 0x02000000 64bit pref]: failed to assign
>
> The apertures of domain 0006 before the commit:
> 6300c0000000-63ffffffffff : PCI Bus 0006:00
> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
> 6300c0000000-6300c8ffffff : PCI Bus 0006:03
> 6300c0000000-6300c1ffffff : 0006:03:00.0
> 6300c0000000-6300c1ffffff : mlx5_core
> 6300c2000000-6300c3ffffff : 0006:03:00.1
> 6300c2000000-6300c3ffffff : mlx5_core
> 6300c4000000-6300c47fffff : 0006:03:00.2
> 6300c4800000-6300c67fffff : 0006:03:00.0
> 6300c6800000-6300c87fffff : 0006:03:00.1
> 6300c9000000-6300c9bfffff : PCI Bus 0006:04
> 6300c9000000-6300c9bfffff : PCI Bus 0006:05
> 6300c9000000-6300c91fffff : PCI Bus 0006:06
> 6300c9200000-6300c93fffff : PCI Bus 0006:07
> 6300c9400000-6300c95fffff : PCI Bus 0006:08
> 6300c9600000-6300c97fffff : PCI Bus 0006:09
>
> After the commit:
> 6300c0000000-63ffffffffff : PCI Bus 0006:00
> 6300c0000000-6300c9ffffff : PCI Bus 0006:01
> 6300c0000000-6300c9ffffff : PCI Bus 0006:02
> 6300c0000000-6300c43fffff : PCI Bus 0006:03
> 6300c0000000-6300c1ffffff : 0006:03:00.0
> 6300c0000000-6300c1ffffff : mlx5_core
> 6300c2000000-6300c3ffffff : 0006:03:00.1
> 6300c2000000-6300c3ffffff : mlx5_core
> 6300c4400000-6300c4dfffff : PCI Bus 0006:04
> 6300c4400000-6300c4dfffff : PCI Bus 0006:05
> 6300c4400000-6300c45fffff : PCI Bus 0006:06
> 6300c4600000-6300c47fffff : PCI Bus 0006:07
> 6300c4800000-6300c49fffff : PCI Bus 0006:08
> 6300c4a00000-6300c4bfffff : PCI Bus 0006:09
>
> We can see that the window of 0006:03 gets shrunken too much and 0006:04
> eats away the window for 0006:03:00.2.
>
> The offending commit distributes the upstream bridge's resources
> multiple times to every downstream bridges, hence makes the aperture
> smaller than desired because calculation of io_per_b, mmio_per_b and
> mmio_pref_per_b becomes incorrect.
>
> Instead, distributing downstream bridges' own resources to resolve the
> issue.
>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=219540
> Cc: Carol Soto <csoto@nvidia.com>
> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Cc: Chris Chiu <chris.chiu@canonical.com>
> Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
> Tested-by: Chia-Lin Kao (AceLan) <acelan.kao@canonical.com>
> Fixes: 7180c1d08639 ("PCI: Distribute available resources for root buses, too")
> Signed-off-by: Kai-Heng Feng <kaihengf@nvidia.com>
Applied with Mika's Reviewed-by to pci/resource for v6.15, thanks!
> ---
> drivers/pci/setup-bus.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 23082bc0ca37..a6e653a4f5b1 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -2105,8 +2105,7 @@ pci_root_bus_distribute_available_resources(struct pci_bus *bus,
> * in case of root bus.
> */
> if (bridge && pci_bridge_resources_not_assigned(dev))
> - pci_bridge_distribute_available_resources(bridge,
> - add_list);
> + pci_bridge_distribute_available_resources(dev, add_list);
> else
> pci_root_bus_distribute_available_resources(b, add_list);
> }
> --
> 2.47.0
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-02-13 21:21 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-04 2:24 [PATCH v2] PCI: Use downstream bridges for distributing resources Kai-Heng Feng
2024-12-04 5:48 ` Mika Westerberg
2024-12-18 11:12 ` Kai-Heng Feng
2025-01-20 6:39 ` Kai-Heng Feng
2025-02-13 5:08 ` Kai-Heng Feng
2025-02-13 21:21 ` Bjorn Helgaas
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox