Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Kai-Heng Feng]

[+Cc Sasha and Intel Wired Lan]

On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
> > On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
> > > > <snip>
> > > > > > A variety of Intel chipsets don't support lane width switching
> > > > > > or speed switching.  When ASPM has been enabled on a dGPU,
> > > > > > these features are utilized and breakage ensues.
> > > > >
> > > > > Maybe this helps explain all the completely unmaintainable ASPM
> > > > > garbage in amdgpu and radeon.
> > > > >
> > > > > If these devices are broken, we need quirks for them.
> > > >
> > > > The problem is which device do you consider "broken"?
> > > > The dGPU that uses these features when the platform advertises ASPM
> > > > or the chipset which doesn't support the features that the device
> > > > uses when ASPM is active?
> > > >
> > > > With this problem I'm talking about the dGPU works fine on hosts
> > > > that support these features.
> > >
> > > Without more details about what's broken and when, I can't say.  What
> > > I *think* is that a device that doesn't work per spec needs a quirk.
> > > Typically it's a device that advertises a capability that doesn't work
> > > correctly.
> >
> > Many silicon vendors use the same "PCI IP" and integrate it into their
> > own chip. That's one of the reason why the capability doesn't really
> > reflect what actually being support.
> > The vendors then send their private datasheet to system integrator so
> > BIOS can enable what's really supported.
>
> It's perfectly fine for the IP to support PCI features that are not
> and can not be enabled in a system design.  But I expect that
> strapping or firmware would disable those features so they are not
> advertised in config space.
>
> If BIOS leaves features disabled because they cannot work, but at the
> same time leaves them advertised in config space, I'd say that's a
> BIOS defect.  In that case, we should have a DMI quirk or something to
> work around the defect.

That means most if not all BIOS are defected.
BIOS vendors and ODM never bothered (and probably will not) to change
the capabilities advertised by config space because "it already works
under Windows".

>
> > So the logic is to ignore the capability and trust the default set
> > by BIOS.
>
> I think limiting ASPM support to whatever BIOS configured at boot-time
> is problematic.  I don't think we can assume that all platforms have
> firmware that configures ASPM as aggressively as possible, and
> obviously firmware won't configure hot-added devices at all (in
> general; I know ACPI _HPX can do some of that).

Totally agree. I was not suggesting to limiting the setting at all.
A boot-time parameter to flip ASPM setting is very useful. If none has
been set, default to BIOS setting.

>
> It's possible we need some kind of policy that limits ASPM to the BIOS
> config until date X, but I don't want to limit that forever.

Desktop can have very different PCIe cards in the slots so BIOS date
isn't a good indicator for ASPM support.
Let alone BIOS date changes on each BIOS upgrade, this means potential
regression risk.

>
> > > > If you follow my idea of hot added devices the policy follows
> > > > the parent would it work for the i225 PCIe device case?
> > >
> > > That doesn't *sound* really robust to me because even if the default
> > > config after hot-add works, the user can change things via sysfs, and
> > > any configuration we set it to should work as well.  If there are
> > > land-mines there, we need a quirk that prevents sysfs from running
> > > into it.
> >
> > For this case it means driver needs to provide a ASPM callback to flip
> > PTM based on ASPM sysfs.
>
> I'm not sure I follow this, but it sounds like you're saying PTM
> doesn't work correctly with some ASPM configurations.  Is this allowed
> by the PCIe spec or is it a hardware defect?  If the latter, maybe we
> just need a quirk to disable PTM on the device.

I'll leave this one to Intel folks.

>
> I'm not sure PTM is valuable enough to add a generic callback
> mechanism to work around a defect in an individual device.

Agree. Right now the issue is only observed when I225 is in a TBT dock.

Kai-Heng

>
> Bjorn
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Bjorn Helgaas]

On Wed, Jun 28, 2023 at 01:09:49PM +0800, Kai-Heng Feng wrote:
> On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
> > > On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:

> > It's perfectly fine for the IP to support PCI features that are not
> > and can not be enabled in a system design.  But I expect that
> > strapping or firmware would disable those features so they are not
> > advertised in config space.
> >
> > If BIOS leaves features disabled because they cannot work, but at the
> > same time leaves them advertised in config space, I'd say that's a
> > BIOS defect.  In that case, we should have a DMI quirk or something to
> > work around the defect.
> 
> That means most if not all BIOS are defected.
> BIOS vendors and ODM never bothered (and probably will not) to change
> the capabilities advertised by config space because "it already works
> under Windows".

This is what seems strange to me.  Are you saying that Windows never
enables these power-saving features?  Or that Windows includes quirks
for all these broken BIOSes?  Neither idea seems very convincing.

> > > So the logic is to ignore the capability and trust the default set
> > > by BIOS.
> >
> > I think limiting ASPM support to whatever BIOS configured at boot-time
> > is problematic.  I don't think we can assume that all platforms have
> > firmware that configures ASPM as aggressively as possible, and
> > obviously firmware won't configure hot-added devices at all (in
> > general; I know ACPI _HPX can do some of that).
> 
> Totally agree. I was not suggesting to limiting the setting at all.
> A boot-time parameter to flip ASPM setting is very useful. If none has
> been set, default to BIOS setting.

A boot-time parameter for debugging and workarounds is fine.  IMO,
needing a boot-time parameter in the course of normal operation is
not OK.

Bjorn
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Mario Limonciello]

On 7/5/23 15:06, Bjorn Helgaas wrote:
> On Wed, Jun 28, 2023 at 01:09:49PM +0800, Kai-Heng Feng wrote:
>> On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>> On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
>>>> On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>>>> On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
> 
>>> It's perfectly fine for the IP to support PCI features that are not
>>> and can not be enabled in a system design.  But I expect that
>>> strapping or firmware would disable those features so they are not
>>> advertised in config space.
>>>
>>> If BIOS leaves features disabled because they cannot work, but at the
>>> same time leaves them advertised in config space, I'd say that's a
>>> BIOS defect.  In that case, we should have a DMI quirk or something to
>>> work around the defect.
>>
>> That means most if not all BIOS are defected.
>> BIOS vendors and ODM never bothered (and probably will not) to change
>> the capabilities advertised by config space because "it already works
>> under Windows".
> 
> This is what seems strange to me.  Are you saying that Windows never
> enables these power-saving features?  Or that Windows includes quirks
> for all these broken BIOSes?  Neither idea seems very convincing.
> 

I see your point.  I was looking through Microsoft documentation for 
hints and came across this:

https://learn.microsoft.com/en-us/windows-hardware/customize/power-settings/pci-express-settings-link-state-power-management

They have a policy knob to globally set L0 or L1 for PCIe links.

They don't explicitly say it, but surely it's based on what the devices 
advertise in the capabilities registers.

>>>> So the logic is to ignore the capability and trust the default set
>>>> by BIOS.
>>>
>>> I think limiting ASPM support to whatever BIOS configured at boot-time
>>> is problematic.  I don't think we can assume that all platforms have
>>> firmware that configures ASPM as aggressively as possible, and
>>> obviously firmware won't configure hot-added devices at all (in
>>> general; I know ACPI _HPX can do some of that).
>>
>> Totally agree. I was not suggesting to limiting the setting at all.
>> A boot-time parameter to flip ASPM setting is very useful. If none has
>> been set, default to BIOS setting.
> 
> A boot-time parameter for debugging and workarounds is fine.  IMO,
> needing a boot-time parameter in the course of normal operation is
> not OK.
> 
> Bjorn

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Kai-Heng Feng]

On Thu, Jul 6, 2023 at 12:07 PM Mario Limonciello
<mario.limonciello@amd.com> wrote:
>
> On 7/5/23 15:06, Bjorn Helgaas wrote:
> > On Wed, Jun 28, 2023 at 01:09:49PM +0800, Kai-Heng Feng wrote:
> >> On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >>> On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
> >>>> On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >>>>> On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
> >
> >>> It's perfectly fine for the IP to support PCI features that are not
> >>> and can not be enabled in a system design.  But I expect that
> >>> strapping or firmware would disable those features so they are not
> >>> advertised in config space.
> >>>
> >>> If BIOS leaves features disabled because they cannot work, but at the
> >>> same time leaves them advertised in config space, I'd say that's a
> >>> BIOS defect.  In that case, we should have a DMI quirk or something to
> >>> work around the defect.
> >>
> >> That means most if not all BIOS are defected.
> >> BIOS vendors and ODM never bothered (and probably will not) to change
> >> the capabilities advertised by config space because "it already works
> >> under Windows".
> >
> > This is what seems strange to me.  Are you saying that Windows never
> > enables these power-saving features?  Or that Windows includes quirks
> > for all these broken BIOSes?  Neither idea seems very convincing.
> >
>
> I see your point.  I was looking through Microsoft documentation for
> hints and came across this:
>
> https://learn.microsoft.com/en-us/windows-hardware/customize/power-settings/pci-express-settings-link-state-power-management
>
> They have a policy knob to globally set L0 or L1 for PCIe links.
>
> They don't explicitly say it, but surely it's based on what the devices
> advertise in the capabilities registers.

So essentially it can be achieved via boot time kernel parameter
and/or sysfs knob.

The main point is OS should stick to the BIOS default, which is the
only ASPM setting tested before putting hardware to the market.

Kai-Heng

>
> >>>> So the logic is to ignore the capability and trust the default set
> >>>> by BIOS.
> >>>
> >>> I think limiting ASPM support to whatever BIOS configured at boot-time
> >>> is problematic.  I don't think we can assume that all platforms have
> >>> firmware that configures ASPM as aggressively as possible, and
> >>> obviously firmware won't configure hot-added devices at all (in
> >>> general; I know ACPI _HPX can do some of that).
> >>
> >> Totally agree. I was not suggesting to limiting the setting at all.
> >> A boot-time parameter to flip ASPM setting is very useful. If none has
> >> been set, default to BIOS setting.
> >
> > A boot-time parameter for debugging and workarounds is fine.  IMO,
> > needing a boot-time parameter in the course of normal operation is
> > not OK.
> >
> > Bjorn
>
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Mario Limonciello]

On 7/14/23 03:17, Kai-Heng Feng wrote:
> On Thu, Jul 6, 2023 at 12:07 PM Mario Limonciello
> <mario.limonciello@amd.com> wrote:
>>
>> On 7/5/23 15:06, Bjorn Helgaas wrote:
>>> On Wed, Jun 28, 2023 at 01:09:49PM +0800, Kai-Heng Feng wrote:
>>>> On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>>>> On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
>>>>>> On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>>>>>> On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
>>>
>>>>> It's perfectly fine for the IP to support PCI features that are not
>>>>> and can not be enabled in a system design.  But I expect that
>>>>> strapping or firmware would disable those features so they are not
>>>>> advertised in config space.
>>>>>
>>>>> If BIOS leaves features disabled because they cannot work, but at the
>>>>> same time leaves them advertised in config space, I'd say that's a
>>>>> BIOS defect.  In that case, we should have a DMI quirk or something to
>>>>> work around the defect.
>>>>
>>>> That means most if not all BIOS are defected.
>>>> BIOS vendors and ODM never bothered (and probably will not) to change
>>>> the capabilities advertised by config space because "it already works
>>>> under Windows".
>>>
>>> This is what seems strange to me.  Are you saying that Windows never
>>> enables these power-saving features?  Or that Windows includes quirks
>>> for all these broken BIOSes?  Neither idea seems very convincing.
>>>
>>
>> I see your point.  I was looking through Microsoft documentation for
>> hints and came across this:
>>
>> https://learn.microsoft.com/en-us/windows-hardware/customize/power-settings/pci-express-settings-link-state-power-management
>>
>> They have a policy knob to globally set L0 or L1 for PCIe links.
>>
>> They don't explicitly say it, but surely it's based on what the devices
>> advertise in the capabilities registers.
> 
> So essentially it can be achieved via boot time kernel parameter
> and/or sysfs knob.
> 
> The main point is OS should stick to the BIOS default, which is the
> only ASPM setting tested before putting hardware to the market.

Unfortunately; I don't think you can jump to this conclusion.

A big difference in the Windows world to Linux world is that OEMs ship 
with a factory Windows image that may set policies like this.  OEM 
"platform" drivers can set registry keys too.

I think the next ASPM issue that comes up what we (collectively) need to 
do is compare ASPM policy and PCI registers for:
1) A "clean" Windows install from Microsoft media before all the OEM 
drivers are installed.
2) A Windows install that the drivers have been installed.
3) A up to date mainline Linux kernel.

Actually as this thread started for determining policy for external PCIe 
devices, maybe that would be good to check with those.

> 
> Kai-Heng
> 
>>
>>>>>> So the logic is to ignore the capability and trust the default set
>>>>>> by BIOS.
>>>>>
>>>>> I think limiting ASPM support to whatever BIOS configured at boot-time
>>>>> is problematic.  I don't think we can assume that all platforms have
>>>>> firmware that configures ASPM as aggressively as possible, and
>>>>> obviously firmware won't configure hot-added devices at all (in
>>>>> general; I know ACPI _HPX can do some of that).
>>>>
>>>> Totally agree. I was not suggesting to limiting the setting at all.
>>>> A boot-time parameter to flip ASPM setting is very useful. If none has
>>>> been set, default to BIOS setting.
>>>
>>> A boot-time parameter for debugging and workarounds is fine.  IMO,
>>> needing a boot-time parameter in the course of normal operation is
>>> not OK.
>>>
>>> Bjorn
>>

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Kai-Heng Feng]

On Sat, Jul 15, 2023 at 12:37 AM Mario Limonciello
<mario.limonciello@amd.com> wrote:
>
> On 7/14/23 03:17, Kai-Heng Feng wrote:
> > On Thu, Jul 6, 2023 at 12:07 PM Mario Limonciello
> > <mario.limonciello@amd.com> wrote:
> >>
> >> On 7/5/23 15:06, Bjorn Helgaas wrote:
> >>> On Wed, Jun 28, 2023 at 01:09:49PM +0800, Kai-Heng Feng wrote:
> >>>> On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >>>>> On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
> >>>>>> On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >>>>>>> On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
> >>>
> >>>>> It's perfectly fine for the IP to support PCI features that are not
> >>>>> and can not be enabled in a system design.  But I expect that
> >>>>> strapping or firmware would disable those features so they are not
> >>>>> advertised in config space.
> >>>>>
> >>>>> If BIOS leaves features disabled because they cannot work, but at the
> >>>>> same time leaves them advertised in config space, I'd say that's a
> >>>>> BIOS defect.  In that case, we should have a DMI quirk or something to
> >>>>> work around the defect.
> >>>>
> >>>> That means most if not all BIOS are defected.
> >>>> BIOS vendors and ODM never bothered (and probably will not) to change
> >>>> the capabilities advertised by config space because "it already works
> >>>> under Windows".
> >>>
> >>> This is what seems strange to me.  Are you saying that Windows never
> >>> enables these power-saving features?  Or that Windows includes quirks
> >>> for all these broken BIOSes?  Neither idea seems very convincing.
> >>>
> >>
> >> I see your point.  I was looking through Microsoft documentation for
> >> hints and came across this:
> >>
> >> https://learn.microsoft.com/en-us/windows-hardware/customize/power-settings/pci-express-settings-link-state-power-management
> >>
> >> They have a policy knob to globally set L0 or L1 for PCIe links.
> >>
> >> They don't explicitly say it, but surely it's based on what the devices
> >> advertise in the capabilities registers.
> >
> > So essentially it can be achieved via boot time kernel parameter
> > and/or sysfs knob.
> >
> > The main point is OS should stick to the BIOS default, which is the
> > only ASPM setting tested before putting hardware to the market.
>
> Unfortunately; I don't think you can jump to this conclusion.
>
> A big difference in the Windows world to Linux world is that OEMs ship
> with a factory Windows image that may set policies like this.  OEM
> "platform" drivers can set registry keys too.

Thanks. This is new to me.

>
> I think the next ASPM issue that comes up what we (collectively) need to
> do is compare ASPM policy and PCI registers for:
> 1) A "clean" Windows install from Microsoft media before all the OEM
> drivers are installed.
> 2) A Windows install that the drivers have been installed.
> 3) A up to date mainline Linux kernel.
>
> Actually as this thread started for determining policy for external PCIe
> devices, maybe that would be good to check with those.

Did that before submitting the patch.
From very limited devices I tested, ASPM is enabled for external
connected PCIe device via TBT ports.

I wonder if there's any particular modification should be improved for
this patch?

Kai-Heng

>
> >
> > Kai-Heng
> >
> >>
> >>>>>> So the logic is to ignore the capability and trust the default set
> >>>>>> by BIOS.
> >>>>>
> >>>>> I think limiting ASPM support to whatever BIOS configured at boot-time
> >>>>> is problematic.  I don't think we can assume that all platforms have
> >>>>> firmware that configures ASPM as aggressively as possible, and
> >>>>> obviously firmware won't configure hot-added devices at all (in
> >>>>> general; I know ACPI _HPX can do some of that).
> >>>>
> >>>> Totally agree. I was not suggesting to limiting the setting at all.
> >>>> A boot-time parameter to flip ASPM setting is very useful. If none has
> >>>> been set, default to BIOS setting.
> >>>
> >>> A boot-time parameter for debugging and workarounds is fine.  IMO,
> >>> needing a boot-time parameter in the course of normal operation is
> >>> not OK.
> >>>
> >>> Bjorn
> >>
>
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Limonciello, Mario]

On 7/16/2023 10:34 PM, Kai-Heng Feng wrote:
> On Sat, Jul 15, 2023 at 12:37 AM Mario Limonciello
> <mario.limonciello@amd.com> wrote:
>>
>> On 7/14/23 03:17, Kai-Heng Feng wrote:
>>> On Thu, Jul 6, 2023 at 12:07 PM Mario Limonciello
>>> <mario.limonciello@amd.com> wrote:
>>>>
>>>> On 7/5/23 15:06, Bjorn Helgaas wrote:
>>>>> On Wed, Jun 28, 2023 at 01:09:49PM +0800, Kai-Heng Feng wrote:
>>>>>> On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>>>>>> On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
>>>>>>>> On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>>>>>>>> On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
>>>>>
>>>>>>> It's perfectly fine for the IP to support PCI features that are not
>>>>>>> and can not be enabled in a system design.  But I expect that
>>>>>>> strapping or firmware would disable those features so they are not
>>>>>>> advertised in config space.
>>>>>>>
>>>>>>> If BIOS leaves features disabled because they cannot work, but at the
>>>>>>> same time leaves them advertised in config space, I'd say that's a
>>>>>>> BIOS defect.  In that case, we should have a DMI quirk or something to
>>>>>>> work around the defect.
>>>>>>
>>>>>> That means most if not all BIOS are defected.
>>>>>> BIOS vendors and ODM never bothered (and probably will not) to change
>>>>>> the capabilities advertised by config space because "it already works
>>>>>> under Windows".
>>>>>
>>>>> This is what seems strange to me.  Are you saying that Windows never
>>>>> enables these power-saving features?  Or that Windows includes quirks
>>>>> for all these broken BIOSes?  Neither idea seems very convincing.
>>>>>
>>>>
>>>> I see your point.  I was looking through Microsoft documentation for
>>>> hints and came across this:
>>>>
>>>> https://learn.microsoft.com/en-us/windows-hardware/customize/power-settings/pci-express-settings-link-state-power-management
>>>>
>>>> They have a policy knob to globally set L0 or L1 for PCIe links.
>>>>
>>>> They don't explicitly say it, but surely it's based on what the devices
>>>> advertise in the capabilities registers.
>>>
>>> So essentially it can be achieved via boot time kernel parameter
>>> and/or sysfs knob.
>>>
>>> The main point is OS should stick to the BIOS default, which is the
>>> only ASPM setting tested before putting hardware to the market.
>>
>> Unfortunately; I don't think you can jump to this conclusion.
>>
>> A big difference in the Windows world to Linux world is that OEMs ship
>> with a factory Windows image that may set policies like this.  OEM
>> "platform" drivers can set registry keys too.
> 
> Thanks. This is new to me.
> 
>>
>> I think the next ASPM issue that comes up what we (collectively) need to
>> do is compare ASPM policy and PCI registers for:
>> 1) A "clean" Windows install from Microsoft media before all the OEM
>> drivers are installed.
>> 2) A Windows install that the drivers have been installed.
>> 3) A up to date mainline Linux kernel.
>>
>> Actually as this thread started for determining policy for external PCIe
>> devices, maybe that would be good to check with those.
> 
> Did that before submitting the patch.
>  From very limited devices I tested, ASPM is enabled for external
> connected PCIe device via TBT ports.
> 
> I wonder if there's any particular modification should be improved for
> this patch?
> 

Knowing this information I personally think the original patch that 
started this thread makes a lot of sense.

Bjorn, what are your thoughts?

> Kai-Heng
> 
>>
>>>
>>> Kai-Heng
>>>
>>>>
>>>>>>>> So the logic is to ignore the capability and trust the default set
>>>>>>>> by BIOS.
>>>>>>>
>>>>>>> I think limiting ASPM support to whatever BIOS configured at boot-time
>>>>>>> is problematic.  I don't think we can assume that all platforms have
>>>>>>> firmware that configures ASPM as aggressively as possible, and
>>>>>>> obviously firmware won't configure hot-added devices at all (in
>>>>>>> general; I know ACPI _HPX can do some of that).
>>>>>>
>>>>>> Totally agree. I was not suggesting to limiting the setting at all.
>>>>>> A boot-time parameter to flip ASPM setting is very useful. If none has
>>>>>> been set, default to BIOS setting.
>>>>>
>>>>> A boot-time parameter for debugging and workarounds is fine.  IMO,
>>>>> needing a boot-time parameter in the course of normal operation is
>>>>> not OK.
>>>>>
>>>>> Bjorn
>>>>
>>
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Bjorn Helgaas]

On Mon, Jul 17, 2023 at 11:51:32AM -0500, Limonciello, Mario wrote:
> On 7/16/2023 10:34 PM, Kai-Heng Feng wrote:
> > On Sat, Jul 15, 2023 at 12:37 AM Mario Limonciello <mario.limonciello@amd.com> wrote:
> > > On 7/14/23 03:17, Kai-Heng Feng wrote:

> > > > The main point is OS should stick to the BIOS default, which is the
> > > > only ASPM setting tested before putting hardware to the market.
> > > 
> > > Unfortunately; I don't think you can jump to this conclusion.

I think using the BIOS default as a limit is problematic.  I think it
would be perfectly reasonable for a BIOS to (a) configure only devices
it needs for console and boot, leaving others at power-on defaults,
and (b) configure devices in the safest configuration possible on the
assumption that an OS can decide the runtime policy itself.

Obviously I'm not a BIOS writer (though I sure wish I could talk to
some!), so maybe these are bad assumptions.

> > > A big difference in the Windows world to Linux world is that OEMs ship
> > > with a factory Windows image that may set policies like this.  OEM
> > > "platform" drivers can set registry keys too.

I suppose this means that the OEM image contains drivers that aren't
in the Microsoft media, and those drivers may set constraints on ASPM
usage?

If you boot the Microsoft media that lacks those drivers, maybe it
doesn't bother to configure ASPM for those devices?  Linux currently
configures ASPM for everything at enumeration-time, so we do it even
if there's no driver.

> > I wonder if there's any particular modification should be improved for
> > this patch?
> 
> Knowing this information I personally think the original patch that started
> this thread makes a lot of sense.

I'm still opposed to using dev_is_removable() as a predicate because I
don't think it has any technical connection to ASPM configuration.

Bjorn
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices

[Kai-Heng Feng]

On Wed, Jul 19, 2023 at 3:24 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Mon, Jul 17, 2023 at 11:51:32AM -0500, Limonciello, Mario wrote:
> > On 7/16/2023 10:34 PM, Kai-Heng Feng wrote:
> > > On Sat, Jul 15, 2023 at 12:37 AM Mario Limonciello <mario.limonciello@amd.com> wrote:
> > > > On 7/14/23 03:17, Kai-Heng Feng wrote:
>
> > > > > The main point is OS should stick to the BIOS default, which is the
> > > > > only ASPM setting tested before putting hardware to the market.
> > > >
> > > > Unfortunately; I don't think you can jump to this conclusion.
>
> I think using the BIOS default as a limit is problematic.  I think it
> would be perfectly reasonable for a BIOS to (a) configure only devices
> it needs for console and boot, leaving others at power-on defaults,
> and (b) configure devices in the safest configuration possible on the
> assumption that an OS can decide the runtime policy itself.

This is not using BIOS as a "limit". OS is still capable of changing
the ASPM policy at boot time or runtime.
The main point is to find a "sane" setting for devices where BIOS
can't program ASPM.

>
> Obviously I'm not a BIOS writer (though I sure wish I could talk to
> some!), so maybe these are bad assumptions.
>
> > > > A big difference in the Windows world to Linux world is that OEMs ship
> > > > with a factory Windows image that may set policies like this.  OEM
> > > > "platform" drivers can set registry keys too.
>
> I suppose this means that the OEM image contains drivers that aren't
> in the Microsoft media, and those drivers may set constraints on ASPM
> usage?
>
> If you boot the Microsoft media that lacks those drivers, maybe it
> doesn't bother to configure ASPM for those devices?  Linux currently
> configures ASPM for everything at enumeration-time, so we do it even
> if there's no driver.

This can be another topic to explore. But sounds like it can break things.

>
> > > I wonder if there's any particular modification should be improved for
> > > this patch?
> >
> > Knowing this information I personally think the original patch that started
> > this thread makes a lot of sense.
>
> I'm still opposed to using dev_is_removable() as a predicate because I
> don't think it has any technical connection to ASPM configuration.

OK. So what should we do instead? Checking if the device is connected
to TBT switch?

Kai-Heng


>
> Bjorn
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan