[PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Johan Hovold]

Add a helper for enabling link states that can be used in contexts where
a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).

This helper will be used to fix a couple of potential deadlocks where
the current helper is called with the lock already held, hence the CC
stable tag.

Fixes: f492edb40b54 ("PCI: vmd: Add quirk to configure PCIe ASPM and LTR")
Cc: stable@vger.kernel.org	# 6.3
Cc: Michael Bottini <michael.a.bottini@linux.intel.com>
Cc: David E. Box <david.e.box@linux.intel.com>
Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
---
 drivers/pci/pcie/aspm.c | 53 +++++++++++++++++++++++++++++++----------
 include/linux/pci.h     |  3 +++
 2 files changed, 43 insertions(+), 13 deletions(-)

diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
index 50b04ae5c394..5eb462772354 100644
--- a/drivers/pci/pcie/aspm.c
+++ b/drivers/pci/pcie/aspm.c
@@ -1109,17 +1109,7 @@ int pci_disable_link_state(struct pci_dev *pdev, int state)
 }
 EXPORT_SYMBOL(pci_disable_link_state);
 
-/**
- * pci_enable_link_state - Clear and set the default device link state so that
- * the link may be allowed to enter the specified states. Note that if the
- * BIOS didn't grant ASPM control to the OS, this does nothing because we can't
- * touch the LNKCTL register. Also note that this does not enable states
- * disabled by pci_disable_link_state(). Return 0 or a negative errno.
- *
- * @pdev: PCI device
- * @state: Mask of ASPM link states to enable
- */
-int pci_enable_link_state(struct pci_dev *pdev, int state)
+static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool locked)
 {
 	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
 
@@ -1136,7 +1126,8 @@ int pci_enable_link_state(struct pci_dev *pdev, int state)
 		return -EPERM;
 	}
 
-	down_read(&pci_bus_sem);
+	if (!locked)
+		down_read(&pci_bus_sem);
 	mutex_lock(&aspm_lock);
 	link->aspm_default = 0;
 	if (state & PCIE_LINK_STATE_L0S)
@@ -1157,12 +1148,48 @@ int pci_enable_link_state(struct pci_dev *pdev, int state)
 	link->clkpm_default = (state & PCIE_LINK_STATE_CLKPM) ? 1 : 0;
 	pcie_set_clkpm(link, policy_to_clkpm_state(link));
 	mutex_unlock(&aspm_lock);
-	up_read(&pci_bus_sem);
+	if (!locked)
+		up_read(&pci_bus_sem);
 
 	return 0;
 }
+
+/**
+ * pci_enable_link_state - Clear and set the default device link state so that
+ * the link may be allowed to enter the specified states. Note that if the
+ * BIOS didn't grant ASPM control to the OS, this does nothing because we can't
+ * touch the LNKCTL register. Also note that this does not enable states
+ * disabled by pci_disable_link_state(). Return 0 or a negative errno.
+ *
+ * @pdev: PCI device
+ * @state: Mask of ASPM link states to enable
+ */
+int pci_enable_link_state(struct pci_dev *pdev, int state)
+{
+	return __pci_enable_link_state(pdev, state, false);
+}
 EXPORT_SYMBOL(pci_enable_link_state);
 
+/**
+ * pci_enable_link_state_locked - Clear and set the default device link state
+ * so that the link may be allowed to enter the specified states. Note that if
+ * the BIOS didn't grant ASPM control to the OS, this does nothing because we
+ * can't touch the LNKCTL register. Also note that this does not enable states
+ * disabled by pci_disable_link_state(). Return 0 or a negative errno.
+ *
+ * @pdev: PCI device
+ * @state: Mask of ASPM link states to enable
+ *
+ * Context: Caller holds pci_bus_sem read lock.
+ */
+int pci_enable_link_state_locked(struct pci_dev *pdev, int state)
+{
+	lockdep_assert_held_read(&pci_bus_sem);
+
+	return __pci_enable_link_state(pdev, state, true);
+}
+EXPORT_SYMBOL(pci_enable_link_state_locked);
+
 static int pcie_aspm_set_policy(const char *val,
 				const struct kernel_param *kp)
 {
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 60ca768bc867..dea043bc1e38 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1829,6 +1829,7 @@ extern bool pcie_ports_native;
 int pci_disable_link_state(struct pci_dev *pdev, int state);
 int pci_disable_link_state_locked(struct pci_dev *pdev, int state);
 int pci_enable_link_state(struct pci_dev *pdev, int state);
+int pci_enable_link_state_locked(struct pci_dev *pdev, int state);
 void pcie_no_aspm(void);
 bool pcie_aspm_support_enabled(void);
 bool pcie_aspm_enabled(struct pci_dev *pdev);
@@ -1839,6 +1840,8 @@ static inline int pci_disable_link_state_locked(struct pci_dev *pdev, int state)
 { return 0; }
 static inline int pci_enable_link_state(struct pci_dev *pdev, int state)
 { return 0; }
+static inline int pci_enable_link_state_locked(struct pci_dev *pdev, int state)
+{ return 0; }
 static inline void pcie_no_aspm(void) { }
 static inline bool pcie_aspm_support_enabled(void) { return false; }
 static inline bool pcie_aspm_enabled(struct pci_dev *pdev) { return false; }
-- 
2.41.0

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Bjorn Helgaas]

[+cc Kai-Heng]

On Tue, Nov 28, 2023 at 09:15:07AM +0100, Johan Hovold wrote:
> Add a helper for enabling link states that can be used in contexts where
> a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).
> 
> This helper will be used to fix a couple of potential deadlocks where
> the current helper is called with the lock already held, hence the CC
> stable tag.
> 
> Fixes: f492edb40b54 ("PCI: vmd: Add quirk to configure PCIe ASPM and LTR")
> Cc: stable@vger.kernel.org	# 6.3
> Cc: Michael Bottini <michael.a.bottini@linux.intel.com>
> Cc: David E. Box <david.e.box@linux.intel.com>
> Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
> ---
>  drivers/pci/pcie/aspm.c | 53 +++++++++++++++++++++++++++++++----------
>  include/linux/pci.h     |  3 +++
>  2 files changed, 43 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index 50b04ae5c394..5eb462772354 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -1109,17 +1109,7 @@ int pci_disable_link_state(struct pci_dev *pdev, int state)
>  }
>  EXPORT_SYMBOL(pci_disable_link_state);
>  
> -/**
> - * pci_enable_link_state - Clear and set the default device link state so that
> - * the link may be allowed to enter the specified states. Note that if the
> - * BIOS didn't grant ASPM control to the OS, this does nothing because we can't
> - * touch the LNKCTL register. Also note that this does not enable states
> - * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> - *
> - * @pdev: PCI device
> - * @state: Mask of ASPM link states to enable
> - */
> -int pci_enable_link_state(struct pci_dev *pdev, int state)
> +static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool locked)
>  {
>  	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
>  
> @@ -1136,7 +1126,8 @@ int pci_enable_link_state(struct pci_dev *pdev, int state)
>  		return -EPERM;
>  	}
>  
> -	down_read(&pci_bus_sem);
> +	if (!locked)
> +		down_read(&pci_bus_sem);
>  	mutex_lock(&aspm_lock);
>  	link->aspm_default = 0;
>  	if (state & PCIE_LINK_STATE_L0S)
> @@ -1157,12 +1148,48 @@ int pci_enable_link_state(struct pci_dev *pdev, int state)
>  	link->clkpm_default = (state & PCIE_LINK_STATE_CLKPM) ? 1 : 0;
>  	pcie_set_clkpm(link, policy_to_clkpm_state(link));
>  	mutex_unlock(&aspm_lock);
> -	up_read(&pci_bus_sem);
> +	if (!locked)
> +		up_read(&pci_bus_sem);
>  
>  	return 0;
>  }
> +
> +/**
> + * pci_enable_link_state - Clear and set the default device link state so that
> + * the link may be allowed to enter the specified states. Note that if the
> + * BIOS didn't grant ASPM control to the OS, this does nothing because we can't
> + * touch the LNKCTL register. Also note that this does not enable states
> + * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> + *
> + * @pdev: PCI device
> + * @state: Mask of ASPM link states to enable
> + */
> +int pci_enable_link_state(struct pci_dev *pdev, int state)
> +{
> +	return __pci_enable_link_state(pdev, state, false);
> +}
>  EXPORT_SYMBOL(pci_enable_link_state);

As far as I can see, we end up with pci_enable_link_state() defined
but never called and pci_enable_link_state_locked() being called only
by pcie-qcom.c and vmd.c.

Can we just rename pci_enable_link_state() to
pci_enable_link_state_locked() and assert that pci_bus_sem is held, so
we don't end up with a function that's never used?

I hope we can obsolete this whole idea someday.  Using pci_walk_bus()
in qcom and vmd to enable ASPM is an ugly hack to work around this
weird idea that "the OS isn't allowed to enable more ASPM states than
the BIOS did because the BIOS might have left ASPM disabled because it
knows about hardware issues."  More history at
https://lore.kernel.org/linux-pci/20230615070421.1704133-1-kai.heng.feng@canonical.com/T/#u

I think we need to get to a point where Linux enables all supported
ASPM features by default.  If we really think x86 BIOS assumes an
implicit contract that the OS will never enable ASPM more
aggressively, we might need some kind of arch quirk for that.

If we can get there, the qcom use of pci_enable_link_state() could go
away, and the vmd use could be replaced by some kind of "if device is
below VMD, get rid of the legacy x86 ASPM assumption" quirk.

Bjorn

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Johan Hovold]

On Thu, Dec 07, 2023 at 02:47:16PM -0600, Bjorn Helgaas wrote:
> [+cc Kai-Heng]
> 
> On Tue, Nov 28, 2023 at 09:15:07AM +0100, Johan Hovold wrote:
> > Add a helper for enabling link states that can be used in contexts where
> > a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).
> > 
> > This helper will be used to fix a couple of potential deadlocks where
> > the current helper is called with the lock already held, hence the CC
> > stable tag.

> As far as I can see, we end up with pci_enable_link_state() defined
> but never called and pci_enable_link_state_locked() being called only
> by pcie-qcom.c and vmd.c.

Correct, I mentioned this in the cover letter.

> Can we just rename pci_enable_link_state() to
> pci_enable_link_state_locked() and assert that pci_bus_sem is held, so
> we don't end up with a function that's never used?

That would work too. I went with adding a new helper to facilitate
stable backports and to mirror pci_disable_link_state(). The variants
are simple wrappers around the implementation so there's no real cost to
having the unused one.

But it seems like you think there will never be a need to call this
helper outside of pci_walk_bus() and if so we can drop the unlocked
variant right away.

Would you prefer basically squashing the first three patches and mark
the result for stable even though that patch will fail to apply to older
kernels as the Qualcomm bits went into -rc1?

Or should I send a follow-on patch removing the unused helper after
merging this series?

The end-result will be identical.

Johan

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Bjorn Helgaas]

On Fri, Dec 08, 2023 at 09:00:56AM +0100, Johan Hovold wrote:
> On Thu, Dec 07, 2023 at 02:47:16PM -0600, Bjorn Helgaas wrote:
> > On Tue, Nov 28, 2023 at 09:15:07AM +0100, Johan Hovold wrote:
> > > Add a helper for enabling link states that can be used in contexts where
> > > a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).
> > > 
> > > This helper will be used to fix a couple of potential deadlocks where
> > > the current helper is called with the lock already held, hence the CC
> > > stable tag.
> 
> > As far as I can see, we end up with pci_enable_link_state() defined
> > but never called and pci_enable_link_state_locked() being called only
> > by pcie-qcom.c and vmd.c.
> 
> Correct, I mentioned this in the cover letter.

Ah, right.  I really don't like these exported locked/unlocked
interfaces because pci_bus_sem is internal to the PCI core, and the
caller shouldn't need to know or be able to specify whether it is held
or not.  They exist for now, but I think we should try to get rid of
them.

> > Can we just rename pci_enable_link_state() to
> > pci_enable_link_state_locked() and assert that pci_bus_sem is held, so
> > we don't end up with a function that's never used?
> 
> That would work too. I went with adding a new helper to facilitate
> stable backports and to mirror pci_disable_link_state(). The variants
> are simple wrappers around the implementation so there's no real cost to
> having the unused one.

Makes good sense.  There's no real machine cost to the unused one; I'm
more concerned about the human cost here.

> But it seems like you think there will never be a need to call this
> helper outside of pci_walk_bus() and if so we can drop the unlocked
> variant right away.
> 
> Would you prefer basically squashing the first three patches and mark
> the result for stable even though that patch will fail to apply to older
> kernels as the Qualcomm bits went into -rc1?
> 
> Or should I send a follow-on patch removing the unused helper after
> merging this series?

I think you did the right thing.

Bjorn

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Ilpo Järvinen]

On Fri, 8 Dec 2023, Bjorn Helgaas wrote:

> On Fri, Dec 08, 2023 at 09:00:56AM +0100, Johan Hovold wrote:
> > On Thu, Dec 07, 2023 at 02:47:16PM -0600, Bjorn Helgaas wrote:
> > > On Tue, Nov 28, 2023 at 09:15:07AM +0100, Johan Hovold wrote:
> > > > Add a helper for enabling link states that can be used in contexts where
> > > > a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).
> > > > 
> > > > This helper will be used to fix a couple of potential deadlocks where
> > > > the current helper is called with the lock already held, hence the CC
> > > > stable tag.
> > 
> > > As far as I can see, we end up with pci_enable_link_state() defined
> > > but never called and pci_enable_link_state_locked() being called only
> > > by pcie-qcom.c and vmd.c.
> > 
> > Correct, I mentioned this in the cover letter.
> 
> Ah, right.  I really don't like these exported locked/unlocked
> interfaces because pci_bus_sem is internal to the PCI core, and the
> caller shouldn't need to know or be able to specify whether it is held
> or not.  They exist for now, but I think we should try to get rid of
> them.
> 
> > > Can we just rename pci_enable_link_state() to
> > > pci_enable_link_state_locked() and assert that pci_bus_sem is held, so
> > > we don't end up with a function that's never used?
> > 
> > That would work too. I went with adding a new helper to facilitate
> > stable backports and to mirror pci_disable_link_state(). The variants
> > are simple wrappers around the implementation so there's no real cost to
> > having the unused one.
> 
> Makes good sense.  There's no real machine cost to the unused one; I'm
> more concerned about the human cost here.

I know these were already applied but I want to correct one small 
misconcept that seems to be floating around thanks the misleading name...

pci_enable_link_state() is not really a pair/mirror of 
pci_disable_link_state() despite its name. It would be better called
pci_set_default_link_state() to better match what it does.

-- 
 i.

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Kai-Heng Feng]

On Fri, Dec 8, 2023 at 4:47 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Kai-Heng]
>
> On Tue, Nov 28, 2023 at 09:15:07AM +0100, Johan Hovold wrote:
> > Add a helper for enabling link states that can be used in contexts where
> > a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).
> >
> > This helper will be used to fix a couple of potential deadlocks where
> > the current helper is called with the lock already held, hence the CC
> > stable tag.
> >
> > Fixes: f492edb40b54 ("PCI: vmd: Add quirk to configure PCIe ASPM and LTR")
> > Cc: stable@vger.kernel.org    # 6.3
> > Cc: Michael Bottini <michael.a.bottini@linux.intel.com>
> > Cc: David E. Box <david.e.box@linux.intel.com>
> > Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
> > ---
> >  drivers/pci/pcie/aspm.c | 53 +++++++++++++++++++++++++++++++----------
> >  include/linux/pci.h     |  3 +++
> >  2 files changed, 43 insertions(+), 13 deletions(-)
> >
> > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> > index 50b04ae5c394..5eb462772354 100644
> > --- a/drivers/pci/pcie/aspm.c
> > +++ b/drivers/pci/pcie/aspm.c
> > @@ -1109,17 +1109,7 @@ int pci_disable_link_state(struct pci_dev *pdev, int state)
> >  }
> >  EXPORT_SYMBOL(pci_disable_link_state);
> >
> > -/**
> > - * pci_enable_link_state - Clear and set the default device link state so that
> > - * the link may be allowed to enter the specified states. Note that if the
> > - * BIOS didn't grant ASPM control to the OS, this does nothing because we can't
> > - * touch the LNKCTL register. Also note that this does not enable states
> > - * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> > - *
> > - * @pdev: PCI device
> > - * @state: Mask of ASPM link states to enable
> > - */
> > -int pci_enable_link_state(struct pci_dev *pdev, int state)
> > +static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool locked)
> >  {
> >       struct pcie_link_state *link = pcie_aspm_get_link(pdev);
> >
> > @@ -1136,7 +1126,8 @@ int pci_enable_link_state(struct pci_dev *pdev, int state)
> >               return -EPERM;
> >       }
> >
> > -     down_read(&pci_bus_sem);
> > +     if (!locked)
> > +             down_read(&pci_bus_sem);
> >       mutex_lock(&aspm_lock);
> >       link->aspm_default = 0;
> >       if (state & PCIE_LINK_STATE_L0S)
> > @@ -1157,12 +1148,48 @@ int pci_enable_link_state(struct pci_dev *pdev, int state)
> >       link->clkpm_default = (state & PCIE_LINK_STATE_CLKPM) ? 1 : 0;
> >       pcie_set_clkpm(link, policy_to_clkpm_state(link));
> >       mutex_unlock(&aspm_lock);
> > -     up_read(&pci_bus_sem);
> > +     if (!locked)
> > +             up_read(&pci_bus_sem);
> >
> >       return 0;
> >  }
> > +
> > +/**
> > + * pci_enable_link_state - Clear and set the default device link state so that
> > + * the link may be allowed to enter the specified states. Note that if the
> > + * BIOS didn't grant ASPM control to the OS, this does nothing because we can't
> > + * touch the LNKCTL register. Also note that this does not enable states
> > + * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> > + *
> > + * @pdev: PCI device
> > + * @state: Mask of ASPM link states to enable
> > + */
> > +int pci_enable_link_state(struct pci_dev *pdev, int state)
> > +{
> > +     return __pci_enable_link_state(pdev, state, false);
> > +}
> >  EXPORT_SYMBOL(pci_enable_link_state);
>
> As far as I can see, we end up with pci_enable_link_state() defined
> but never called and pci_enable_link_state_locked() being called only
> by pcie-qcom.c and vmd.c.
>
> Can we just rename pci_enable_link_state() to
> pci_enable_link_state_locked() and assert that pci_bus_sem is held, so
> we don't end up with a function that's never used?
>
> I hope we can obsolete this whole idea someday.  Using pci_walk_bus()
> in qcom and vmd to enable ASPM is an ugly hack to work around this
> weird idea that "the OS isn't allowed to enable more ASPM states than
> the BIOS did because the BIOS might have left ASPM disabled because it
> knows about hardware issues."  More history at
> https://lore.kernel.org/linux-pci/20230615070421.1704133-1-kai.heng.feng@canonical.com/T/#u
>
> I think we need to get to a point where Linux enables all supported
> ASPM features by default.  If we really think x86 BIOS assumes an
> implicit contract that the OS will never enable ASPM more
> aggressively, we might need some kind of arch quirk for that.

The reality is that PC ODM toggles ASPM to workaround hardware
defects, assuming that OS will honor what's set by the BIOS.
If ASPM gets enabled for all devices, many devices will break.

Kai-Heng

>
> If we can get there, the qcom use of pci_enable_link_state() could go
> away, and the vmd use could be replaced by some kind of "if device is
> below VMD, get rid of the legacy x86 ASPM assumption" quirk.
>
> Bjorn

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Bjorn Helgaas]

On Tue, Dec 12, 2023 at 11:48:27AM +0800, Kai-Heng Feng wrote:
> On Fri, Dec 8, 2023 at 4:47 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> ...

> > I hope we can obsolete this whole idea someday.  Using pci_walk_bus()
> > in qcom and vmd to enable ASPM is an ugly hack to work around this
> > weird idea that "the OS isn't allowed to enable more ASPM states than
> > the BIOS did because the BIOS might have left ASPM disabled because it
> > knows about hardware issues."  More history at
> > https://lore.kernel.org/linux-pci/20230615070421.1704133-1-kai.heng.feng@canonical.com/T/#u
> >
> > I think we need to get to a point where Linux enables all supported
> > ASPM features by default.  If we really think x86 BIOS assumes an
> > implicit contract that the OS will never enable ASPM more
> > aggressively, we might need some kind of arch quirk for that.
> 
> The reality is that PC ODM toggles ASPM to workaround hardware
> defects, assuming that OS will honor what's set by the BIOS.
> If ASPM gets enabled for all devices, many devices will break.

That's why I mentioned some kind of arch quirk.  Maybe we're forced to
do that for x86, for instance.  But even that is a stop-gap.

The idea that the BIOS ASPM config is some kind of handoff protocol is
really unsupportable.

Do we have concrete examples of where enabling ASPM for a device that
advertises ASPM support will break something?

Bjorn

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[David E. Box]

On Tue, 2023-12-12 at 15:27 -0600, Bjorn Helgaas wrote:
> On Tue, Dec 12, 2023 at 11:48:27AM +0800, Kai-Heng Feng wrote:
> > On Fri, Dec 8, 2023 at 4:47 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > ...
> 
> > > I hope we can obsolete this whole idea someday.  Using pci_walk_bus()
> > > in qcom and vmd to enable ASPM is an ugly hack to work around this
> > > weird idea that "the OS isn't allowed to enable more ASPM states than
> > > the BIOS did because the BIOS might have left ASPM disabled because it
> > > knows about hardware issues."  More history at
> > > https://lore.kernel.org/linux-pci/20230615070421.1704133-1-kai.heng.feng@canonical.com/T/#u
> > > 
> > > I think we need to get to a point where Linux enables all supported
> > > ASPM features by default.  If we really think x86 BIOS assumes an
> > > implicit contract that the OS will never enable ASPM more
> > > aggressively, we might need some kind of arch quirk for that.
> > 
> > The reality is that PC ODM toggles ASPM to workaround hardware
> > defects, assuming that OS will honor what's set by the BIOS.
> > If ASPM gets enabled for all devices, many devices will break.
> 
> That's why I mentioned some kind of arch quirk.  Maybe we're forced to
> do that for x86, for instance.  But even that is a stop-gap.
> 
> The idea that the BIOS ASPM config is some kind of handoff protocol is
> really unsupportable.

To be clear, you are not talking about a situation where ACPI_FADT_NO_ASPM or
_OSC PCIe disallow OS ASPM control, right? Everyone agrees that this should be
honored? The question is what to do by default when the OS is not restricted by
these mechanisms?

Reading the mentioned thread, I too think that using the BIOS config as the
default would be the safest option, but only to avoid breaking systems, not
because of an implied contract between the BIOS and OS. However, enabling all
capable ASPM features is the ideal option. If the OS isn't limited by
ACPI_FADT_NO_ASPM or _OSC PCIe, then ASPM enabling is fully under its control.
If this doesn't work for some devices then they are broken and need a quirk.

David

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Bjorn Helgaas]

On Wed, Dec 13, 2023 at 11:48:41AM -0800, David E. Box wrote:
> On Tue, 2023-12-12 at 15:27 -0600, Bjorn Helgaas wrote:
> > On Tue, Dec 12, 2023 at 11:48:27AM +0800, Kai-Heng Feng wrote:
> > > On Fri, Dec 8, 2023 at 4:47 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > ...
> > 
> > > > I hope we can obsolete this whole idea someday.  Using pci_walk_bus()
> > > > in qcom and vmd to enable ASPM is an ugly hack to work around this
> > > > weird idea that "the OS isn't allowed to enable more ASPM states than
> > > > the BIOS did because the BIOS might have left ASPM disabled because it
> > > > knows about hardware issues."  More history at
> > > > https://lore.kernel.org/linux-pci/20230615070421.1704133-1-kai.heng.feng@canonical.com/T/#u
> > > > 
> > > > I think we need to get to a point where Linux enables all supported
> > > > ASPM features by default.  If we really think x86 BIOS assumes an
> > > > implicit contract that the OS will never enable ASPM more
> > > > aggressively, we might need some kind of arch quirk for that.
> > > 
> > > The reality is that PC ODM toggles ASPM to workaround hardware
> > > defects, assuming that OS will honor what's set by the BIOS.
> > > If ASPM gets enabled for all devices, many devices will break.
> > 
> > That's why I mentioned some kind of arch quirk.  Maybe we're forced to
> > do that for x86, for instance.  But even that is a stop-gap.
> > 
> > The idea that the BIOS ASPM config is some kind of handoff protocol is
> > really unsupportable.
> 
> To be clear, you are not talking about a situation where
> ACPI_FADT_NO_ASPM or _OSC PCIe disallow OS ASPM control, right?
> Everyone agrees that this should be honored? The question is what to
> do by default when the OS is not restricted by these mechanisms?

Exactly.  The OS should respect ACPI_FADT_NO_ASPM and _OSC.

I think there are a couple exceptions where we want to disable ASPM
even if the platform said the OS shouldn't touch ASPM at all, but
that's a special case.

> Reading the mentioned thread, I too think that using the BIOS config
> as the default would be the safest option, but only to avoid
> breaking systems, not because of an implied contract between the
> BIOS and OS. However, enabling all capable ASPM features is the
> ideal option. If the OS isn't limited by ACPI_FADT_NO_ASPM or _OSC
> PCIe, then ASPM enabling is fully under its control.  If this
> doesn't work for some devices then they are broken and need a quirk.

Agreed.  It may not be practical to identify such devices, so we may
need a broader arch-based and/or date-based quirk.

I'd be shocked if Windows treated the BIOS config as a "do not exceed
this" situation, so my secret hope is that some of these "broken"
devices are really caused by defects in the Linux ASPM support or the
driver, and that we can fix them if we find out about them.

But I have no details about any of these alleged broken devices, so
it's hard to make progress on them.  Maybe we should log a debug note
if the device advertises ASPM support that BIOS didn't enable.

Bjorn

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[David E. Box]

On Wed, 2023-12-13 at 14:45 -0600, Bjorn Helgaas wrote:
> On Wed, Dec 13, 2023 at 11:48:41AM -0800, David E. Box wrote:
> > On Tue, 2023-12-12 at 15:27 -0600, Bjorn Helgaas wrote:
> > > On Tue, Dec 12, 2023 at 11:48:27AM +0800, Kai-Heng Feng wrote:
> > > > On Fri, Dec 8, 2023 at 4:47 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > ...
> > > 
> > > > > I hope we can obsolete this whole idea someday.  Using pci_walk_bus()
> > > > > in qcom and vmd to enable ASPM is an ugly hack to work around this
> > > > > weird idea that "the OS isn't allowed to enable more ASPM states than
> > > > > the BIOS did because the BIOS might have left ASPM disabled because it
> > > > > knows about hardware issues."  More history at
> > > > > https://lore.kernel.org/linux-pci/20230615070421.1704133-1-kai.heng.feng@canonical.com/T/#u
> > > > > 
> > > > > I think we need to get to a point where Linux enables all supported
> > > > > ASPM features by default.  If we really think x86 BIOS assumes an
> > > > > implicit contract that the OS will never enable ASPM more
> > > > > aggressively, we might need some kind of arch quirk for that.
> > > > 
> > > > The reality is that PC ODM toggles ASPM to workaround hardware
> > > > defects, assuming that OS will honor what's set by the BIOS.
> > > > If ASPM gets enabled for all devices, many devices will break.
> > > 
> > > That's why I mentioned some kind of arch quirk.  Maybe we're forced to
> > > do that for x86, for instance.  But even that is a stop-gap.
> > > 
> > > The idea that the BIOS ASPM config is some kind of handoff protocol is
> > > really unsupportable.
> > 
> > To be clear, you are not talking about a situation where
> > ACPI_FADT_NO_ASPM or _OSC PCIe disallow OS ASPM control, right?
> > Everyone agrees that this should be honored? The question is what to
> > do by default when the OS is not restricted by these mechanisms?
> 
> Exactly.  The OS should respect ACPI_FADT_NO_ASPM and _OSC.
> 
> I think there are a couple exceptions where we want to disable ASPM
> even if the platform said the OS shouldn't touch ASPM at all, but
> that's a special case.
> 
> > Reading the mentioned thread, I too think that using the BIOS config
> > as the default would be the safest option, but only to avoid
> > breaking systems, not because of an implied contract between the
> > BIOS and OS. However, enabling all capable ASPM features is the
> > ideal option. If the OS isn't limited by ACPI_FADT_NO_ASPM or _OSC
> > PCIe, then ASPM enabling is fully under its control.  If this
> > doesn't work for some devices then they are broken and need a quirk.
> 
> Agreed.  It may not be practical to identify such devices, so we may
> need a broader arch-based and/or date-based quirk.
> 
> I'd be shocked if Windows treated the BIOS config as a "do not exceed
> this" situation, so my secret hope is that some of these "broken"
> devices are really caused by defects in the Linux ASPM support or the
> driver, and that we can fix them if we find out about them.
> 
> But I have no details about any of these alleged broken devices, so
> it's hard to make progress on them.  

I don't have a sense of the scope either. But I could see BIOS not enabling
features that would provide no added power savings benefit. We use ASPM to
manage package power. There are Intel devices that certainly don't require L1SS
for the SoC to achieve the deepest power savings. L1 alone is fine for them. I
don't know what the test coveragae is for unenabled features. I've sent these
questions to our BIOS folks.

> Maybe we should log a debug note
> if the device advertises ASPM support that BIOS didn't enable.

Good idea.

David

> 
> Bjorn

Re: [PATCH v2 1/6] PCI/ASPM: Add locked helper for enabling link state

[Bjorn Helgaas]

On Wed, Dec 13, 2023 at 03:39:24PM -0800, David E. Box wrote:
> On Wed, 2023-12-13 at 14:45 -0600, Bjorn Helgaas wrote:
> ...

> > I'd be shocked if Windows treated the BIOS config as a "do not exceed
> > this" situation, so my secret hope is that some of these "broken"
> > devices are really caused by defects in the Linux ASPM support or the
> > driver, and that we can fix them if we find out about them.
> > 
> > But I have no details about any of these alleged broken devices, so
> > it's hard to make progress on them.  
> 
> I don't have a sense of the scope either. But I could see BIOS not
> enabling features that would provide no added power savings benefit.
> We use ASPM to manage package power. There are Intel devices that
> certainly don't require L1SS for the SoC to achieve the deepest
> power savings. L1 alone is fine for them. I don't know what the test
> coverage is for unenabled features. I've sent these questions to
> our BIOS folks.

Once upon a time there was a push to make it so firmware only had to
enumerate boot and console devices and it could skip enumeration and
configuration of other devices.  But I don't think we've made much
progress on that, at least for x86, possibly because Linux depends so
much on BIOS resource assignment.  IMO that's a Linux deficiency.

Bjorn

[PATCH v2 2/6] PCI: vmd: Fix deadlock when enabling ASPM

[Johan Hovold]

The vmd_pm_enable_quirk() helper is called from pci_walk_bus() during
probe to enable ASPM for controllers with VMD_FEAT_BIOS_PM_QUIRK set.

Since pci_walk_bus() already holds a pci_bus_sem read lock, use the new
locked helper to enable link states in order to avoid a potential
deadlock (e.g. in case someone takes a write lock before reacquiring
the read lock).

Fixes: f492edb40b54 ("PCI: vmd: Add quirk to configure PCIe ASPM and LTR")
Cc: stable@vger.kernel.org      # 6.3
Cc: Michael Bottini <michael.a.bottini@linux.intel.com>
Cc: David E. Box <david.e.box@linux.intel.com>
Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
---
 drivers/pci/controller/vmd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 94ba61fe1c44..0452cbc362ee 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -751,7 +751,7 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata)
 	if (!(features & VMD_FEAT_BIOS_PM_QUIRK))
 		return 0;
 
-	pci_enable_link_state(pdev, PCIE_LINK_STATE_ALL);
+	pci_enable_link_state_locked(pdev, PCIE_LINK_STATE_ALL);
 
 	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_LTR);
 	if (!pos)
-- 
2.41.0