Re: [PATCH 1/6] PCI/ASPM: Add locked helper for enabling link state

["David E. Box" <david.e.box@linux.intel.com>]

On Tue, 2023-11-14 at 14:55 +0100, Johan Hovold wrote:
> Add a helper for enabling link states that can be used in contexts where
> a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).
> 
> This helper will be used to fix a couple of potential deadlocks where
> the current helper is called with the lock already held, hence the CC
> stable tag.

Thanks for sending a patch to address this. It had fallen off my radar.

> 
> Fixes: f492edb40b54 ("PCI: vmd: Add quirk to configure PCIe ASPM and LTR")
> Cc: stable@vger.kernel.org   # 6.3
> Cc: Michael Bottini <michael.a.bottini@linux.intel.com>
> Cc: David E. Box <david.e.box@linux.intel.com>
> Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
> ---
>  drivers/pci/pcie/aspm.c | 53 +++++++++++++++++++++++++++++++----------
>  include/linux/pci.h     |  3 +++
>  2 files changed, 43 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index 50b04ae5c394..8cf8cc2d6bba 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -1109,17 +1109,7 @@ int pci_disable_link_state(struct pci_dev *pdev, int
> state)
>  }
>  EXPORT_SYMBOL(pci_disable_link_state);
>  
> -/**
> - * pci_enable_link_state - Clear and set the default device link state so
> that
> - * the link may be allowed to enter the specified states. Note that if the
> - * BIOS didn't grant ASPM control to the OS, this does nothing because we
> can't
> - * touch the LNKCTL register. Also note that this does not enable states
> - * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> - *
> - * @pdev: PCI device
> - * @state: Mask of ASPM link states to enable
> - */
> -int pci_enable_link_state(struct pci_dev *pdev, int state)
> +static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool
> locked)
>  {
>         struct pcie_link_state *link = pcie_aspm_get_link(pdev);
>  
> @@ -1136,7 +1126,8 @@ int pci_enable_link_state(struct pci_dev *pdev, int
> state)
>                 return -EPERM;
>         }
>  
> -       down_read(&pci_bus_sem);
> +       if (!locked)
> +               down_read(&pci_bus_sem);

This solution is similar to the original one I proposed [1]. It just creates a
separate locked function rather than using a flag. While the API is consistent
with pci_disable_link_state_locked(), its usage is not. The vmd driver calls
pci_enable_link_state() from pci_walk_bus() which was problematic for Bjorn [2].
I owed an attempt to implement this as a fixup. However, now
qcom_pcie_enabled_aspm() is also using it from pci_walk_bus(). The fixup could
make sense for VMD since there it's fixing missing BIOS settings, but I'm not
sure about qcom. Bjorn, do you still see issues with the use in pci_bus_walk()?

[1]
https://lore.kernel.org/lkml/20230321233849.3408339-1-david.e.box@linux.intel.com/

[2] https://lore.kernel.org/lkml/20230322205702.GA2493123@bhelgaas/

David

>         mutex_lock(&aspm_lock);
>         link->aspm_default = 0;
>         if (state & PCIE_LINK_STATE_L0S)
> @@ -1157,12 +1148,48 @@ int pci_enable_link_state(struct pci_dev *pdev, int
> state)
>         link->clkpm_default = (state & PCIE_LINK_STATE_CLKPM) ? 1 : 0;
>         pcie_set_clkpm(link, policy_to_clkpm_state(link));
>         mutex_unlock(&aspm_lock);
> -       up_read(&pci_bus_sem);
> +       if (!locked)
> +               up_read(&pci_bus_sem);
>  
>         return 0;
>  }
> +
> +/**
> + * pci_enable_link_state - Clear and set the default device link state so
> that
> + * the link may be allowed to enter the specified states. Note that if the
> + * BIOS didn't grant ASPM control to the OS, this does nothing because we
> can't
> + * touch the LNKCTL register. Also note that this does not enable states
> + * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> + *
> + * @pdev: PCI device
> + * @state: Mask of ASPM link states to enable
> + */
> +int pci_enable_link_state(struct pci_dev *pdev, int state)
> +{
> +       return __pci_enable_link_state(pdev, state, false);
> +}
>  EXPORT_SYMBOL(pci_enable_link_state);
>  
> +/**
> + * pci_enable_link_state - Clear and set the default device link state so
> that
> + * the link may be allowed to enter the specified states. Note that if the
> + * BIOS didn't grant ASPM control to the OS, this does nothing because we
> can't
> + * touch the LNKCTL register. Also note that this does not enable states
> + * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> + *
> + * @pdev: PCI device
> + * @state: Mask of ASPM link states to enable
> + *
> + * Context: Caller holds pci_bus_sem read lock.
> + */
> +int pci_enable_link_state_locked(struct pci_dev *pdev, int state)
> +{
> +       lockdep_assert_held_read(&pci_bus_sem);
> +
> +       return __pci_enable_link_state(pdev, state, true);
> +}
> +EXPORT_SYMBOL(pci_enable_link_state_locked);
> +
>  static int pcie_aspm_set_policy(const char *val,
>                                 const struct kernel_param *kp)
>  {
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 60ca768bc867..dea043bc1e38 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1829,6 +1829,7 @@ extern bool pcie_ports_native;
>  int pci_disable_link_state(struct pci_dev *pdev, int state);
>  int pci_disable_link_state_locked(struct pci_dev *pdev, int state);
>  int pci_enable_link_state(struct pci_dev *pdev, int state);
> +int pci_enable_link_state_locked(struct pci_dev *pdev, int state);
>  void pcie_no_aspm(void);
>  bool pcie_aspm_support_enabled(void);
>  bool pcie_aspm_enabled(struct pci_dev *pdev);
> @@ -1839,6 +1840,8 @@ static inline int pci_disable_link_state_locked(struct
> pci_dev *pdev, int state)
>  { return 0; }
>  static inline int pci_enable_link_state(struct pci_dev *pdev, int state)
>  { return 0; }
> +static inline int pci_enable_link_state_locked(struct pci_dev *pdev, int
> state)
> +{ return 0; }
>  static inline void pcie_no_aspm(void) { }
>  static inline bool pcie_aspm_support_enabled(void) { return false; }
>  static inline bool pcie_aspm_enabled(struct pci_dev *pdev) { return false; }