Re: [PATCH v5] PCI/ASPM: Add back L1 PM Substate save and restore

[Bjorn Helgaas <helgaas@kernel.org>]

[+cc Michael]

On Thu, Dec 28, 2023 at 04:31:12PM -0600, Bjorn Helgaas wrote:
> On Wed, Dec 20, 2023 at 05:12:50PM -0800, David E. Box wrote:
> ...

> > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > index 55bc3576a985..3c4b2647b4ca 100644
> > --- a/drivers/pci/pci.c
> > +++ b/drivers/pci/pci.c

> > @@ -1579,7 +1579,7 @@ static void pci_restore_pcie_state(struct pci_dev *dev)
> >  {
> > ...

> > +        So we restore here only the
> > +	 * LNKCTL register with the ASPM control field clear. ASPM will
> > +	 * be restored in pci_restore_aspm_state().
> > +	 */
> > +	val = cap[i++] & ~PCI_EXP_LNKCTL_ASPMC;
> > +	pcie_capability_write_word(dev, PCI_EXP_LNKCTL, val);
> 
> When CONFIG_PCIEASPM is not set, we will clear ASPMC here and never
> restore it.  I don't know if this ever happens.  Do we need to worry
> about this?  Might firmware restore ASPMC itself before we get here?
> What do we want to happen in this case?
> 
> Since ASPM is intertwined with the PCIe Capability, can we call
> pci_restore_aspm_state() from here instead of from
> pci_restore_state()?
> 
> Calling it here would make it easier to see the required ordering
> (LNKCTL with ASPMC cleared, restore ASPM L1SS, restore ASPMC) and
> it would be obvious that none of the other stuff in
> pci_restore_state() is relevant (PASID, PRI, ATS, VC, etc).
> 
> If that could be done, I think it would make sense to do the same with
> pci_save_aspm_state() even though it's a little more independent.

The lspci output in Michael's report at
https://lore.kernel.org/r/76c61361-b8b4-435f-a9f1-32b716763d62@5challer.de
reminded me that LTR is important for L1.2, and we currently have
this:

  pci_restore_state
    pci_restore_ltr_state
    pci_restore_pcie_state

I wonder if pci_restore_ltr_state() should be called from
pci_restore_pcie_state() as well?  It's intimately connected to ASPM,
and that connection isn't very clear in the current code.

Bjorn