Re: [Bug 217321] New: Intel platforms can't sleep deeper than PC3 during long idle

[Bjorn Helgaas <helgaas@kernel.org>]

[+cc Koba, Ajay, Tasev, Mark, Thomas, regressions list]

On Tue, Apr 11, 2023 at 03:42:29PM -0500, Bjorn Helgaas wrote:
> On Tue, Apr 11, 2023 at 08:32:04AM +0000, bugzilla-daemon@kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=217321
> > ... 
> >         Regression: No
> > 
> > [Symptom]
> > Intel cpu can't sleep deeper than pcˇ during long idle
> > ~~~
> > Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10
> > 15.08   75.02   0.00    0.00    0.00    0.00    0.00
> > 15.09   75.02   0.00    0.00    0.00    0.00    0.00
> > ^CPkg%pc2       Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10
> > 15.38   68.97   0.00    0.00    0.00    0.00    0.00
> > 15.38   68.96   0.00    0.00    0.00    0.00    0.00
> > ~~~
> > [How to Reproduce]
> > 1. run turbostat to monitor
> > 2. leave machine idle
> > 3. turbostat show cpu only go into pc2~pc3.
> > 
> > [Misc]
> > The culprit are this 
> > a7152be79b62) Revert "PCI/ASPM: Save L1 PM Substates Capability for
> > suspend/resume”
> > 
> > if revert a7152be79b62, the issue is gone
> 
> Relevant commits:
> 
>   4ff116d0d5fd ("PCI/ASPM: Save L1 PM Substates Capability for suspend/resume")
>   a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"")
> 
> 4ff116d0d5fd appeared in v6.1-rc1.  Prior to 4ff116d0d5fd, ASPM L1 PM
> Substates configuration was not preserved across suspend/resume, so
> the system *worked* after resume, but used more power than expected.
> 
> But 4ff116d0d5fd caused resume to fail completely on some systems, so
> a7152be79b62 reverted it.  With a7152be79b62 reverted, ASPM L1 PM
> Substates configuration is likely not preserved across suspend/resume.
> a7152be79b62 appeared in v6.2-rc8 and was backported to the v6.1
> stable series starting with v6.1.12.
> 
> KobaKo, you don't mention any suspend/resume in this bug report, but
> neither patch should make any difference unless suspend/resume is
> involved.  Does the platform sleep as expected *before* suspend, but
> fail to sleep after resume?
>
> Or maybe some individual device was suspended via runtime power
> management, and that device lost its L1 PM Substates config?  I don't
> know if there's a way to disable runtime PM easily.

Koba, per your bugzilla update, the issue happens even without
suspend/resume.  And we don't know whether some particular device is
responsible.

But if we save/restore L1SS state, we can sleep deeper than PC3.  If
we don't preserve L1SS state, we can't.

We definitely want to preserve the L1SS state, but we can't simply
apply 4ff116d0d5fd ("PCI/ASPM: Save L1 PM Substates Capability for
suspend/resume") again because it caused its own regressions [1,2,3]

So somebody needs to figure out what was wrong with 4ff116d0d5fd, fix
it, verify that it doesn't cause the issues reported by Tasev, Thomas,
and Mark, and then we can apply it.

Bjorn

[1] https://git.kernel.org/linus/a7152be79b62
[2] https://bugzilla.kernel.org/show_bug.cgi?id=216782
[3] https://bugzilla.kernel.org/show_bug.cgi?id=216877