[Bug 219112] New: Machine will not wake from suspend if KVM VM is running

[Bug 219112] New: Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

            Bug ID: 219112
           Summary: Machine will not wake from suspend if KVM VM is
                    running
           Product: Virtualization
           Version: unspecified
          Hardware: AMD
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: alex.delorenzo@gmail.com
        Regression: No

Steps to reproduce:

1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a
VM. 

2) Suspend to RAM

3) Wake from suspend

Expected outcome: Machine should resume from suspend when a KVM VM is running

Observed outcome: Machine will not resume from suspend

I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into
the same problem. Unfortunately, I can't find a kernel version where this
problem isn't present. Nothing gets printed to the system or kernel logs.

This is on an AMD Ryzen 7 5850U laptop.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Re: [Bug 219112] New: Machine will not wake from suspend if KVM VM is running

[Sean Christopherson]

On Tue, Jul 30, 2024, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=219112
> 
>             Bug ID: 219112
>            Summary: Machine will not wake from suspend if KVM VM is
>                     running
>            Product: Virtualization
>            Version: unspecified
>           Hardware: AMD
>                 OS: Linux
>             Status: NEW
>           Severity: normal
>           Priority: P3
>          Component: kvm
>           Assignee: virtualization_kvm@kernel-bugs.osdl.org
>           Reporter: alex.delorenzo@gmail.com
>         Regression: No
> 
> Steps to reproduce:
> 
> 1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a
> VM. 
> 
> 2) Suspend to RAM
> 
> 3) Wake from suspend
> 
> Expected outcome: Machine should resume from suspend when a KVM VM is running
> 
> Observed outcome: Machine will not resume from suspend

As in, suspend/resume the host, and the host never resumes?

> I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into

Have you tried older kernels?  If so, is bisecting possible?  E.g. there was a
somewhat related rework in 6.1 (IIRC).

> the same problem. Unfortunately, I can't find a kernel version where this
> problem isn't present. Nothing gets printed to the system or kernel logs.
> 
> This is on an AMD Ryzen 7 5850U laptop.
> 
> -- 
> You may reply to this email to add a comment.
> 
> You are receiving this mail because:
> You are watching the assignee of the bug.

[Bug 219112] Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #1 from Sean Christopherson (seanjc@google.com) ---
On Tue, Jul 30, 2024, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=219112
> 
>             Bug ID: 219112
>            Summary: Machine will not wake from suspend if KVM VM is
>                     running
>            Product: Virtualization
>            Version: unspecified
>           Hardware: AMD
>                 OS: Linux
>             Status: NEW
>           Severity: normal
>           Priority: P3
>          Component: kvm
>           Assignee: virtualization_kvm@kernel-bugs.osdl.org
>           Reporter: alex.delorenzo@gmail.com
>         Regression: No
> 
> Steps to reproduce:
> 
> 1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start
> a
> VM. 
> 
> 2) Suspend to RAM
> 
> 3) Wake from suspend
> 
> Expected outcome: Machine should resume from suspend when a KVM VM is running
> 
> Observed outcome: Machine will not resume from suspend

As in, suspend/resume the host, and the host never resumes?

> I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into

Have you tried older kernels?  If so, is bisecting possible?  E.g. there was a
somewhat related rework in 6.1 (IIRC).

> the same problem. Unfortunately, I can't find a kernel version where this
> problem isn't present. Nothing gets printed to the system or kernel logs.
> 
> This is on an AMD Ryzen 7 5850U laptop.
> 
> -- 
> You may reply to this email to add a comment.
> 
> You are receiving this mail because:
> You are watching the assignee of the bug.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219112] Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #2 from AlexDeLorenzo.dev (alex.delorenzo@gmail.com) ---
(In reply to Sean Christopherson from comment #1)
> > Expected outcome: Machine should resume from suspend when a KVM VM is
> running
> > 
> > Observed outcome: Machine will not resume from suspend
> 
> As in, suspend/resume the host, and the host never resumes?

Yup, the host never resumes. It's also might be possible that the host never
suspends correctly in the first place. I've noticed that sometimes the final
lock screen image freezes on external monitors, suggesting that a full suspend
wasn't successful. 

> > I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran
> into
> 
> Have you tried older kernels?  If so, is bisecting possible?  E.g. there was
> a
> somewhat related rework in 6.1 (IIRC).


Several months ago, this wasn't a problem when I was using older kernels.
Unfortunately, I don't remember what versions I was using at that point, just
that they were 6.x.

I'll try with some 6.1 era kernels to see if the problem is still there.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219112] Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

Brandon Nielsen (nielsenb@jetfuse.net) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |nielsenb@jetfuse.net

--- Comment #3 from Brandon Nielsen (nielsenb@jetfuse.net) ---
I have verified this on my systems as well. One with an AMD 5800x3D, another
with an Intel i7-8550U. Both have virtualization support enabled in their
respective BIOSes. Usermode VMs used on both machines. Kernel 6.11.5 currently,
but I haven't been using VMs frequently lately so I don't know when this
behavior started.

I think suspend is being entered properly "kernel: PM: suspend entry (deep)" is
seen in the journal, and I think exiting suspend at least starts "kernel: PM:
suspend exit" is seen in the journal along with a bunch of other resuming stuff
(network interfaces being brought up, clocks being synchronized, etc.).

Unfortunately the displays never wake up and the machine never becomes usable.

I even see my power button press being logged when I eventually need to hard
kill the machine so I can reboot.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219112] Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #4 from Brandon Nielsen (nielsenb@jetfuse.net) ---
I am unable to recreate with non-user mode VMs.

So, at least in my case, the real issue is probably in systemd:
https://bugzilla.redhat.com/show_bug.cgi?id=2321268

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219112] Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

Sean Christopherson (seanjc@google.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |seanjc@google.com

--- Comment #5 from Sean Christopherson (seanjc@google.com) ---
Heh, that was quite the rabbit hole.  Poking around reminded me of a bug report
from Tejun about KVM worker threads not freezing in certain scenarios[*].

Long explanation very short, can you try setting KVM's "nx_huge_pages" module
param to "never"?  E.g. via

  echo "never" | sudo tee /sys/module/kvm/parameters/nx_huge_pages

Note, the module param is writable even while KVM is loaded, but it needs to be
set to "never' _before_ launching any VMs, in which case KVM won't spawn the
associated worker thread.  I.e. the "never" setting only affects worker threads
for new VMs.

AFAIK, that's the only problematic flavor of worker thread in KVM.  If that
makes the suspend/resume problem go away, then odds are very good the
underlying issue is the same one Tejun reported.

[*] https://lore.kernel.org/all/ZyAnSAw34jwWicJl@slm.duckdns.org

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219112] Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #6 from Brandon Nielsen (nielsenb@jetfuse.net) ---
(In reply to Sean Christopherson from comment #5)
> Heh, that was quite the rabbit hole.  Poking around reminded me of a bug
> report from Tejun about KVM worker threads not freezing in certain
> scenarios[*].
> 
> Long explanation very short, can you try setting KVM's "nx_huge_pages"
> module param to "never"?  E.g. via
> 
>   echo "never" | sudo tee /sys/module/kvm/parameters/nx_huge_pages
> 
> Note, the module param is writable even while KVM is loaded, but it needs to
> be set to "never' _before_ launching any VMs, in which case KVM won't spawn
> the associated worker thread.  I.e. the "never" setting only affects worker
> threads for new VMs.
> 
> AFAIK, that's the only problematic flavor of worker thread in KVM.  If that
> makes the suspend/resume problem go away, then odds are very good the
> underlying issue is the same one Tejun reported.
> 
> [*] https://lore.kernel.org/all/ZyAnSAw34jwWicJl@slm.duckdns.org

Interesting, so on my Intel laptop that is set to "Y", on my AMD desktop that
is set to "N", in both cases, setting it to "never" seems to fix suspend. I
can't find the documentation for that particular tunable, but that seems odd.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219112] Machine will not wake from suspend if KVM VM is running

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #7 from Sean Christopherson (seanjc@google.com) ---
(In reply to Brandon Nielsen from comment #6)
> (In reply to Sean Christopherson from comment #5)
> > Heh, that was quite the rabbit hole.  Poking around reminded me of a bug
> > report from Tejun about KVM worker threads not freezing in certain
> > scenarios[*].
> > 
> > Long explanation very short, can you try setting KVM's "nx_huge_pages"
> > module param to "never"?  E.g. via
> > 
> >   echo "never" | sudo tee /sys/module/kvm/parameters/nx_huge_pages
> > 
> > Note, the module param is writable even while KVM is loaded, but it needs
> to
> > be set to "never' _before_ launching any VMs, in which case KVM won't spawn
> > the associated worker thread.  I.e. the "never" setting only affects worker
> > threads for new VMs.
> > 
> > AFAIK, that's the only problematic flavor of worker thread in KVM.  If that
> > makes the suspend/resume problem go away, then odds are very good the
> > underlying issue is the same one Tejun reported.
> > 
> > [*] https://lore.kernel.org/all/ZyAnSAw34jwWicJl@slm.duckdns.org
> 
> Interesting, so on my Intel laptop that is set to "Y", on my AMD desktop
> that is set to "N", in both cases, setting it to "never" seems to fix
> suspend. I can't find the documentation for that particular tunable, but
> that seems odd.

The behavior is expected and working as intended.  The parameter controls a
mitigation that is enabled by default if and only if the CPU is affected by
L1TF.  But, KVM creates a kthread related to the mitigation even if the
mitigation is disabled, so that KVM doesn't need to try and spawn/destroy the
kthread on-demand if the user/admin toggles the mitigation while the VM is
running.

The "never" setting is a relatively recent addition that allows the admin to
commit to never enabling the mitigation (without reloading KVM), specifically
so that KVM can skip creating the kthread, e.g. to reduce VM creation latency.

As for documentation, I'll add that to the todo list, I simply forgot to add it
when introducing the "never" option.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.