[Bug 219787] New: Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[Bug 219787] New: Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

            Bug ID: 219787
           Summary: Guest's applications crash with EXCEPTION_SINGLE_STEP
                    (0x80000004)
           Product: Virtualization
           Version: unspecified
          Hardware: AMD
                OS: Linux
            Status: NEW
          Severity: high
          Priority: P3
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: rangemachine@gmail.com
        Regression: No

Created attachment 307665
  --> https://bugzilla.kernel.org/attachment.cgi?id=307665&action=edit
Debugger attached to Steam.exe

Overview
========
Linux 6.13 update introduced problem with Windows guest's applications on AMD
processors. Several applications crash with EXCEPTION_SINGLE_STEP (0x80000004).
The list of confirmed software: CrystalDiskMark, Visual Studio Code, Steam,
Looking Glass server, Windows Tweaker. 
It never happened prior 6.13 update. I also checked 6.13.3rc and 6.14.1rc
updates, problem persists there too. I did quick check differences in KVM/SVM
between 6.12 and 6.13 and did not found anything that could set trapflag, so
problem could be somewhere deeper inside kernel.

Steps to reproduce
==================
Run VM with Windows guest, launch any software from the list.

Hardware
========
CPU: AMD Ryzen 7 9800X3D (16) @ 5.27 GHz
MB: TUF GAMING X870-PLUS WIFI

Additional Information
======================
Steam crashes when downloading game, Looking Glass crashes on WinAPI
QueryPerformanceCountrer call. Tested on Window 11 22H2/23H2/24H2.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

rangemachine@gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Kernel Version|                            |6.13

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #1 from rangemachine@gmail.com ---
Created attachment 307666
  --> https://bugzilla.kernel.org/attachment.cgi?id=307666&action=edit
CrystalDiskMark installation

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Sean Christopherson (seanjc@google.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |seanjc@google.com

--- Comment #2 from Sean Christopherson (seanjc@google.com) ---
Are you able to bisect to an exact commit?  There are significant KVM changes
in 6.13, but they're almost all related to memory management.  I can't think of
anything that would manifest as an unexpected single step #DB, especially not
with any consistency.

And just to double check, the only difference in the setup is that the host
kernel was upgraded from v6.12 => v6.13?  E.g. there was no QEMU update or
guest-side changes?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

whanos@sergal.fun changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |whanos@sergal.fun

--- Comment #3 from whanos@sergal.fun ---
I have been able to reproduce this bug too on Linux 6.13.3 - Specifically
whilst attempting to download/install any game via Steam in a GPU passthrough
enabled Windows KVM guest.

Downgrading to Linux 6.12.9 - with no other changes made, immediately resolves
the issue for me.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #4 from rangemachine@gmail.com ---
(In reply to Sean Christopherson from comment #2)
> Are you able to bisect to an exact commit?  There are significant KVM
> changes in 6.13, but they're almost all related to memory management.  I
> can't think of anything that would manifest as an unexpected single step
> #DB, especially not with any consistency.
> 
> And just to double check, the only difference in the setup is that the host
> kernel was upgraded from v6.12 => v6.13?  E.g. there was no QEMU update or
> guest-side changes?

I was not able to bisect yet, sorry. And yes, I double checked, the only change
is kernel upgraded from v6.12.10 to v6.13.2 (did not checked v6.13.3 yet, but
rc version had some behaviour).

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Re: [Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[Sean Christopherson]

On Thu, Feb 20, 2025, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=219787
> 
> whanos@sergal.fun changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |whanos@sergal.fun
> 
> --- Comment #3 from whanos@sergal.fun ---
> I have been able to reproduce this bug too on Linux 6.13.3 - Specifically
> whilst attempting to download/install any game via Steam in a GPU passthrough
> enabled Windows KVM guest.

Are you also running an AMD system?

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #5 from Sean Christopherson (seanjc@google.com) ---
On Thu, Feb 20, 2025, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=219787
> 
> whanos@sergal.fun changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |whanos@sergal.fun
> 
> --- Comment #3 from whanos@sergal.fun ---
> I have been able to reproduce this bug too on Linux 6.13.3 - Specifically
> whilst attempting to download/install any game via Steam in a GPU passthrough
> enabled Windows KVM guest.

Are you also running an AMD system?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Re: [Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[Sean Christopherson]

On Thu, Feb 20, 2025, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=219787
> 
> --- Comment #4 from rangemachine@gmail.com ---
> (In reply to Sean Christopherson from comment #2)
> > Are you able to bisect to an exact commit?  There are significant KVM
> > changes in 6.13, but they're almost all related to memory management.  I
> > can't think of anything that would manifest as an unexpected single step
> > #DB, especially not with any consistency.
> > 
> > And just to double check, the only difference in the setup is that the host
> > kernel was upgraded from v6.12 => v6.13?  E.g. there was no QEMU update or
> > guest-side changes?
> 
> I was not able to bisect yet, sorry.

No need to be sorry, you didn't introduce the bug :-)

> And yes, I double checked, the only change is kernel upgraded from v6.12.10
> to v6.13.2 (did not checked v6.13.3 yet, but rc version had some behaviour).

Please let me know if you'll be able to bisect (or not).  Unless I have a random
epiphany, this will likely require bisection.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #6 from Sean Christopherson (seanjc@google.com) ---
On Thu, Feb 20, 2025, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=219787
> 
> --- Comment #4 from rangemachine@gmail.com ---
> (In reply to Sean Christopherson from comment #2)
> > Are you able to bisect to an exact commit?  There are significant KVM
> > changes in 6.13, but they're almost all related to memory management.  I
> > can't think of anything that would manifest as an unexpected single step
> > #DB, especially not with any consistency.
> > 
> > And just to double check, the only difference in the setup is that the host
> > kernel was upgraded from v6.12 => v6.13?  E.g. there was no QEMU update or
> > guest-side changes?
> 
> I was not able to bisect yet, sorry.

No need to be sorry, you didn't introduce the bug :-)

> And yes, I double checked, the only change is kernel upgraded from v6.12.10
> to v6.13.2 (did not checked v6.13.3 yet, but rc version had some behaviour).

Please let me know if you'll be able to bisect (or not).  Unless I have a
random
epiphany, this will likely require bisection.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #7 from whanos@sergal.fun ---
(In reply to Sean Christopherson from comment #5)
> On Thu, Feb 20, 2025, bugzilla-daemon@kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=219787
> > 
> > whanos@sergal.fun changed:
> > 
> >            What    |Removed                     |Added
> >
> ----------------------------------------------------------------------------
> >                  CC|                            |whanos@sergal.fun
> > 
> > --- Comment #3 from whanos@sergal.fun ---
> > I have been able to reproduce this bug too on Linux 6.13.3 - Specifically
> > whilst attempting to download/install any game via Steam in a GPU
> passthrough
> > enabled Windows KVM guest.
> 
> Are you also running an AMD system?

Yep. I am running a 9800X3D in an X670E chipset motherboard. 
I honestly wonder if this bug only affects people using a 9800X3D.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #8 from rangemachine@gmail.com ---
(In reply to Sean Christopherson from comment #6)
> Please let me know if you'll be able to bisect (or not).  Unless I have a
> random
> epiphany, this will likely require bisection.

Yes, I will try to bisect it tomorrow.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #9 from rangemachine@gmail.com ---
Created attachment 307690
  --> https://bugzilla.kernel.org/attachment.cgi?id=307690&action=edit
bisection-log

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #10 from rangemachine@gmail.com ---
Created attachment 307691
  --> https://bugzilla.kernel.org/attachment.cgi?id=307691&action=edit
bisection-config-culprit

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #11 from rangemachine@gmail.com ---
Here we go:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=408eb7417a92c5354c7be34f7425b305dfe30ad9

Double-checked both reverting commit or unsetting X86_BUS_LOCK_DETECT fixes the
problem.

Added bisection log and config to attachments.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Ravi Bangoria (ravi.bangoria@amd.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ravi.bangoria@amd.com

--- Comment #12 from Ravi Bangoria (ravi.bangoria@amd.com) ---
Thanks for the bug report. This is what is probably happening:

BusLockTrap is controlled through DEBUGCTL MSR and currently DEBUGCTL MSR is
saved/restored on guest entry/exit only if LBRV is enabled. So, if BusLockTrap
is enabled on the host, it will remain enabled even after guest entry and thus,
if some process inside the guest causes a BusLock, KVM will inject #DB from
host to the guest.

I had a KVM patch[1] but couldn't get back to work on it. Let me try to
spend some time and respin it.

[1] https://lore.kernel.org/all/20240808062937.1149-5-ravi.bangoria@amd.com

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #13 from Sean Christopherson (seanjc@google.com) ---
On Fri, Feb 21, 2025, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=219787
> 
> Ravi Bangoria (ravi.bangoria@amd.com) changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |ravi.bangoria@amd.com
> 
> --- Comment #12 from Ravi Bangoria (ravi.bangoria@amd.com) ---
> Thanks for the bug report. This is what is probably happening:
> 
> BusLockTrap is controlled through DEBUGCTL MSR and currently DEBUGCTL MSR is
> saved/restored on guest entry/exit only if LBRV is enabled. So, if
> BusLockTrap
> is enabled on the host, it will remain enabled even after guest entry and
> thus,
> if some process inside the guest causes a BusLock, KVM will inject #DB from
> host to the guest.

*sigh*

Bluntly, that's horrific architecture.  Why on earth isn't debugctl
automatically
context switched when BusLockTrap is supported?

And does AMD do _any_ testing?  This doesn't even require a full reproducer,
e.g. the existing debug KVM-Unit-Test fails on my system (Turin) without ever
generating a split/bus lock.  AFAICT, the CPU is reporting bus locks in DR6 on
#DBs that are most definitely not due to bus locks.

> I had a KVM patch[1] but couldn't get back to work on it. Let me try to
> spend some time and respin it.
> 
> [1] https://lore.kernel.org/all/20240808062937.1149-5-ravi.bangoria@amd.com

Virtualizing BusLockTrap won't do a damn thing.  If the guest isn't using LBRs
or BusLockTrap, then KVM won't enable LBR virtualization and so will run the
guest with the host's DEBUGCTL.

Furthermore, running with the host's DEBUGCTL is a bug irrespective of
BusLockTrap.  It just happens to be fatal with BusLockTrap, but running with
BTF=1 and whatever other bits may be enabled in the host most definitely isn't
correct.

Bug reporters, can you test the attached patches?  I have a reproducer in the
form of a KVM test, but I haven't actually tested a Windows guest.  Assuming
squashing DEBUGCTL remedies the issue, I'll post patches after I've done a bit
more testing.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #14 from Sean Christopherson (seanjc@google.com) ---
Created attachment 307695
  --> https://bugzilla.kernel.org/attachment.cgi?id=307695&action=edit
0002-KVM-SVM-Manually-zero-restore-DEBUGCTL-if-LBR-virtua.patch

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #15 from rangemachine@gmail.com ---
(In reply to Sean Christopherson from comment #13)
> Bug reporters, can you test the attached patches?  I have a reproducer in the
> form of a KVM test, but I haven't actually tested a Windows guest.  Assuming
> squashing DEBUGCTL remedies the issue, I'll post patches after I've done a
> bit
> more testing.

Tested, these 2 patches solves the issue.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #16 from Ravi Bangoria (ravi.bangoria@amd.com) ---
(In reply to Sean Christopherson from comment #13)

> And does AMD do _any_ testing?  This doesn't even require a full reproducer,
> e.g. the existing debug KVM-Unit-Test fails on my system (Turin) without ever
> generating a split/bus lock.  AFAICT, the CPU is reporting bus locks in DR6
> on
> #DBs that are most definitely not due to bus locks.

It seems, the CPU is preserving SW written DR6[BusLockDetected] while
generating the #DB when the CPL is 0 and DEBUGCTL[BusLockTrapEn] is set.

Since most of the x86/debug.c KUT tests clears DR6[BusLockDetected] before
executing the test, the bit remains cleared at the exception time which causes
tests to fail.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Jon Betti (jonbetti@gmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jonbetti@gmail.com

--- Comment #17 from Jon Betti (jonbetti@gmail.com) ---
(In reply to whanos from comment #7)
> I honestly wonder if this bug only affects people using a 9800X3D.

I'm running a 9950X and had repro as another data point. (I had this issue for
a few weeks but thought it was Steam until I started looking at crash dumps in
Windows... and I thankfully stumbled onto this bug which restored my sanity
:).)

(In reply to rangemachine from comment #15)
> (In reply to Sean Christopherson from comment #13)
> > Bug reporters, can you test the attached patches?
> Tested, these 2 patches solves the issue.

+1. Patched my kernel and the issue went away (again: 'twas Steam for me that
threw the exception).

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #18 from Ravi Bangoria (ravi.bangoria@amd.com) ---
(In reply to Ravi Bangoria from comment #16)
> It seems, the CPU is preserving SW written DR6[BusLockDetected] while
> generating the #DB when the CPL is 0 and DEBUGCTL[BusLockTrapEn] is set.

My bad, the behavior is same for CPL 3 as well. Apparently, it's a correct
behavior as documented in the AMD Architecture Programmer's Manual. I've posted
a KUT patch to KVM mailing list. (More details in the patch). Please review.

https://lore.kernel.org/r/20250224112601.6504-1-ravi.bangoria@amd.com

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Brian (maps@knutsson.it) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maps@knutsson.it

--- Comment #19 from Brian (maps@knutsson.it) ---
I can confirm I have the same problem.

CPU is AMD 9950 X3D, and I have the problem both in Virtual Box, and VMware
Workstation.

Can anyone point to a guide to get around this problem now? I have been running
Linux for many years, but it almost just as many years since I compiled my own
kernel.

I am running Ubuntu 25.04

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #20 from Ravi Bangoria (ravi.bangoria@amd.com) ---
The KVM fix went in v6.14-rc6. These are the corresponding patches:

https://git.kernel.org/torvalds/c/ee89e8013383    v6.14-rc6
https://git.kernel.org/torvalds/c/d0eac42f5cec    v6.14-rc6
https://git.kernel.org/torvalds/c/fb71c7959356    v6.14-rc6
https://git.kernel.org/torvalds/c/433265870ab3    v6.14-rc6
https://git.kernel.org/torvalds/c/189ecdb3e112    v6.14-rc6
https://git.kernel.org/torvalds/c/5ecdb48dd918    v6.16-rc1

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #21 from Brian (maps@knutsson.it) ---
How can I still have the problem on Ubuntu 6.14.0-22-generic with VMware
Workstation and VirtualBox?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Artem S. Tashkinov (aros@gmx.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #22 from Artem S. Tashkinov (aros@gmx.com) ---
(In reply to Brian from comment #21)
> How can I still have the problem on Ubuntu 6.14.0-22-generic with VMware
> Workstation and VirtualBox?

VMWare Workstation and VirtualBox use their own supervisors (kernel modules).

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.