A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Brian Cowan]

Hi all, looking for hints on a wild crash.

The company I work for has a kernel driver used to literally make a db
query result look like a filesystem… The “database” in question being
a proprietary SCM repository… (ClearCase, for those who have been
around forever… Like me…)

We have a crash on mounting the remote repository ONE way (ClearCase
“Automatic views”) but not another (ClearCase “Dynamic views”) where
both use the same kernel driver… The guest OS is RHEL 7.8, not
registered with RH (since the VM is only supposed to last a couple of
days.) The host OS is Ubuntu 20.04.2 LTS, though that does not seem to
matter.

The wild part is that this only happens when the ClearCase host is a
KVM guest, and only on 6th-generation or newer . It does NOT happen
on:
* VMWare Virtual machines configured identically
* VirtualBox Virtual machines Configured identically
* 2nd generation intel core hosts running the same KVM release.
(because OF COURSE my office "secondary desktop" host is ancient...
* A 4th generation I7 host running Ubuntu 22.04 and that version’s
default KVM. (Because I am a laptop packrat. That laptop had been
sitting on a bookshelf for 3+ years and I went "what if...")

If I edit the KVM configuration and change the “mirror host CPU”
option to use the 2nd or 4th generation CPU options, the crash stops
happening… If this was happening on physical machines, the VM crash
would make sense, but it's literally a hypervisor-specific crash.

Any hints, tips, or comments would be most appreciated... Never
thought I'd be trying to debug kernel/hypervisor interactions, but
here I am...

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Jim Mattson]

On Wed, May 18, 2022 at 10:14 AM Brian Cowan <brcowan@gmail.com> wrote:
>
> Hi all, looking for hints on a wild crash.
>
> The company I work for has a kernel driver used to literally make a db
> query result look like a filesystem… The “database” in question being
> a proprietary SCM repository… (ClearCase, for those who have been
> around forever… Like me…)
>
> We have a crash on mounting the remote repository ONE way (ClearCase
> “Automatic views”) but not another (ClearCase “Dynamic views”) where
> both use the same kernel driver… The guest OS is RHEL 7.8, not
> registered with RH (since the VM is only supposed to last a couple of
> days.) The host OS is Ubuntu 20.04.2 LTS, though that does not seem to
> matter.
>
> The wild part is that this only happens when the ClearCase host is a
> KVM guest, and only on 6th-generation or newer . It does NOT happen
> on:
> * VMWare Virtual machines configured identically
> * VirtualBox Virtual machines Configured identically
> * 2nd generation intel core hosts running the same KVM release.
> (because OF COURSE my office "secondary desktop" host is ancient...
> * A 4th generation I7 host running Ubuntu 22.04 and that version’s
> default KVM. (Because I am a laptop packrat. That laptop had been
> sitting on a bookshelf for 3+ years and I went "what if...")
>
> If I edit the KVM configuration and change the “mirror host CPU”
> option to use the 2nd or 4th generation CPU options, the crash stops
> happening… If this was happening on physical machines, the VM crash
> would make sense, but it's literally a hypervisor-specific crash.
>
> Any hints, tips, or comments would be most appreciated... Never
> thought I'd be trying to debug kernel/hypervisor interactions, but
> here I am...

Guest crash or hypervisor crash? If the former, can you provide the
guest's console output?

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Brian Cowan]

Guest crash.. Which console? Would the vmcore-dmesg be of use? I used
"virsh --debug=0 rhel7.6 --console --force-boot" and got nothing on
the console.

This might be because this is running a GUI login. I found
http://libvirt.org/kbase/debuglogs.html

I set the debug to go to a file, then set the filters to use the "Less
verbose logging for QEMU VMs" on that page... I also tried a log
filter of "1:*".

I saw RESET events that then specify "guest-reset" but I don't see
anything other than startup messages in the domain logs...  Problem
is, of course, I have little or no idea what to look for, or whether I
should modify the domain log settings somehow... At least I admit I'm
clueless...

On Wed, May 18, 2022 at 1:30 PM Jim Mattson <jmattson@google.com> wrote:
> Guest crash or hypervisor crash? If the former, can you provide the
> guest's console output?

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Jim Mattson]

On Wed, May 18, 2022 at 11:48 AM Brian Cowan <brcowan@gmail.com> wrote:
>
> Guest crash.. Which console? Would the vmcore-dmesg be of use? I used
> "virsh --debug=0 rhel7.6 --console --force-boot" and got nothing on
> the console.

I know next to nothing about virsh, but I think you want:
% virsh console rhel7.6

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Sean Christopherson]

On Wed, May 18, 2022, Brian Cowan wrote:
> Hi all, looking for hints on a wild crash.
> 
> The company I work for has a kernel driver used to literally make a db
> query result look like a filesystem… The “database” in question being
> a proprietary SCM repository… (ClearCase, for those who have been
> around forever… Like me…)
> 
> We have a crash on mounting the remote repository ONE way (ClearCase
> “Automatic views”) but not another (ClearCase “Dynamic views”) where
> both use the same kernel driver… The guest OS is RHEL 7.8, not
> registered with RH (since the VM is only supposed to last a couple of
> days.) The host OS is Ubuntu 20.04.2 LTS, though that does not seem to
> matter.
> 
> The wild part is that this only happens when the ClearCase host is a
> KVM guest, and only on 6th-generation or newer . It does NOT happen
> on:
> * VMWare Virtual machines configured identically
> * VirtualBox Virtual machines Configured identically
> * 2nd generation intel core hosts running the same KVM release.
> (because OF COURSE my office "secondary desktop" host is ancient...

Heh, Sandy Bridge isn't ancient, we still get bug reports for Core2 :-)

> * A 4th generation I7 host running Ubuntu 22.04 and that version’s
> default KVM. (Because I am a laptop packrat. That laptop had been
> sitting on a bookshelf for 3+ years and I went "what if...")

What kernel version is the 6th gen (Skylake) 20.04.2 running?  Same question for
the 4th gen (Haswell) 22.04.  And if it's not too much trouble, can you try running
the Skylake with 22.04 kernel, or vice versa?  Not super high priority if it's a
pain, the fact that the bug goes away based on what's advertised to the guest
suggests this might be a guest bug.  But, it could also be a KVM bug that's
specific to a feature that's only supported in Skylake+.

> If I edit the KVM configuration and change the “mirror host CPU”
> option to use the 2nd or 4th generation CPU options, the crash stops
> happening… If this was happening on physical machines, the VM crash
> would make sense, but it's literally a hypervisor-specific crash.
> 
> Any hints, tips, or comments would be most appreciated... Never
> thought I'd be trying to debug kernel/hypervisor interactions, but
> here I am...

It might be that there's a guest bug.  And even if it's not a guest bug, you can
likely identify exactly what feature is problematic, though it might require
invoking QEMU directly (I don't know exactly what level of vCPU customization
libvirt allows).

First thing to try: does it repro by explicitly specifying "Skylake-Client" as the
vCPU model?  No idea what libvirt calls that.  If that works, then I think XSAVES
would be to blame; AFAICT that's the only thing that might be exposed by "mirror
host CPU" and not the explicit "Skylake-Client".  XSAVE being to blame seems unlikely
though.

Assuming "Skylake-Client" fails, the next step would be to disable features that
are in "Skylake-Client" but not "Haswell", one by one, to figure out what's to
blame.

In QEMU, the featuers I see being in Skylake but not Haswell are:

  3dnowprefetch, rdseed, adx, smap, xsavec, xgetbv1

Again, no idea if/how libvirt exposes that level of granularity.  For running
QEMU directly, removing all those features would be:

  -cpu Skylake-Client,-3dnowprefetch,-rdseed,-adx,-smap,-xsavec,-xgetbv1

My money is on SMAP :-)

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Brian Cowan]

Disabling smap seems to fix the problem... Now for the hard question: WHY?

I went from the "mirror host CPU" to "Skylake client" with "security
mitigations" enabled. I then added disabling SMAP... by editing the
virsh xml configuration. This left me with this in the XML definition:
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Skylake-Client</model>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='spec-ctrl'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='disable' name='smap'/>
  </cpu>

I then started the VM, did the exact same thing that crashed, and the
crash didn't happen.

Resetting it to just "Skylake client" with CPU security mitigations
enabled crashes again

Failing override:
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Skylake-Client</model>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='spec-ctrl'/>
    <feature policy='require' name='ssbd'/>
  </cpu>

If I go back to mirroring the host configuration, it (still) still
crashes, and this is what the CPU section looks like at runtime:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='ibrs'/>
    <feature policy='require' name='amd-stibp'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='disable' name='mpx'/>
  </cpu>

On Wed, May 18, 2022 at 5:26 PM Sean Christopherson <seanjc@google.com> wrote:
>
> On Wed, May 18, 2022, Brian Cowan wrote:
> > Hi all, looking for hints on a wild crash.
> >
> > The company I work for has a kernel driver used to literally make a db
> > query result look like a filesystem… The “database” in question being
> > a proprietary SCM repository… (ClearCase, for those who have been
> > around forever… Like me…)
> >
> > We have a crash on mounting the remote repository ONE way (ClearCase
> > “Automatic views”) but not another (ClearCase “Dynamic views”) where
> > both use the same kernel driver… The guest OS is RHEL 7.8, not
> > registered with RH (since the VM is only supposed to last a couple of
> > days.) The host OS is Ubuntu 20.04.2 LTS, though that does not seem to
> > matter.
> >
> > The wild part is that this only happens when the ClearCase host is a
> > KVM guest, and only on 6th-generation or newer . It does NOT happen
> > on:
> > * VMWare Virtual machines configured identically
> > * VirtualBox Virtual machines Configured identically
> > * 2nd generation intel core hosts running the same KVM release.
> > (because OF COURSE my office "secondary desktop" host is ancient...
>
> Heh, Sandy Bridge isn't ancient, we still get bug reports for Core2 :-)
>
> > * A 4th generation I7 host running Ubuntu 22.04 and that version’s
> > default KVM. (Because I am a laptop packrat. That laptop had been
> > sitting on a bookshelf for 3+ years and I went "what if...")
>
> What kernel version is the 6th gen (Skylake) 20.04.2 running?  Same question for
> the 4th gen (Haswell) 22.04.  And if it's not too much trouble, can you try running
> the Skylake with 22.04 kernel, or vice versa?  Not super high priority if it's a
> pain, the fact that the bug goes away based on what's advertised to the guest
> suggests this might be a guest bug.  But, it could also be a KVM bug that's
> specific to a feature that's only supported in Skylake+.
>
> > If I edit the KVM configuration and change the “mirror host CPU”
> > option to use the 2nd or 4th generation CPU options, the crash stops
> > happening… If this was happening on physical machines, the VM crash
> > would make sense, but it's literally a hypervisor-specific crash.
> >
> > Any hints, tips, or comments would be most appreciated... Never
> > thought I'd be trying to debug kernel/hypervisor interactions, but
> > here I am...
>
> It might be that there's a guest bug.  And even if it's not a guest bug, you can
> likely identify exactly what feature is problematic, though it might require
> invoking QEMU directly (I don't know exactly what level of vCPU customization
> libvirt allows).
>
> First thing to try: does it repro by explicitly specifying "Skylake-Client" as the
> vCPU model?  No idea what libvirt calls that.  If that works, then I think XSAVES
> would be to blame; AFAICT that's the only thing that might be exposed by "mirror
> host CPU" and not the explicit "Skylake-Client".  XSAVE being to blame seems unlikely
> though.
>
> Assuming "Skylake-Client" fails, the next step would be to disable features that
> are in "Skylake-Client" but not "Haswell", one by one, to figure out what's to
> blame.
>
> In QEMU, the featuers I see being in Skylake but not Haswell are:
>
>   3dnowprefetch, rdseed, adx, smap, xsavec, xgetbv1
>
> Again, no idea if/how libvirt exposes that level of granularity.  For running
> QEMU directly, removing all those features would be:
>
>   -cpu Skylake-Client,-3dnowprefetch,-rdseed,-adx,-smap,-xsavec,-xgetbv1
>
> My money is on SMAP :-)

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Sean Christopherson]

On Fri, May 20, 2022, Brian Cowan wrote:
> Disabling smap seems to fix the problem...

Mwhahaha, I should have found someone to bet me real money :-)

> Now for the hard question: WHY?

The most likely scenario it that there's a SMAP violation (#PF due to a kernel
access to user data without an override to tell the CPU that the access is intentional)
somewhere in the guest that crashes/panics the guest kernel.  Assuming that's the
case, there are three-ish possibilities:

  1. There's a bug your company's custom kernel driver.
  2. There's a SMAP violation somewhere else in RHEL 7.8, which is an 8+ year old
     frankenkernel...
  3. There's a bug in your version of KVM related to SMAP virtualization

#3 begs the question, does this fail on bare metal that supports SMAP?  If so,
then that rules out #3.

If the crash occurs only when doing stuff related to your custom driver, #1 is
most likely the culprit.

One way to try and debug further would be to disable EPT in KVM (load kvm_intel with
ept=0) and then use KVM tracepoints to see when the guest dies.  If it's a SMAP
violation, there should be an injected SMAP #PF shortly before the guest dies.

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Brian Cowan]

Well, the weird thing is that this is hypervisor-specific. KVM=kaboom.
VirtualBox is happy, and we can't make this happen on
roughly-analogous ESX hosts. I can't directly test on my (ubuntu)
laptop because the driver won't build on the too-new ubuntu 20.04.2
"Hardware enablement" kernel as it's too new. But either all the other
hypervisors are doing this wrong and allowing this access, or KVM is.

Not being a kernel expert makes this interesting. I'm passing the
possibility list over the wall to the kernel folks, but most of the
evidence we're seeing **seems** to point to KVM...

On Fri, May 20, 2022 at 11:22 AM Sean Christopherson <seanjc@google.com> wrote:
>
> On Fri, May 20, 2022, Brian Cowan wrote:
> > Disabling smap seems to fix the problem...
>
> Mwhahaha, I should have found someone to bet me real money :-)
>
> > Now for the hard question: WHY?
>
> The most likely scenario it that there's a SMAP violation (#PF due to a kernel
> access to user data without an override to tell the CPU that the access is intentional)
> somewhere in the guest that crashes/panics the guest kernel.  Assuming that's the
> case, there are three-ish possibilities:
>
>   1. There's a bug your company's custom kernel driver.
>   2. There's a SMAP violation somewhere else in RHEL 7.8, which is an 8+ year old
>      frankenkernel...
>   3. There's a bug in your version of KVM related to SMAP virtualization
>
> #3 begs the question, does this fail on bare metal that supports SMAP?  If so,
> then that rules out #3.
>
> If the crash occurs only when doing stuff related to your custom driver, #1 is
> most likely the culprit.
>
> One way to try and debug further would be to disable EPT in KVM (load kvm_intel with
> ept=0) and then use KVM tracepoints to see when the guest dies.  If it's a SMAP
> violation, there should be an injected SMAP #PF shortly before the guest dies.

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Jim Mattson]

On Fri, May 20, 2022 at 3:03 PM Brian Cowan <brcowan@gmail.com> wrote:
>
> Well, the weird thing is that this is hypervisor-specific. KVM=kaboom.
> VirtualBox is happy, and we can't make this happen on
> roughly-analogous ESX hosts. I can't directly test on my (ubuntu)
> laptop because the driver won't build on the too-new ubuntu 20.04.2
> "Hardware enablement" kernel as it's too new. But either all the other
> hypervisors are doing this wrong and allowing this access, or KVM is.
>
> Not being a kernel expert makes this interesting. I'm passing the
> possibility list over the wall to the kernel folks, but most of the
> evidence we're seeing **seems** to point to KVM...

Which version of kvm? Any unusual kvm module parameters?

Does the guest under the happy hypervisors report that it has smap (in
/proc/cpuidinfo)?

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Brian Cowan]

Virtualbox 6.1.34: Nope, smap not there for the guest even though it
is there for the host (Ubuntu 20.04, but with a too-new kernel I can't
test the driver against. Now... Where did I put that spare SSD....)
VMWare Player: Have to check to see if SMAP is allowed there shortly...

On Fri, May 20, 2022 at 7:09 PM Jim Mattson <jmattson@google.com> wrote:
>
> On Fri, May 20, 2022 at 3:03 PM Brian Cowan <brcowan@gmail.com> wrote:
> >
> > Well, the weird thing is that this is hypervisor-specific. KVM=kaboom.
> > VirtualBox is happy, and we can't make this happen on
> > roughly-analogous ESX hosts. I can't directly test on my (ubuntu)
> > laptop because the driver won't build on the too-new ubuntu 20.04.2
> > "Hardware enablement" kernel as it's too new. But either all the other
> > hypervisors are doing this wrong and allowing this access, or KVM is.
> >
> > Not being a kernel expert makes this interesting. I'm passing the
> > possibility list over the wall to the kernel folks, but most of the
> > evidence we're seeing **seems** to point to KVM...
>
> Which version of kvm? Any unusual kvm module parameters?
>
> Does the guest under the happy hypervisors report that it has smap (in
> /proc/cpuidinfo)?

Re: A really weird guest crash, that ONLY happens on KVM, and ONLY on 6th gen+ Intel Core CPU's

[Brian Cowan]

Welp, I had to reinstall VMWare Player on this host, and after finding
out that the only way to get past the "join the customer experience
improvement program" prompt was to run the blasted thing as root ONCE,
I was able to build a new VM... SMAP is enabled there... Annnnd.
Kaboom. I looked at the ESX servers that I have my Linux test VM's on,
and no SMAP. So *of course* there's no crash.

So, in the immortal words of Emily Latella: Oh... Never Mind!

At least I now know.

On Tue, May 24, 2022 at 11:30 AM Brian Cowan <brcowan@gmail.com> wrote:
>
> Virtualbox 6.1.34: Nope, smap not there for the guest even though it
> is there for the host (Ubuntu 20.04, but with a too-new kernel I can't
> test the driver against. Now... Where did I put that spare SSD....)
> VMWare Player: Have to check to see if SMAP is allowed there shortly...
>
> On Fri, May 20, 2022 at 7:09 PM Jim Mattson <jmattson@google.com> wrote:
> >
> > On Fri, May 20, 2022 at 3:03 PM Brian Cowan <brcowan@gmail.com> wrote:
> > >
> > > Well, the weird thing is that this is hypervisor-specific. KVM=kaboom.
> > > VirtualBox is happy, and we can't make this happen on
> > > roughly-analogous ESX hosts. I can't directly test on my (ubuntu)
> > > laptop because the driver won't build on the too-new ubuntu 20.04.2
> > > "Hardware enablement" kernel as it's too new. But either all the other
> > > hypervisors are doing this wrong and allowing this access, or KVM is.
> > >
> > > Not being a kernel expert makes this interesting. I'm passing the
> > > possibility list over the wall to the kernel folks, but most of the
> > > evidence we're seeing **seems** to point to KVM...
> >
> > Which version of kvm? Any unusual kvm module parameters?
> >
> > Does the guest under the happy hypervisors report that it has smap (in
> > /proc/cpuidinfo)?