[Bug 217574] New: kvm_intel loads only after suspend

[Bug 217574] New: kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

            Bug ID: 217574
           Summary: kvm_intel loads only after suspend
           Product: Virtualization
           Version: unspecified
          Hardware: Intel
                OS: Linux
            Status: NEW
          Severity: blocking
          Priority: P3
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: drigoslkx@gmail.com
        Regression: No

Hi, I tested this bug on Slackware 5.15.29, 5.15.97 and 6.1
I tested too on ubuntu 23.04, Fedora 38 and Linux Mint 21.01 and got the same
results.

When I boot Linux, it shows the error
kvm: CPU 0 feature inconsistency!
(Depending on the kernel version it throws this message for each cpu core)

The kernel loads the kvm module but not kvm_intel.
(Without kvm_intel kvm doens't work).
If I try a manual load, doing modprobe kvm_intel it shows
modprobe: ERROR: could not insert 'kvm_intel': Input/output error

So I take some time break, when I back, the system was suspended, I turned it
on again, and tried to load modprobe kvm_intel, it works, and kvm got working
now.

Tested this in all other distros described above and got the same results.
I only get kvm works after a suspend.

My specs
Motherboard Jingsha x99
CPU Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

Chao Gao (chao.gao@intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |chao.gao@intel.com

--- Comment #1 from Chao Gao (chao.gao@intel.com) ---
Could you check if each MSR indexed from 0x480 to 0x492 is consistent across
all CPUs?


To read an MSR (e.g., 0x480) on all CPUs, run
$ sudo rdmsr -a 0x480


Please do the check after bootup and suspension.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

Sean Christopherson (seanjc@google.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |seanjc@google.com

--- Comment #2 from Sean Christopherson (seanjc@google.com) ---
Ya, as Chao alluded to, the problem appears to be that not all CPUs have have
the same VMX configuration, which is reflected in the MSRs Chao mentioned.  My
best guess is that going through a suspend+resume cycle either wipes out a
problematic ucode update, or applies a "good" ucode update to all CPUs, and
thus resolves the inconsistent VMX configuration across CPUs.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

--- Comment #3 from drigohighlander (drigoslkx@gmail.com) ---
Hi, here is the output after bootup:

bash-5.2# rdmsr -a 0x480
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012

after suspension:
bash-5.2# rdmsr -a 0x480
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
da040000000012
bash-5.2# 

lscpu:
bash-5.2# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  36
  On-line CPU(s) list:   0-35
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel
  Model name:            Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz
    BIOS Model name:     Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz  CPU @
2.3GHz
    BIOS CPU family:     179
    CPU family:          6
    Model:               63
    Thread(s) per core:  2
    Core(s) per socket:  18
    Socket(s):           1
    Stepping:            2
    CPU(s) scaling MHz:  64%
    CPU max MHz:         3800.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            4589.05
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mc
                         a cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss 
                         ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc
arc
                         h_perfmon pebs bts rep_good nopl xtopology nonstop_tsc 
                         cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl
vm
                         x smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca
ss
                         e4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c
rdra
                         nd lahf_lm abm cpuid_fault epb invpcid_single pti
intel
                         _ppin tpr_shadow vnmi flexpriority ept vpid ept_ad
fsgs
                         base tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid
rt
                         m cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat
pl
                         n pts
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   576 KiB (18 instances)
  L1i:                   576 KiB (18 instances)
  L2:                    4.5 MiB (18 instances)
  L3:                    45 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-35
Vulnerabilities:         
  Itlb multihit:         KVM: Mitigation: VMX disabled
  L1tf:                  Mitigation; PTE Inversion; VMX conditional cache
flushe
                         s, SMT vulnerable
  Mds:                   Vulnerable: Clear CPU buffers attempted, no microcode; 
                         SMT vulnerable
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Vulnerable: Clear CPU buffers attempted, no microcode; 
                         SMT vulnerable
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user
pointer
                          sanitization
  Spectre v2:            Mitigation; Retpolines, STIBP disabled, RSB filling,
PB
                         RSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected
bash-5.2#

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

--- Comment #4 from Sean Christopherson (seanjc@google.com) ---
Can you check all MSRs in the range 0x480-0x491, i.e. all the known VMX MSRs,
and just report back any divergences between CPUs?  The values for MSRs that
are consistent across all CPUs aren't interesting at this time.  What we
*suspect* is going on is that one or more CPUs has different MSRs in one or
more of the VMX MSRs.  Before we can debug further, we need to first confirm
that that is indeed why KVM is refusing to load.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

--- Comment #5 from drigohighlander (drigoslkx@gmail.com) ---
Hi folks, sorry for my misunderstood.
I ran the command for all MSRs in the range 0x480-0x491 after bootup and after
a suspend and I didn't find any inconsistencies.

How can I share the file of the outputs?

Thank you

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

--- Comment #6 from Chao Gao (chao.gao@intel.com) ---
I found this: https://ubuntuforums.org/archive/index.php/t-2344602.html. it was
reported on the same CPU E5-2696 v3.

Could you run below commands after bootup and suspension, and share the output?

# rdmsr -a 0x48b |uniq
# cat /proc/cpuinfo |grep microcode |uniq

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

--- Comment #7 from drigohighlander (drigoslkx@gmail.com) ---
Hi here is the outputs:

After bootup:
bash-5.2# rdmsr -a 0x48b |uniq
2007fff00000000
7fff00000000
2007fff00000000
7fff00000000

bash-5.2# cat /proc/cpuinfo |grep microcode |uniq
microcode       : 0x39

After suspend:
bash-5.2# rdmsr -a 0x48b |uniq
7fff00000000

bash-5.2# cat /proc/cpuinfo |grep microcode |uniq
microcode       : 0x39

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 217574] kvm_intel loads only after suspend

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=217574

--- Comment #8 from Chao Gao (chao.gao@intel.com) ---
Yes. MSR 0x48b is inconsistent.

Your microcode is too old; according to [1], the latest one for your system
(Family-Model-Stepping: 06-3f-02) is 0x49. Could you update the microcode and
retry?

[1]:
https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/blob/main/releasenote.md

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.