Re: [tip:x86/alternatives] [x86/alternatives] ee8962082a: WARNING:at_arch/x86/kernel/cpu/cpuid-deps.c:#do_clear_cpu_cap

[Sean Christopherson <seanjc@google.com>]

On Tue, May 07, 2024, Borislav Petkov wrote:
> From: Borislav Petkov <bp@alien8.de>
> To: Oliver Sang <oliver.sang@intel.com>
> Cc: Sean Christopherson <seanjc@google.com>, oe-lkp@lists.linux.dev,
> 	lkp@intel.com, linux-kernel@vger.kernel.org, x86@kernel.org,
> 	Ingo Molnar <mingo@kernel.org>, Srikanth Aithal <sraithal@amd.com>
> Bcc: bp@alien8.de
> Subject: Re: [tip:x86/alternatives] [x86/alternatives] ee8962082a:
>  WARNING:at_arch/x86/kernel/cpu/cpuid-deps.c:#do_clear_cpu_cap
> Reply-To: 
> In-Reply-To: <ZjnTW4XQwVHEiSaW@xsang-OptiPlex-9020>
> 
> On Tue, May 07, 2024 at 03:08:11PM +0800, Oliver Sang wrote:
> > I applied the debug pach ontop of lastest Linus master:
> > 
> > 1621a826233a7 debug patch from Boris for ee8962082a
> > dccb07f2914cd (HEAD, linus/master) Merge tag 'for-6.9-rc7-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
> > 
> > attached dmesg and cpuinfo (a little diff, so I attached it again)
> 
> Thanks, now what are we seeing here:
> 
> [    0.763720][    T0] x86/cpu: init_ia32_feat_ctl: CPU0: FEAT_CTL: 0x5, tboot: 0

...

> FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX, bit 2 is set. So that conditional is
> not true either. And the pr_err_once() doesn't appear in dmesg.
> 
> BUT(!), look what the original dmesg said:
> 
> [    0.055225][    T0] x86/cpu: VMX (outside TXT) disabled by BIOS
> 
> So that FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX bit was not set back then. Why?
> 
> Oliver, have you done any BIOS config changes in the meantime?
> 
> This all looks really weird.
> 
> The other possibility would be if something changed between -rc3
> (the branch x86/alternatives is based on) and -rc7. Unlikely but by now
> everything's possible.
>
> What could also be the case is, the BSP's FEAT_CTL is 0x0 (unconfigured,
> whatever), we'd go in, set FEAT_CTL_LOCKED and that'll lock the bit in
> all FEAT_CTLs on all cores, then it'll set
> FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX.o

I would say it's beyond unlikely that a kernel change is responsible.  In both
traces, FEAT_CTL.LOCKED is '1' before init_ia32_feat_ctl() runs, i.e. the MSR was
already locked by BIOS.  And that is by far the most common scenario, it's all
but unheard of for BIOS to leave FEAT_CTL unlocked.

For giggles, I hacked QEMU to simulate FEAT_CTL being (a) unlocked by BIOS and
(b) locked with VMX disabled.  For both (a) and (b), an -rc3 based kernel behaves
as expected, i.e. configures the MSR correctly for (a), and complains once on the
BSP about VMX being disabled for (b).  Neither case triggers the WARN_ON()
alternatives being applied.

Oliver, are you able to reproduce the WARN using the "original" kernel?  If not,
then I don't think it's more time looking at this from a kernel perspective, as
it's more or less guaranteed to be some sort of environmental issue.