Re: Xen HVM regression on certain Intel CPUs

["H. Peter Anvin" <hpa@zytor.com>]

On 03/27/2013 09:04 AM, Konrad Rzeszutek Wilk wrote:
>>>
>>> From the Xen debugging console it was possible to gather a bit more data which
>>> pointed to a failure very close to setting CR4 in startup_32. On this particular
>>> hardware the saved CR4 (about to be set) was 0x1407f0.
>>>
>>> This would set two flags that somehow feel dangerous: PGE (page global enable)
>>> and SMEP (supervisor mode execution protection). SMEP turns out to be the main
>>> offender and the following change allows the APs to start:
>>>
>>> --- a/arch/x86/realmode/rm/trampoline_64.S
>>> +++ b/arch/x86/realmode/rm/trampoline_64.S
>>> @@ -93,7 +93,9 @@ ENTRY(startup_32)
>>>         movl    %edx, %fs
>>>         movl    %edx, %gs
>>>
>>> -       movl    pa_tr_cr4, %eax
>>> +       movl    $X86_CR4_SMEP, %eax
>>> +       notl    %eax
>>> +       andl    pa_tr_cr4, %eax
>>>         movl    %eax, %cr4              # Enable PAE mode
>>>
>>>         # Setup trampoline 4 level pagetables
>>>
>>> Now I am not completely convinced that this is really the way to go. Likely the
>>> Xen hypervisor should not start up the guest with CR4 on the BP containing those
>>> flags. But maybe it still makes sense to mask some dangerous ones off in the
>>> realmode code (btw, it seemed that masking the assignments in arch_setup or
>>> setup_realmode did not work).
>>>
>>> And finally I am wondering why the SMEP flag in CR4 is set anyway. My
>>> understanding would be that this should only be done if cpuid[7].ebx has bit7
>>> set. And this does not seem to be the case at least on the one box I was doing
>>> the bisection on.
>>
>> Seems that I was relying on the wrong source of information when checking SMEP
>> support. The cpuid command seems at fail. But /proc/cpuinfo reports it. So that
>> at least explains where that comes from... sorry for that.
> 
> OK, so if you boot Xen with smep=1 (which disables SMEP, kind of counterintuive flag)
> that would work fine?
> 
> CC-ing the Intel folks who added this in.
> 

If it is present in /proc/cpuinfo and not in cpuid it means the kernel
thinks it has SMEP but the CPU doesn't... an obvious case of fail.
However, *where the hell* does the bit come from in the first place?

That is what we need to track down.

When you say Xen HVM, am I correct in assuming that neither CPUID nor
CR4 operations in the main kernel are run through paravirt_ops?

	-hpa