Re: Xen HVM regression on certain Intel CPUs

[Stefan Bader <stefan.bader@canonical.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4254 bytes --]

On 27.03.2013 17:45, Stefan Bader wrote:
> On 27.03.2013 17:04, Konrad Rzeszutek Wilk wrote:
>> On Wed, Mar 27, 2013 at 04:53:16PM +0100, Stefan Bader wrote:
>>> On 27.03.2013 16:26, Stefan Bader wrote:
>>>> Recently I ran some experiments on newer hardware and realized that when booting
>>>> any kernel newer or equal to v3.5 (Xen version 4.2.1) in 64bit mode would fail
>>>> to bring up any APs (message about CPU Stuck). I was able to normally bisect
>>>> into a range of realmode changes and then manually drill down to the following
>>>> commit:
>>>>
>>>> commit cda846f101fb1396b6924f1d9b68ac3d42de5403
>>>> Author: Jarkko Sakkinen <jarkko.sakkinen@intel.com>
>>>> Date:   Tue May 8 21:22:46 2012 +0300
>>>>
>>>>     x86, realmode: read cr4 and EFER from kernel for 64-bit trampoline
>>>>
>>>>     This patch changes 64-bit trampoline so that CR4 and
>>>>     EFER are provided by the kernel instead of using fixed
>>>>     values.
>>>>
>>>> From the Xen debugging console it was possible to gather a bit more data which
>>>> pointed to a failure very close to setting CR4 in startup_32. On this particular
>>>> hardware the saved CR4 (about to be set) was 0x1407f0.
>>>>
>>>> This would set two flags that somehow feel dangerous: PGE (page global enable)
>>>> and SMEP (supervisor mode execution protection). SMEP turns out to be the main
>>>> offender and the following change allows the APs to start:
>>>>
>>>> --- a/arch/x86/realmode/rm/trampoline_64.S
>>>> +++ b/arch/x86/realmode/rm/trampoline_64.S
>>>> @@ -93,7 +93,9 @@ ENTRY(startup_32)
>>>>         movl    %edx, %fs
>>>>         movl    %edx, %gs
>>>>
>>>> -       movl    pa_tr_cr4, %eax
>>>> +       movl    $X86_CR4_SMEP, %eax
>>>> +       notl    %eax
>>>> +       andl    pa_tr_cr4, %eax
>>>>         movl    %eax, %cr4              # Enable PAE mode
>>>>
>>>>         # Setup trampoline 4 level pagetables
>>>>
>>>> Now I am not completely convinced that this is really the way to go. Likely the
>>>> Xen hypervisor should not start up the guest with CR4 on the BP containing those
>>>> flags. But maybe it still makes sense to mask some dangerous ones off in the
>>>> realmode code (btw, it seemed that masking the assignments in arch_setup or
>>>> setup_realmode did not work).
>>>>
>>>> And finally I am wondering why the SMEP flag in CR4 is set anyway. My
>>>> understanding would be that this should only be done if cpuid[7].ebx has bit7
>>>> set. And this does not seem to be the case at least on the one box I was doing
>>>> the bisection on.
>>>
>>> Seems that I was relying on the wrong source of information when checking SMEP
>>> support. The cpuid command seems at fail. But /proc/cpuinfo reports it. So that
>>> at least explains where that comes from... sorry for that.
>>
>> OK, so if you boot Xen with smep=1 (which disables SMEP, kind of counterintuive flag)
>> that would work fine?
> 
> Rebooting with smep=1 as a hv argument does not fix it. But I would be careful
> since I just quickly did this without checking whether Xen 4.2.1 undestands the
> flag already.

I will need more time to look into this (and unlikely today) but it feels like
at least the cpuid flags passed on to HVM guest may be not influenced by the
smep boot argument. Probably rather something I could do by masking in the
config of the guest (which could be another pain as I normally configure those
via libvirt).

> 
> Second using x86info --all on bare metal does show bits set for cpuid[7] and
> /proc/cpuinfo values are consistent across BP and APs. So I am a tool for using
> the wrong tool there.
> 
> So I would say the main issue to look at is why reading cr4 as a HVM guest
> produces the flags on boot. Surely the hypervisor itself has set certain things
> up but likely there are some epxectations about the initial setup on boot.
> 
>>
>> CC-ing the Intel folks who added this in.
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel