Re: Xen HVM regression on certain Intel CPUs

[Stefan Bader <stefan.bader@canonical.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2673 bytes --]

On 27.03.2013 16:26, Stefan Bader wrote:
> Recently I ran some experiments on newer hardware and realized that when booting
> any kernel newer or equal to v3.5 (Xen version 4.2.1) in 64bit mode would fail
> to bring up any APs (message about CPU Stuck). I was able to normally bisect
> into a range of realmode changes and then manually drill down to the following
> commit:
> 
> commit cda846f101fb1396b6924f1d9b68ac3d42de5403
> Author: Jarkko Sakkinen <jarkko.sakkinen@intel.com>
> Date:   Tue May 8 21:22:46 2012 +0300
> 
>     x86, realmode: read cr4 and EFER from kernel for 64-bit trampoline
> 
>     This patch changes 64-bit trampoline so that CR4 and
>     EFER are provided by the kernel instead of using fixed
>     values.
> 
> From the Xen debugging console it was possible to gather a bit more data which
> pointed to a failure very close to setting CR4 in startup_32. On this particular
> hardware the saved CR4 (about to be set) was 0x1407f0.
> 
> This would set two flags that somehow feel dangerous: PGE (page global enable)
> and SMEP (supervisor mode execution protection). SMEP turns out to be the main
> offender and the following change allows the APs to start:
> 
> --- a/arch/x86/realmode/rm/trampoline_64.S
> +++ b/arch/x86/realmode/rm/trampoline_64.S
> @@ -93,7 +93,9 @@ ENTRY(startup_32)
>         movl    %edx, %fs
>         movl    %edx, %gs
> 
> -       movl    pa_tr_cr4, %eax
> +       movl    $X86_CR4_SMEP, %eax
> +       notl    %eax
> +       andl    pa_tr_cr4, %eax
>         movl    %eax, %cr4              # Enable PAE mode
> 
>         # Setup trampoline 4 level pagetables
> 
> Now I am not completely convinced that this is really the way to go. Likely the
> Xen hypervisor should not start up the guest with CR4 on the BP containing those
> flags. But maybe it still makes sense to mask some dangerous ones off in the
> realmode code (btw, it seemed that masking the assignments in arch_setup or
> setup_realmode did not work).
> 
> And finally I am wondering why the SMEP flag in CR4 is set anyway. My
> understanding would be that this should only be done if cpuid[7].ebx has bit7
> set. And this does not seem to be the case at least on the one box I was doing
> the bisection on.

Seems that I was relying on the wrong source of information when checking SMEP
support. The cpuid command seems at fail. But /proc/cpuinfo reports it. So that
at least explains where that comes from... sorry for that.
> 
> -Stefan
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel