xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

I somehow thought that this has been fixed but I've been
getting reports that people are running into this.

What kind of fix do I need the in the kernel? I see this:
 255         xen_cpuid(&ax, &bx, &cx, &dx);
 256 
 257         xsave_mask =
 258                 (1 << (X86_FEATURE_XSAVE % 32)) |
 259                 (1 << (X86_FEATURE_OSXSAVE % 32));
 260 
 261         /* Xen will set CR4.OSXSAVE if supported and not disabled by force */
 262         if ((cx & xsave_mask) != xsave_mask)
 263                 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE & OSXSAVE */
 264 }

But do I need some other one?

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jan Beulich]

>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> I somehow thought that this has been fixed but I've been
> getting reports that people are running into this.

"this" being what? I too thought that all xsave related issues were
sorted out by now.

Jan

> What kind of fix do I need the in the kernel? I see this:
>  255         xen_cpuid(&ax, &bx, &cx, &dx);
>  256 
>  257         xsave_mask =
>  258                 (1 << (X86_FEATURE_XSAVE % 32)) |
>  259                 (1 << (X86_FEATURE_OSXSAVE % 32));
>  260 
>  261         /* Xen will set CR4.OSXSAVE if supported and not disabled by 
> force */
>  262         if ((cx & xsave_mask) != xsave_mask)
>  263                 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE & 
> OSXSAVE */
>  264 }
> 
> But do I need some other one?

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[AP]

On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>> I somehow thought that this has been fixed but I've been
>> getting reports that people are running into this.
>
> "this" being what? I too thought that all xsave related issues were
> sorted out by now.

I see the following crash if I run without xsave=0 with Ubuntu 11.10
3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.

[    7.509303] invalid opcode: 0000 [#1] SMP
[    7.509319] CPU 0
[    7.509325] Modules linked in:
[    7.509337]
[    7.509347] Pid: 0, comm: swapper Not tainted 3.0.0-17-generic
#30-Ubuntu LENOVO 4286CTO/4286CTO
[    7.509371] RIP: e030:[<ffffffff810140ec>]  [<ffffffff810140ec>]
xstate_enable+0x3c/0x50
[    7.509399] RSP: e02b:ffffffff81c01e58  EFLAGS: 00010046
[    7.509409] RAX: 0000000000000007 RBX: ffffffff81c01e94 RCX: 0000000000000000
[    7.509419] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000002660
[    7.509430] RBP: ffffffff81c01e58 R08: ffffffff81c01e90 R09: ffffffff81c01e94
[    7.509440] R10: 00000000ffffffff R11: 00000000ffffffff R12: ffffffff81c01e90
[    7.509501] R13: ffffffff81c01e8c R14: ffffffff81c01e88 R15: ffff8801e97ecd00
[    7.509519] FS:  0000000000000000(0000) GS:ffff8801e97e0000(0000)
knlGS:0000000000000000
[    7.509530] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    7.509539] CR2: 0000000000000000 CR3: 0000000001c03000 CR4: 0000000000002660
[    7.509550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    7.509561] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    7.509573] Process swapper (pid: 0, threadinfo ffffffff81c00000,
task ffffffff81c0b020)
[    7.509582] Stack:
[    7.509589]  ffffffff81c01ec8 ffffffff81cd97ac 0000000000000040
0000000000000000
[    7.509613]  ffffffff81007b4f ffffffff81004057 0000024000000007
0000000000000340
[    7.509637]  ffff8801e97eb100 0000000000000008 0000000000000004
0000000000000000
[    7.509660] Call Trace:
[    7.509679]  [<ffffffff81cd97ac>] xstate_enable_boot_cpu+0xa9/0x174
[    7.509698]  [<ffffffff81007b4f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[    7.509712]  [<ffffffff81004057>] ? xen_write_cr0+0x77/0x90
[    7.509730]  [<ffffffff815d06eb>] xsave_init+0x26/0x28
[    7.509744]  [<ffffffff815d2932>] cpu_init+0x2bb/0x2d8
[    7.509759]  [<ffffffff81cd5ff4>] trap_init+0x169/0x171
[    7.509774]  [<ffffffff81cd0a27>] start_kernel+0x1d0/0x3df
[    7.509789]  [<ffffffff81cd0388>] x86_64_start_reservations+0x132/0x136
[    7.509805]  [<ffffffff81cd3ec6>] xen_start_kernel+0x5ac/0x5b3
[    7.509814] Code: 00 04 00 ff 14 25 10 33 c1 81 48 89 c7 48 81 cf
00 00 04 00 ff 14 25 18 33 c1 81 48 8b 05 0d 15 db 00 31 c9 48 89 c2
48 c1 ea 20 <0f> 01 d1 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00
00 55
[    7.510045] RIP  [<ffffffff810140ec>] xstate_enable+0x3c/0x50
[    7.510062]  RSP <ffffffff81c01e58>
[    7.510086] ---[ end trace a7919e7f17c0a725 ]---
[    7.510097] Kernel panic - not syncing: Attempted to kill the idle task!
[    7.510109] Pid: 0, comm: swapper Tainted: G      D
3.0.0-17-generic #30-Ubuntu
[    7.510119] Call Trace:
[    7.510134]  [<ffffffff815dca66>] panic+0x91/0x194
[    7.510151]  [<ffffffff8106344b>] do_exit+0x40b/0x440
[    7.510168]  [<ffffffff815f4350>] oops_end+0xb0/0xf0
[    7.510181]  [<ffffffff8100d938>] die+0x58/0x90
[    7.510195]  [<ffffffff815f3a34>] do_trap+0xc4/0x170
[    7.510211]  [<ffffffff8100af25>] do_invalid_op+0x95/0xb0
[    7.510226]  [<ffffffff810140ec>] ? xstate_enable+0x3c/0x50
[    7.510240]  [<ffffffff81007b62>] ? check_events+0x12/0x20
[    7.510255]  [<ffffffff81008239>] ? get_phys_to_machine+0x9/0x70
[    7.510269]  [<ffffffff81005c69>] ? pte_mfn_to_pfn+0x89/0xf0
[    7.510283]  [<ffffffff8100743d>] ? xen_force_evtchn_callback+0xd/0x10
[    7.510299]  [<ffffffff815fc2db>] invalid_op+0x1b/0x20
[    7.510315]  [<ffffffff810140ec>] ? xstate_enable+0x3c/0x50
[    7.510329]  [<ffffffff810140dc>] ? xstate_enable+0x2c/0x50
[    7.510343]  [<ffffffff81cd97ac>] xstate_enable_boot_cpu+0xa9/0x174
[    7.510358]  [<ffffffff81007b4f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[    7.510371]  [<ffffffff81004057>] ? xen_write_cr0+0x77/0x90
[    7.510385]  [<ffffffff815d06eb>] xsave_init+0x26/0x28
[    7.510399]  [<ffffffff815d2932>] cpu_init+0x2bb/0x2d8
[    7.510413]  [<ffffffff81cd5ff4>] trap_init+0x169/0x171
[    7.510427]  [<ffffffff81cd0a27>] start_kernel+0x1d0/0x3df
[    7.510441]  [<ffffffff81cd0388>] x86_64_start_reservations+0x132/0x136
[    7.510457]  [<ffffffff81cd3ec6>] xen_start_kernel+0x5ac/0x5b3
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.


>> What kind of fix do I need the in the kernel? I see this:
>>  255         xen_cpuid(&ax, &bx, &cx, &dx);
>>  256
>>  257         xsave_mask =
>>  258                 (1 << (X86_FEATURE_XSAVE % 32)) |
>>  259                 (1 << (X86_FEATURE_OSXSAVE % 32));
>>  260
>>  261         /* Xen will set CR4.OSXSAVE if supported and not disabled by
>> force */
>>  262         if ((cx & xsave_mask) != xsave_mask)
>>  263                 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE &
>> OSXSAVE */
>>  264 }
>>
>> But do I need some other one?
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jan Beulich]

>>> On 02.05.12 at 20:42, AP <apxeng@gmail.com> wrote:
> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>>> I somehow thought that this has been fixed but I've been
>>> getting reports that people are running into this.
>>
>> "this" being what? I too thought that all xsave related issues were
>> sorted out by now.
> 
> I see the following crash if I run without xsave=0 with Ubuntu 11.10
> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.

And in the thread starting at
http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
I gave debugging instructions that apparently no-one followed so
far. Without someone seeing the problem doing so I don't think we
will ever get anywhere with this (unless, as also indicated there,
someone can spot something wrong with the code that non-obvious
to everyone else).

Jan

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[AP]

On Thu, May 3, 2012 at 2:15 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 02.05.12 at 20:42, AP <apxeng@gmail.com> wrote:
>> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>>>> I somehow thought that this has been fixed but I've been
>>>> getting reports that people are running into this.
>>>
>>> "this" being what? I too thought that all xsave related issues were
>>> sorted out by now.
>>
>> I see the following crash if I run without xsave=0 with Ubuntu 11.10
>> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
>> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.
>
> And in the thread starting at
> http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
> I gave debugging instructions that apparently no-one followed so
> far. Without someone seeing the problem doing so I don't think we
> will ever get anywhere with this (unless, as also indicated there,
> someone can spot something wrong with the code that non-obvious
> to everyone else).

I missed that thread. I will add some debugging to
pv_guest_cr4_fixup() and the XSETBV handling in
emulate_privileged_op() and post the output.

Thanks,
AP

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[AP]

[-- Attachment #1: Type: text/plain, Size: 4079 bytes --]

On Thu, May 3, 2012 at 11:09 AM, AP <apxeng@gmail.com> wrote:
> On Thu, May 3, 2012 at 2:15 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 02.05.12 at 20:42, AP <apxeng@gmail.com> wrote:
>>> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>>>>> I somehow thought that this has been fixed but I've been
>>>>> getting reports that people are running into this.
>>>>
>>>> "this" being what? I too thought that all xsave related issues were
>>>> sorted out by now.
>>>
>>> I see the following crash if I run without xsave=0 with Ubuntu 11.10
>>> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
>>> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.
>>
>> And in the thread starting at
>> http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
>> I gave debugging instructions that apparently no-one followed so
>> far. Without someone seeing the problem doing so I don't think we
>> will ever get anywhere with this (unless, as also indicated there,
>> someone can spot something wrong with the code that non-obvious
>> to everyone else).
>
> I missed that thread. I will add some debugging to
> pv_guest_cr4_fixup() and the XSETBV handling in
> emulate_privileged_op() and post the output.

I ran with the attached xsave_debug patch and saw the following output:
<snip>
(XEN) CPU: After generic identify, caps: bfebfbff 28100800 00000000
00000000 17bae3ff 00000000 00000001 00000000
(XEN) CPU: After vendor identify, caps: bfebfbff 28100800 00000000
00000000 17bae3ff 00000000 00000001 00000000
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) CPU: After all inits, caps: bfebfbff 28100800 00000000 00003f40
17bae3ff 00000000 00000001 00000000
<snip>
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
[    6.791945] invalid opcode: 0000 [#1] SMP
<snip>

>From the above I realized that X86_CR4_OSXSAVE was never getting set
in v->arch.pv_vcpu.ctrlreg[4]. So I tried the following patch:

diff -r 5a0d60bb536b xen/arch/x86/domain.c
--- a/xen/arch/x86/domain.c	Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/arch/x86/domain.c	Fri May 04 12:23:57 2012 -0700
@@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
         hv_cr4_mask &= ~X86_CR4_DE;
     if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
         hv_cr4_mask &= ~X86_CR4_FSGSBASE;
-    if ( xsave_enabled(v) )
-        hv_cr4_mask &= ~X86_CR4_OSXSAVE;

     if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
         gdprintk(XENLOG_WARNING,
diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
--- a/xen/include/asm-x86/domain.h	Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/include/asm-x86/domain.h	Fri May 04 12:23:57 2012 -0700
@@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
      & ~X86_CR4_DE)
 #define real_cr4_to_pv_guest_cr4(c)                         \
     ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
-             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
+             | X86_CR4_SMEP))

 void domain_cpuid(struct domain *d,
                   unsigned int  input,

That allowed the system to boot successfully though I did see the
following message:
(XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660

Not sure if the above patch is right fix but I hope it was at least
helpful in pointing at where the problem might be.

BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.

Thanks,
AP

[-- Attachment #2: xsave_debug.patch --]
[-- Type: application/octet-stream, Size: 2895 bytes --]

diff -r 5a0d60bb536b xen/arch/x86/cpu/common.c
--- a/xen/arch/x86/cpu/common.c	Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/arch/x86/cpu/common.c	Fri May 04 12:01:42 2012 -0700
@@ -305,6 +305,7 @@ static void __cpuinit squash_the_stupid_
 
 #endif
 
+#define NOISY_CAPS 1
 /*
  * This does the hard work of actually picking apart the CPU stuff...
  */
diff -r 5a0d60bb536b xen/arch/x86/domain.c
--- a/xen/arch/x86/domain.c	Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/arch/x86/domain.c	Fri May 04 12:01:42 2012 -0700
@@ -699,6 +699,10 @@ unsigned long pv_guest_cr4_fixup(const s
                  "Attempt to change CR4 flags %08lx -> %08lx\n",
                  hv_cr4, guest_cr4);
 
+    gdprintk(XENLOG_INFO, "vcpu[%d] hv_cr4: 0x%lx hv_cr4_mask: 0x%lx "
+             "returning cr4: 0x%lx\n", v->vcpu_id, hv_cr4, hv_cr4_mask,
+             (hv_cr4 & hv_cr4_mask) | (guest_cr4 & ~hv_cr4_mask));
+
     return (hv_cr4 & hv_cr4_mask) | (guest_cr4 & ~hv_cr4_mask);
 }
 
diff -r 5a0d60bb536b xen/arch/x86/traps.c
--- a/xen/arch/x86/traps.c	Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/arch/x86/traps.c	Fri May 04 12:01:42 2012 -0700
@@ -846,6 +846,8 @@ static void pv_cpuid(struct cpu_user_reg
         __clear_bit(X86_FEATURE_DCA % 32, &c);
         if ( !xsave_enabled(current) )
         {
+            gdprintk(XENLOG_INFO, "vcpu[%d] Clearing XSAVE\n",
+                     get_processor_id());
             __clear_bit(X86_FEATURE_XSAVE % 32, &c);
             __clear_bit(X86_FEATURE_AVX % 32, &c);
         }
@@ -869,8 +871,12 @@ static void pv_cpuid(struct cpu_user_reg
         break;
 
     case 0x0000000d: /* XSAVE */
-        if ( !xsave_enabled(current) )
+        gdprintk(XENLOG_INFO, "vcpu[%d] cpuid XSAVE", get_processor_id());
+        if ( !xsave_enabled(current) ) {
+            printk(" not supported\n");
             goto unsupported;
+        }
+        printk(" supported\n");
         break;
 
     case 0x80000001:
@@ -2243,6 +2249,9 @@ static int emulate_privileged_op(struct 
             if ( lock || rep_prefix || opsize_prefix
                  || !(v->arch.pv_vcpu.ctrlreg[4] & X86_CR4_OSXSAVE) )
             {
+                gdprintk(XENLOG_ERR, "XSETBV: lock: %d rep_prefix: %d "
+                         "opsize_prefix: %d cr4: 0x%lx\n", lock, rep_prefix,
+                         opsize_prefix, v->arch.pv_vcpu.ctrlreg[4]);
                 do_guest_trap(TRAP_invalid_op, regs, 0);
                 goto skip;
             }
@@ -2395,6 +2404,9 @@ static int emulate_privileged_op(struct 
 
         case 4: /* Write CR4 */
             v->arch.pv_vcpu.ctrlreg[4] = pv_guest_cr4_fixup(v, *reg);
+            gdprintk(XENLOG_INFO, "vcpu[%d] pv cr4: 0x%lx write CR4: 0x%lx\n",
+                     v->vcpu_id, v->arch.pv_vcpu.ctrlreg[4],
+                     pv_guest_cr4_to_real_cr4(v));
             write_cr4(pv_guest_cr4_to_real_cr4(v));
             break;
 

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jan Beulich]

>>> On 04.05.12 at 21:30, AP <apxeng@gmail.com> wrote:
> From the above I realized that X86_CR4_OSXSAVE was never getting set
> in v->arch.pv_vcpu.ctrlreg[4].

Yes, that was the observation in the previous thread too, but the
reporter didn't seem interested in continuing on from there.


> So I tried the following patch:
>
> diff -r 5a0d60bb536b xen/arch/x86/domain.c
> --- a/xen/arch/x86/domain.c	Fri Apr 27 21:10:59 2012 -0700
> +++ b/xen/arch/x86/domain.c	Fri May 04 12:23:57 2012 -0700
> @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
>          hv_cr4_mask &= ~X86_CR4_DE;
>      if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
>          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> -    if ( xsave_enabled(v) )
> -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> 
>      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
>          gdprintk(XENLOG_WARNING,
> diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> --- a/xen/include/asm-x86/domain.h	Fri Apr 27 21:10:59 2012 -0700
> +++ b/xen/include/asm-x86/domain.h	Fri May 04 12:23:57 2012 -0700
> @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
>       & ~X86_CR4_DE)
>  #define real_cr4_to_pv_guest_cr4(c)                         \
>      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> +             | X86_CR4_SMEP))
> 
>  void domain_cpuid(struct domain *d,
>                    unsigned int  input,

No, this is specifically the wrong thing. From what we know so far
(i.e. the outcome of the above printing you added) the problem in
in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
attempting XSETBV). What your patch efectively does is take away
control from the guest kernels to control the (virtual) CR4 flag...

> That allowed the system to boot successfully though I did see the
> following message:
> (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660

... which is what this message is telling you.

> Not sure if the above patch is right fix but I hope it was at least
> helpful in pointing at where the problem might be.
> 
> BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.

Sure, as it's a kernel problem. It's the kernel that needs logging added,
to find out why the CR4 write supposedly happening immediately
prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
happen, or doesn't set the flag. Perhaps something fishy going on
with the paravirt ops patching, since the disassembly of the opcode
bytes shown with the oops message are indicating that the right
thing is being attempted:

ff 14 25 10 33 c1 81 	callq  *0xffffffff81c13310
48 89 c7             	mov    %rax,%rdi
48 81 cf 00 00 04 00 	or     $0x40000,%rdi
                     	       ^^^^^^^^
ff 14 25 18 33 c1 81 	callq  *0xffffffff81c13318
48 8b 05 0d 15 db 00 	mov    0xdb150d(%rip),%rax
31 c9                	xor    %ecx,%ecx
48 89 c2             	mov    %rax,%rdx
48 c1 ea 20          	shr    $0x20,%rdx
0f 01 d1             	xsetbv 
5d                   	pop    %rbp
c3                   	retq   

The primary thing that strikes me as odd is that both calls are still
indirect ones, even though I thought that they should get replaced
by direct ones (or even the actual instruction, namely in the
read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?

And the dumped %rdi value indicates that bit 18 did _not_ get set.

Jan

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

On Mon, May 07, 2012 at 08:15:44AM +0100, Jan Beulich wrote:
> >>> On 04.05.12 at 21:30, AP <apxeng@gmail.com> wrote:
> > From the above I realized that X86_CR4_OSXSAVE was never getting set
> > in v->arch.pv_vcpu.ctrlreg[4].
> 
> Yes, that was the observation in the previous thread too, but the
> reporter didn't seem interested in continuing on from there.
> 
> 
> > So I tried the following patch:
> >
> > diff -r 5a0d60bb536b xen/arch/x86/domain.c
> > --- a/xen/arch/x86/domain.c	Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/arch/x86/domain.c	Fri May 04 12:23:57 2012 -0700
> > @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> >          hv_cr4_mask &= ~X86_CR4_DE;
> >      if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
> >          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> > -    if ( xsave_enabled(v) )
> > -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> > 
> >      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
> >          gdprintk(XENLOG_WARNING,
> > diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> > --- a/xen/include/asm-x86/domain.h	Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/include/asm-x86/domain.h	Fri May 04 12:23:57 2012 -0700
> > @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> >       & ~X86_CR4_DE)
> >  #define real_cr4_to_pv_guest_cr4(c)                         \
> >      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> > -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> > +             | X86_CR4_SMEP))
> > 
> >  void domain_cpuid(struct domain *d,
> >                    unsigned int  input,
> 
> No, this is specifically the wrong thing. From what we know so far
> (i.e. the outcome of the above printing you added) the problem in
> in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> attempting XSETBV). What your patch efectively does is take away
> control from the guest kernels to control the (virtual) CR4 flag...
> 
> > That allowed the system to boot successfully though I did see the
> > following message:
> > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
> 
> ... which is what this message is telling you.
> 
> > Not sure if the above patch is right fix but I hope it was at least
> > helpful in pointing at where the problem might be.
> > 
> > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.
> 
> Sure, as it's a kernel problem. It's the kernel that needs logging added,
> to find out why the CR4 write supposedly happening immediately
> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> happen, or doesn't set the flag. Perhaps something fishy going on
> with the paravirt ops patching, since the disassembly of the opcode
> bytes shown with the oops message are indicating that the right
> thing is being attempted:
> 
> ff 14 25 10 33 c1 81 	callq  *0xffffffff81c13310 [so xen_read_cr4 which is native_read_cr4]
> 48 89 c7             	mov    %rax,%rdi
> 48 81 cf 00 00 04 00 	or     $0x40000,%rdi
>                      	       ^^^^^^^^
> ff 14 25 18 33 c1 81 	callq  *0xffffffff81c13318 [so xen_write_cr4] - which is filtering X86_CR4_PGE and X86_CR4_PSE]
> 48 8b 05 0d 15 db 00 	mov    0xdb150d(%rip),%rax
> 31 c9                	xor    %ecx,%ecx
> 48 89 c2             	mov    %rax,%rdx
> 48 c1 ea 20          	shr    $0x20,%rdx
> 0f 01 d1             	xsetbv 
> 5d                   	pop    %rbp
> c3                   	retq   
> 
> The primary thing that strikes me as odd is that both calls are still
> indirect ones, even though I thought that they should get replaced
> by direct ones (or even the actual instruction, namely in the
> read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?

They do get replaced (during runtime - and this is done by the
alternative_instructions). Is this output from objdump or straight
from the memory (so using the Xen debugger?).


> 
> And the dumped %rdi value indicates that bit 18 did _not_ get set.

That would imply that xen_write_cr4, which is just mov to cr4
is getting trapped but somehow the hypervisor isn't setting the
rdi value? Or maybe the the native_write_cr4 ends up
filtering in the wrong order?

   0xffffffff8102e650 <xen_write_cr4>:  push   %rbp
   0xffffffff8102e651 <xen_write_cr4+1>:        and    $0x6f,%dil
   0xffffffff8102e655 <xen_write_cr4+5>:        mov    %rsp,%rbp
   0xffffffff8102e658 <xen_write_cr4+8>:        mov    %rdi,%cr4
   0xffffffff8102e65b <xen_write_cr4+11>:       leaveq
   0xffffffff8102e65c <xen_write_cr4+12>:       retq


> 
> Jan

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jeremy Fitzhardinge]

On 05/07/2012 12:15 AM, Jan Beulich wrote:
> The primary thing that strikes me as odd is that both calls are still
> indirect ones, even though I thought that they should get replaced
> by direct ones (or even the actual instruction, namely in the
> read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?

Patching happens as an explicit step during (moderately late) boot, not
as a side-effect of the first call, so it doesn't surprise me too much
that they haven't been updated at this point.

    J

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[AP]

On Mon, May 7, 2012 at 7:15 AM, Jan Beulich <JBeulich@suse.com> wrote:
>
> >>> On 04.05.12 at 21:30, AP <apxeng@gmail.com> wrote:
> > From the above I realized that X86_CR4_OSXSAVE was never getting set
> > in v->arch.pv_vcpu.ctrlreg[4].
>
> Yes, that was the observation in the previous thread too, but the
> reporter didn't seem interested in continuing on from there.
>
>
> > So I tried the following patch:
> >
> > diff -r 5a0d60bb536b xen/arch/x86/domain.c
> > --- a/xen/arch/x86/domain.c   Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/arch/x86/domain.c   Fri May 04 12:23:57 2012 -0700
> > @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> >          hv_cr4_mask &= ~X86_CR4_DE;
> >      if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
> >          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> > -    if ( xsave_enabled(v) )
> > -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> >
> >      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
> >          gdprintk(XENLOG_WARNING,
> > diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> > --- a/xen/include/asm-x86/domain.h    Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/include/asm-x86/domain.h    Fri May 04 12:23:57 2012 -0700
> > @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> >       & ~X86_CR4_DE)
> >  #define real_cr4_to_pv_guest_cr4(c)                         \
> >      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> > -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> > +             | X86_CR4_SMEP))
> >
> >  void domain_cpuid(struct domain *d,
> >                    unsigned int  input,
>
> No, this is specifically the wrong thing. From what we know so far
> (i.e. the outcome of the above printing you added) the problem in
> in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> attempting XSETBV). What your patch efectively does is take away
> control from the guest kernels to control the (virtual) CR4 flag...
>
> > That allowed the system to boot successfully though I did see the
> > following message:
> > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
>
> ... which is what this message is telling you.
>
> > Not sure if the above patch is right fix but I hope it was at least
> > helpful in pointing at where the problem might be.
> >
> > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > xsave=1.
>
> Sure, as it's a kernel problem. It's the kernel that needs logging added,
> to find out why the CR4 write supposedly happening immediately
> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> happen, or doesn't set the flag. Perhaps something fishy going on

xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.

static void xen_write_cr4(unsigned long cr4)
{
    cr4 &= ~X86_CR4_PGE;
    cr4 &= ~X86_CR4_PSE;
    cr4 &= ~X86_CR4_OSXSAVE;

    native_write_cr4(cr4);
}

I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
is what I see:

<snip>
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
[    6.838041] set_in_cr4: cr4: 0x42660
[    6.841743] xen_write_cr4: cr4: 0x2660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
[    6.860546] xstate_enable: Exec xsetbv
(XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
[    6.870509] invalid opcode: 0000 [#1] SMP
<snip>

Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to boot.

(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    7.554262] Setting X86_CR4_OSXSAVE
[    7.557869] set_in_cr4: cr4: 0x42660
[    7.561551] xen_write_cr4: cr4: 0x42660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x42660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
[    7.580522] Exec xsetbv
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340

Thanks,
AP

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

> > No, this is specifically the wrong thing. From what we know so far
> > (i.e. the outcome of the above printing you added) the problem in
> > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > attempting XSETBV). What your patch efectively does is take away
> > control from the guest kernels to control the (virtual) CR4 flag...
> >
> > > That allowed the system to boot successfully though I did see the
> > > following message:
> > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
> >
> > ... which is what this message is telling you.
> >
> > > Not sure if the above patch is right fix but I hope it was at least
> > > helpful in pointing at where the problem might be.
> > >
> > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > xsave=1.
> >
> > Sure, as it's a kernel problem. It's the kernel that needs logging added,
> > to find out why the CR4 write supposedly happening immediately
> > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > happen, or doesn't set the flag. Perhaps something fishy going on
> 
> xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.

Where did you see that code? Looking at the Linus's tree this is what I see
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD

So who added that code? I am not seeing it in v3.0 either?
> 
> static void xen_write_cr4(unsigned long cr4)
> {
>     cr4 &= ~X86_CR4_PGE;
>     cr4 &= ~X86_CR4_PSE;
>     cr4 &= ~X86_CR4_OSXSAVE;
> 
>     native_write_cr4(cr4);
> }
> 
> I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
> is what I see:
> 
> <snip>
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
> [    6.838041] set_in_cr4: cr4: 0x42660
> [    6.841743] xen_write_cr4: cr4: 0x2660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x2660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
> [    6.860546] xstate_enable: Exec xsetbv
> (XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
> [    6.870509] invalid opcode: 0000 [#1] SMP
> <snip>
> 
> Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to boot.
> 
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    7.554262] Setting X86_CR4_OSXSAVE
> [    7.557869] set_in_cr4: cr4: 0x42660
> [    7.561551] xen_write_cr4: cr4: 0x42660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x42660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
> [    7.580522] Exec xsetbv
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
> 
> Thanks,
> AP

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[AP]

[-- Attachment #1.1: Type: text/plain, Size: 2032 bytes --]

On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:
>
> > > No, this is specifically the wrong thing. From what we know so far
> > > (i.e. the outcome of the above printing you added) the problem in
> > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > attempting XSETBV). What your patch efectively does is take away
> > > control from the guest kernels to control the (virtual) CR4 flag...
> > >
> > > > That allowed the system to boot successfully though I did see the
> > > > following message:
> > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > 00002660
> > >
> > > ... which is what this message is telling you.
> > >
> > > > Not sure if the above patch is right fix but I hope it was at least
> > > > helpful in pointing at where the problem might be.
> > > >
> > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > xsave=1.
> > >
> > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > added,
> > > to find out why the CR4 write supposedly happening immediately
> > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > happen, or doesn't set the flag. Perhaps something fishy going on
> >
> > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
>
> Where did you see that code? Looking at the Linus's tree this is what I
> see
>
>
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
>
> So who added that code? I am not seeing it in v3.0 either?

This is in the Ubuntu 11.10 kernel:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812

After some digging around, it looks like this is an Ubuntu 11.10 only patch:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b

Thanks,
AP

[-- Attachment #1.2: Type: text/html, Size: 3101 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> >
> > > > No, this is specifically the wrong thing. From what we know so far
> > > > (i.e. the outcome of the above printing you added) the problem in
> > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > attempting XSETBV). What your patch efectively does is take away
> > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > >
> > > > > That allowed the system to boot successfully though I did see the
> > > > > following message:
> > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > 00002660
> > > >
> > > > ... which is what this message is telling you.
> > > >
> > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > helpful in pointing at where the problem might be.
> > > > >
> > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > xsave=1.
> > > >
> > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > added,
> > > > to find out why the CR4 write supposedly happening immediately
> > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > >
> > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> >
> > Where did you see that code? Looking at the Linus's tree this is what I
> > see
> >
> >
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> >
> > So who added that code? I am not seeing it in v3.0 either?
> 
> This is in the Ubuntu 11.10 kernel:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> 
> After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b

Which seems to say that Amazon's HV is advertising the OXSAVE bit?

Lets ping them and see if they have some recomendations.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Matt Wilson]

On Tue, May 08, 2012 at 09:39:57AM -0700, Konrad Rzeszutek Wilk wrote:
> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com> wrote:
> > 
> > This is in the Ubuntu 11.10 kernel:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> > 
> > After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
> 
> Which seems to say that Amazon's HV is advertising the OXSAVE bit?
> 
> Lets ping them and see if they have some recomendations.

Hi,

The kernel requirements for EC2 are documented in our User Guide here:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html#PVGRUB_compatible_kernels

Specifically, there's a callout regarding XSAVE: 
  "Kernels that disable the pv-ops XSAVE hypercall are known to work
   on all instance types, whereas those that enable this hypercall
   will fail to launch in some cases. Similarly, non-pv-ops kernels
   that do not adhere to the Xen 3.0.2 interface might fail to launch
   in some cases."

Apologies for the miswording of the note, it should say something like
"kernels that disable the XSAVE capability" instead.

Cheers,

Matt

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> >
> > > > No, this is specifically the wrong thing. From what we know so far
> > > > (i.e. the outcome of the above printing you added) the problem in
> > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > attempting XSETBV). What your patch efectively does is take away
> > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > >
> > > > > That allowed the system to boot successfully though I did see the
> > > > > following message:
> > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > 00002660
> > > >
> > > > ... which is what this message is telling you.
> > > >
> > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > helpful in pointing at where the problem might be.
> > > > >
> > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > xsave=1.
> > > >
> > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > added,
> > > > to find out why the CR4 write supposedly happening immediately
> > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > >
> > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> >
> > Where did you see that code? Looking at the Linus's tree this is what I
> > see
> >
> >
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> >
> > So who added that code? I am not seeing it in v3.0 either?
> 
> This is in the Ubuntu 11.10 kernel:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> 
> After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b


Ugh. There are looks to be a bug in Fedora as well: https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com> wrote:
> > >
> > > > > No, this is specifically the wrong thing. From what we know so far
> > > > > (i.e. the outcome of the above printing you added) the problem in
> > > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > > attempting XSETBV). What your patch efectively does is take away
> > > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > > >
> > > > > > That allowed the system to boot successfully though I did see the
> > > > > > following message:
> > > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > > 00002660
> > > > >
> > > > > ... which is what this message is telling you.
> > > > >
> > > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > > helpful in pointing at where the problem might be.
> > > > > >
> > > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > > xsave=1.
> > > > >
> > > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > > added,
> > > > > to find out why the CR4 write supposedly happening immediately
> > > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > > >
> > > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> > >
> > > Where did you see that code? Looking at the Linus's tree this is what I
> > > see
> > >
> > >
> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> > >
> > > So who added that code? I am not seeing it in v3.0 either?
> > 
> > This is in the Ubuntu 11.10 kernel:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> > 
> > After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
> 
> 
> Ugh. There are looks to be a bug in Fedora as well: https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> 

CC-ing Intel. It seems that the userspace programs are crashingon
Sandybridge machines even if 'xsave=0' is provided on the command line.
Any advice?
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Ian Campbell]

On Wed, 2012-05-09 at 14:11 +0100, Konrad Rzeszutek Wilk wrote:
> On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> > On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > > konrad.wilk@oracle.com> wrote:
> > > >
> > > > > > No, this is specifically the wrong thing. From what we know so far
> > > > > > (i.e. the outcome of the above printing you added) the problem in
> > > > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > > > attempting XSETBV). What your patch efectively does is take away
> > > > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > > > >
> > > > > > > That allowed the system to boot successfully though I did see the
> > > > > > > following message:
> > > > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > > > 00002660
> > > > > >
> > > > > > ... which is what this message is telling you.
> > > > > >
> > > > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > > > helpful in pointing at where the problem might be.
> > > > > > >
> > > > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > > > xsave=1.
> > > > > >
> > > > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > > > added,
> > > > > > to find out why the CR4 write supposedly happening immediately
> > > > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > > > >
> > > > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> > > >
> > > > Where did you see that code? Looking at the Linus's tree this is what I
> > > > see
> > > >
> > > >
> > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> > > >
> > > > So who added that code? I am not seeing it in v3.0 either?
> > > 
> > > This is in the Ubuntu 11.10 kernel:
> > > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> > > 
> > > After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> > > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
> > 
> > 
> > Ugh. There are looks to be a bug in Fedora as well: https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> > 
> 
> CC-ing Intel. It seems that the userspace programs are crashingon
> Sandybridge machines even if 'xsave=0' is provided on the command line.

Is this related to the glibc bug which uses xsave/avx without properly
following the defined procedure to detect if it is available, as
described in http://marc.info/?l=xen-devel&m=133612371602480 ?

That bug has come up a couple of times recently, I'm not sure if this is
the same one though.

Ian.

> Any advice?
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jan Beulich]

>>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
>> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
>> > After some digging around, it looks like this is an Ubuntu 11.10 only 
> patch:
>> > 
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb 
> a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416
> 0b
>> 
>> 
>> Ugh. There are looks to be a bug in Fedora as well: 
> https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
>> 
> 
> CC-ing Intel. It seems that the userspace programs are crashingon
> Sandybridge machines even if 'xsave=0' is provided on the command line.
> Any advice?

Going through the bug, all I can see are bogus attempts to pass
"xsave=" (with value 0 or 1) to the kernel. That's a hypervisor
option though, and the only kernel option that's relevant here
is "noxsave" afaik.

Further, when the hypervisor has xsave support enabled, there's
(in the pv case) nothing the kernel can do to hide the functionality
from applications, as the hardware's CR4.OSXSAVE will be set, and
hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel
command line when "xsave=1" (or xsave enabled by default as in
4.2) just doesn't make any sense.

And finally one always has to keep in mind that there is this nice
glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
when trying to determine whether AVX or FMA is available.

Jan

Jan

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

On Wed, May 09, 2012 at 02:34:46PM +0100, Jan Beulich wrote:
> >>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> > On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> >> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> >> > After some digging around, it looks like this is an Ubuntu 11.10 only 
> > patch:
> >> > 
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb 
> > a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416
> > 0b
> >> 
> >> 
> >> Ugh. There are looks to be a bug in Fedora as well: 
> > https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> >> 
> > 
> > CC-ing Intel. It seems that the userspace programs are crashingon
> > Sandybridge machines even if 'xsave=0' is provided on the command line.
> > Any advice?
> 
> Going through the bug, all I can see are bogus attempts to pass
> "xsave=" (with value 0 or 1) to the kernel. That's a hypervisor
> option though, and the only kernel option that's relevant here
> is "noxsave" afaik.
> 
> Further, when the hypervisor has xsave support enabled, there's
> (in the pv case) nothing the kernel can do to hide the functionality
> from applications, as the hardware's CR4.OSXSAVE will be set, and
> hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel
> command line when "xsave=1" (or xsave enabled by default as in
> 4.2) just doesn't make any sense.
> 
> And finally one always has to keep in mind that there is this nice
> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
> when trying to determine whether AVX or FMA is available.

It has this:
146 
147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
148     {
149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE) != 0
151           && ({ unsigned int xcrlow;
152                 unsigned int xcrhigh;
153                 asm ("xgetbv"
154                      : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
155                 (xcrlow & 6) == 6; }))
156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
157     }


And when I ran a little silly C program (attached) to probe the CPUID flags, I got:
 /root/test-xsave 
SSE3 CMOV
AVX  XSAVE
Trying XGETBV
Illegal instruction (core dumped)

Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest
still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable
is _not_ set. But then looking at the sources I see:

# ifdef USE_AS_STRCASECMP_L
102 ENTRY(__strcasecmp)
103         .type   __strcasecmp, @gnu_indirect_function
104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
105         jne     1f
106         call    __init_cpu_features
107 1:
108 #  ifdef HAVE_AVX_SUPPORT
109         leaq    __strcasecmp_avx(%rip), %rax
110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
111         jnz     2f
112 #  endif

which would imply that the AVX bit is sampled here instead of the
YMM one.

Perhaps this is needed:

--- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig	2012-05-09 13:32:10.832000122 -0400
+++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c	2012-05-09 13:33:31.043000138 -0400
@@ -154,6 +154,8 @@ __init_cpu_features (void)
 		     : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
 		(xcrlow & 6) == 6; }))
 	__cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
+	else
+	__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
     }
 
   __cpu_features.family = family;

?

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jeff Law]

On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>>
>> And finally one always has to keep in mind that there is this nice
>> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
>> when trying to determine whether AVX or FMA is available.
>
> It has this:
> 146
> 147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
> 148     {
> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
> 150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_OSXSAVE) != 0
> 151&&  ({ unsigned int xcrlow;
> 152                 unsigned int xcrhigh;
> 153                 asm ("xgetbv"
> 154                      : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
> 155                 (xcrlow&  6) == 6; }))
> 156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
> 157     }
>
>
> And when I ran a little silly C program (attached) to probe the CPUID flags, I got:
>   /root/test-xsave
> SSE3 CMOV
> AVX  XSAVE
> Trying XGETBV
> Illegal instruction (core dumped)
>
> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest
> still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable
> is _not_ set. But then looking at the sources I see:
>
> # ifdef USE_AS_STRCASECMP_L
> 102 ENTRY(__strcasecmp)
> 103         .type   __strcasecmp, @gnu_indirect_function
> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
> 105         jne     1f
> 106         call    __init_cpu_features
> 107 1:
> 108 #  ifdef HAVE_AVX_SUPPORT
> 109         leaq    __strcasecmp_avx(%rip), %rax
> 110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
> 111         jnz     2f
> 112 #  endif
>
> which would imply that the AVX bit is sampled here instead of the
> YMM one.
[ ... ]
I think Uli's position is that this code only uses AVX encodings, but 
not the YMM registers and thus the right check is for AVX.

That doesn't make sense to me given the text under availability and 
support here:

http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/

According to my reading AVX can only be used if the hardware supports 
AVX *and* the OS supports XSAVE.    The only weasel language is "To use 
the Intel AVX extensions reliably in most settings ..."  Which Uli might 
be relying upon for his position.

Ironically, the code in init-arch used to look like:

  if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
     {
       /* Reset the AVX bit in case OSXSAVE is disabled.  */
       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & 
bit_OSXSAVE) == 0
           || ({ unsigned int xcrlow;
               unsigned int xcrhigh;
               asm ("xgetbv"
                    : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
               (xcrlow & 6) != 6; }))
         __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
     }


Which I think would have done the right thing.   Uli changed it to the 
form you quoted just 2 hours after installing the version I quoted.

If i'm going to make the claim Uli is wrong, some clarification from 
Intel would be appreciated.


jeff

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

[-- Attachment #1.1: Type: text/plain, Size: 6851 bytes --]

On May 10, 2012 3:40 PM, "Jeff Law" <law@redhat.com> wrote:
>
> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>>>
>>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at
CPUID.OSXSAVE
>>> when trying to determine whether AVX or FMA is available.
>>
>>
>> It has this:
>> 146
>> 147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
>>
>> 148     {
>> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>> 150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&
 bit_OSXSAVE) != 0
>> 151&&  ({ unsigned int xcrlow;
>>
>> 152                 unsigned int xcrhigh;
>> 153                 asm ("xgetbv"
>> 154                      : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>> 155                 (xcrlow&  6) == 6; }))
>>
>> 156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
>> 157     }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID
flags, I got:
>>  /root/test-xsave
>> SSE3 CMOV
>> AVX  XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit
PV guest
>> still crashes on that machine) - which means that in multi-lib the
bit_YMM_Usable
>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103         .type   __strcasecmp, @gnu_indirect_function
>> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
>> 105         jne     1f
>> 106         call    __init_cpu_features
>> 107 1:
>> 108 #  ifdef HAVE_AVX_SUPPORT
>> 109         leaq    __strcasecmp_avx(%rip), %rax
>> 110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
>> 111         jnz     2f
>> 112 #  endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
>
> [ ... ]
> I think Uli's position is that this code only uses AVX encodings, but not
the YMM registers and thus the right check is for AVX.
>
> That doesn't make sense to me given the text under availability and
support here:
>
>
http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/
>
> According to my reading AVX can only be used if the hardware supports AVX
*and* the OS supports XSAVE.    The only weasel language is "To use the
Intel AVX extensions reliably in most settings ..."  Which Uli might be
relying upon for his position.
>
> Ironically, the code in init-arch used to look like:
>
>
>  if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
>    {
>
>      /* Reset the AVX bit in case OSXSAVE is disabled.  */
>      if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE)
== 0
>          || ({ unsigned int xcrlow;
>              unsigned int xcrhigh;
>              asm ("xgetbv"
>
>                   : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>              (xcrlow & 6) != 6; }))
>
>        __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
>    }
>
>
> Which I think would have done the right thing.   Uli changed it to the
form you quoted just 2 hours after installing the version I quoted.

Sadly no as it would have executed the xgetv instruction. Since the first
part of the boolean logic returns false.
>
> If i'm going to make the claim Uli is wrong, some clarification from
Intel would be appreciated.
>
>
> jeff
>
 On May 10, 2012 3:40 PM, "Jeff Law" <law@redhat.com> wrote:

> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>
>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
>>> when trying to determine whether AVX or FMA is available.
>>>
>>
>> It has this:
>> 146
>> 147   if (__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx&  bit_AVX)
>> 148     {
>> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>> 150       if ((__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx&
>>  bit_OSXSAVE) != 0
>> 151&&  ({ unsigned int xcrlow;
>> 152                 unsigned int xcrhigh;
>> 153                 asm ("xgetbv"
>> 154                      : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>> 155                 (xcrlow&  6) == 6; }))
>> 156         __cpu_features.feature[index_**YMM_Usable] |= bit_YMM_Usable;
>> 157     }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID
>> flags, I got:
>>  /root/test-xsave
>> SSE3 CMOV
>> AVX  XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit
>> PV guest
>> still crashes on that machine) - which means that in multi-lib the
>> bit_YMM_Usable
>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103         .type   __strcasecmp, @gnu_indirect_function
>> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%**rip)
>> 105         jne     1f
>> 106         call    __init_cpu_features
>> 107 1:
>> 108 #  ifdef HAVE_AVX_SUPPORT
>> 109         leaq    __strcasecmp_avx(%rip), %rax
>> 110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+**
>> index_AVX(%rip)
>> 111         jnz     2f
>> 112 #  endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
>>
> [ ... ]
> I think Uli's position is that this code only uses AVX encodings, but not
> the YMM registers and thus the right check is for AVX.
>
> That doesn't make sense to me given the text under availability and
> support here:
>
> http://software.intel.com/en-**us/articles/introduction-to-**
> intel-advanced-vector-**extensions/<http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/>
>
> According to my reading AVX can only be used if the hardware supports AVX
> *and* the OS supports XSAVE.    The only weasel language is "To use the
> Intel AVX extensions reliably in most settings ..."  Which Uli might be
> relying upon for his position.
>
> Ironically, the code in init-arch used to look like:
>
>  if (__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx & bit_AVX)
>    {
>      /* Reset the AVX bit in case OSXSAVE is disabled.  */
>      if ((__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx & bit_OSXSAVE)
> == 0
>          || ({ unsigned int xcrlow;
>              unsigned int xcrhigh;
>              asm ("xgetbv"
>                   : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>              (xcrlow & 6) != 6; }))
>        __cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx &= ~bit_AVX;
>    }
>
>
> Which I think would have done the right thing.   Uli changed it to the
> form you quoted just 2 hours after installing the version I quoted.
>
> If i'm going to make the claim Uli is wrong, some clarification from Intel
> would be appreciated.
>
>
> jeff
>
>

[-- Attachment #1.2: Type: text/html, Size: 8907 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk]

>> Ironically, the code in init-arch used to look like:
>>
>>
>>  if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
>>    {
>>
>>      /* Reset the AVX bit in case OSXSAVE is disabled.  */
>>      if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE) ==
>> 0
>>          || ({ unsigned int xcrlow;
>>              unsigned int xcrhigh;
>>              asm ("xgetbv"
>>
>>                   : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>>              (xcrlow & 6) != 6; }))
>>
>>        __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
>>    }
>>
>>
>> Which I think would have done the right thing.   Uli changed it to the
>> form you quoted just 2 hours after installing the version I quoted.
>
> Sadly no as it would have executed the xgetv instruction. Since the first
> part of the boolean logic returns false.

<sigh> And that is what I get from typing this while stopping at
lights and being in a hurry and doing this on a cellphone.
Please ignore what I said above - the earlier version would have worked correct.

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jeff Law]

On 05/10/2012 06:58 PM, Konrad Rzeszutek Wilk wrote:
>>> Ironically, the code in init-arch used to look like:
>>>
>>>
>>>   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
>>>     {
>>>
>>>       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>>>       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_OSXSAVE) ==
>>> 0
>>>           || ({ unsigned int xcrlow;
>>>               unsigned int xcrhigh;
>>>               asm ("xgetbv"
>>>
>>>                    : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>>>               (xcrlow&  6) != 6; }))
>>>
>>>         __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&= ~bit_AVX;
>>>     }
>>>
>>>
>>> Which I think would have done the right thing.   Uli changed it to the
>>> form you quoted just 2 hours after installing the version I quoted.
>>
>> Sadly no as it would have executed the xgetv instruction. Since the first
>> part of the boolean logic returns false.
>
> <sigh>  And that is what I get from typing this while stopping at
> lights and being in a hurry and doing this on a cellphone.
> Please ignore what I said above - the earlier version would have worked correct.
No worries :-)

Had you not gotten involved, I never would have seen the xen-devel 
thread which then referred me to the appropriate Intel doc to confirm 
that your patch should fix the problem.  Your input on this issue has 
been greatly appreciated.

jeff

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jan Beulich]

>>> On 10.05.12 at 21:39, Jeff Law <law@redhat.com> wrote:
> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
>>> when trying to determine whether AVX or FMA is available.
>>
>> It has this:
>> 146
>> 147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_AVX)
>> 148     {
>> 149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
>> 150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&  bit_OSXSAVE) 
> != 0
>> 151&&  ({ unsigned int xcrlow;
>> 152                 unsigned int xcrhigh;
>> 153                 asm ("xgetbv"
>> 154                      : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>> 155                 (xcrlow&  6) == 6; }))
>> 156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
>> 157     }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID flags, 
> I got:
>>   /root/test-xsave
>> SSE3 CMOV
>> AVX  XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV 
> guest
>> still crashes on that machine) - which means that in multi-lib the 
> bit_YMM_Usable
>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103         .type   __strcasecmp, @gnu_indirect_function
>> 104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
>> 105         jne     1f
>> 106         call    __init_cpu_features
>> 107 1:
>> 108 #  ifdef HAVE_AVX_SUPPORT
>> 109         leaq    __strcasecmp_avx(%rip), %rax
>> 110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
>> 111         jnz     2f
>> 112 #  endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
> [ ... ]
> I think Uli's position is that this code only uses AVX encodings, but 
> not the YMM registers and thus the right check is for AVX.

Which is wrong: (Almost?) all VEX encoded operations act on the full
YMM registers, even if they operate only on 128 bits - the upper 128
bits get zeroed in that case.

Plus all exception type descriptions in the SDM are clearly indicating
that VEX encoded AVX instructions require CR4.OSXSAVE (mirrored
in CPUID.OSXSAVE) to be set along with the corresponding specific
CPUID feature flag(s).

Jan

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jeff Law]

On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:

>
> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest
> still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable
> is _not_ set. But then looking at the sources I see:
What's even more amusing?  Just after installing the incorrect feature 
check and introducing bit_YMM_Usable, Uli reverted all the tests which 
had been checking for usable YMM regs and made them check AVX again.

  one.
>
> Perhaps this is needed:
>
> --- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig	2012-05-09 13:32:10.832000122 -0400
> +++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c	2012-05-09 13:33:31.043000138 -0400
> @@ -154,6 +154,8 @@ __init_cpu_features (void)
>   		     : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>   		(xcrlow&  6) == 6; }))
>   	__cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
> +	else
> +	__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&= ~bit_AVX;
>       }
>
>     __cpu_features.family = family;
I certainly agree.  The big problem here is testing...  I still can't 
test it :(  Against my better judgment I may have to throw a glibc with 
that change over the wall and hope.  There's been far too much of that 
already.

jeff

Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Jan Beulich]

>>> On 08.05.12 at 01:57, AP <apxeng@gmail.com> wrote:
> On Mon, May 7, 2012 at 7:15 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> Sure, as it's a kernel problem. It's the kernel that needs logging added,
>> to find out why the CR4 write supposedly happening immediately
>> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
>> happen, or doesn't set the flag. Perhaps something fishy going on
> 
> xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> 
> static void xen_write_cr4(unsigned long cr4)
> {
>     cr4 &= ~X86_CR4_PGE;
>     cr4 &= ~X86_CR4_PSE;
>     cr4 &= ~X86_CR4_OSXSAVE;
> 
>     native_write_cr4(cr4);
> }

That's definitely not the case in _any_ upstream Linux release.
Which means that this must be introduced by a distro-specific
(and broken) patch.

Jan