Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> > No, this is specifically the wrong thing. From what we know so far
> > (i.e. the outcome of the above printing you added) the problem in
> > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > attempting XSETBV). What your patch efectively does is take away
> > control from the guest kernels to control the (virtual) CR4 flag...
> >
> > > That allowed the system to boot successfully though I did see the
> > > following message:
> > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
> >
> > ... which is what this message is telling you.
> >
> > > Not sure if the above patch is right fix but I hope it was at least
> > > helpful in pointing at where the problem might be.
> > >
> > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > xsave=1.
> >
> > Sure, as it's a kernel problem. It's the kernel that needs logging added,
> > to find out why the CR4 write supposedly happening immediately
> > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > happen, or doesn't set the flag. Perhaps something fishy going on
> 
> xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.

Where did you see that code? Looking at the Linus's tree this is what I see
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD

So who added that code? I am not seeing it in v3.0 either?
> 
> static void xen_write_cr4(unsigned long cr4)
> {
>     cr4 &= ~X86_CR4_PGE;
>     cr4 &= ~X86_CR4_PSE;
>     cr4 &= ~X86_CR4_OSXSAVE;
> 
>     native_write_cr4(cr4);
> }
> 
> I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
> is what I see:
> 
> <snip>
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
> [    6.838041] set_in_cr4: cr4: 0x42660
> [    6.841743] xen_write_cr4: cr4: 0x2660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x2660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
> [    6.860546] xstate_enable: Exec xsetbv
> (XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
> [    6.870509] invalid opcode: 0000 [#1] SMP
> <snip>
> 
> Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to boot.
> 
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    7.554262] Setting X86_CR4_OSXSAVE
> [    7.557869] set_in_cr4: cr4: 0x42660
> [    7.561551] xen_write_cr4: cr4: 0x42660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x42660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
> [    7.580522] Exec xsetbv
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
> 
> Thanks,
> AP