Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, May 09, 2012 at 02:34:46PM +0100, Jan Beulich wrote:
> >>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> > On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> >> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> >> > After some digging around, it looks like this is an Ubuntu 11.10 only 
> > patch:
> >> > 
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb 
> > a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416
> > 0b
> >> 
> >> 
> >> Ugh. There are looks to be a bug in Fedora as well: 
> > https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> >> 
> > 
> > CC-ing Intel. It seems that the userspace programs are crashingon
> > Sandybridge machines even if 'xsave=0' is provided on the command line.
> > Any advice?
> 
> Going through the bug, all I can see are bogus attempts to pass
> "xsave=" (with value 0 or 1) to the kernel. That's a hypervisor
> option though, and the only kernel option that's relevant here
> is "noxsave" afaik.
> 
> Further, when the hypervisor has xsave support enabled, there's
> (in the pv case) nothing the kernel can do to hide the functionality
> from applications, as the hardware's CR4.OSXSAVE will be set, and
> hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel
> command line when "xsave=1" (or xsave enabled by default as in
> 4.2) just doesn't make any sense.
> 
> And finally one always has to keep in mind that there is this nice
> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
> when trying to determine whether AVX or FMA is available.

It has this:
146 
147   if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
148     {
149       /* Reset the AVX bit in case OSXSAVE is disabled.  */
150       if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE) != 0
151           && ({ unsigned int xcrlow;
152                 unsigned int xcrhigh;
153                 asm ("xgetbv"
154                      : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
155                 (xcrlow & 6) == 6; }))
156         __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
157     }


And when I ran a little silly C program (attached) to probe the CPUID flags, I got:
 /root/test-xsave 
SSE3 CMOV
AVX  XSAVE
Trying XGETBV
Illegal instruction (core dumped)

Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest
still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable
is _not_ set. But then looking at the sources I see:

# ifdef USE_AS_STRCASECMP_L
102 ENTRY(__strcasecmp)
103         .type   __strcasecmp, @gnu_indirect_function
104         cmpl    $0, __cpu_features+KIND_OFFSET(%rip)
105         jne     1f
106         call    __init_cpu_features
107 1:
108 #  ifdef HAVE_AVX_SUPPORT
109         leaq    __strcasecmp_avx(%rip), %rax
110         testl   $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
111         jnz     2f
112 #  endif

which would imply that the AVX bit is sampled here instead of the
YMM one.

Perhaps this is needed:

--- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig	2012-05-09 13:32:10.832000122 -0400
+++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c	2012-05-09 13:33:31.043000138 -0400
@@ -154,6 +154,8 @@ __init_cpu_features (void)
 		     : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
 		(xcrlow & 6) == 6; }))
 	__cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
+	else
+	__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
     }
 
   __cpu_features.family = family;

?