From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. Date: Wed, 9 May 2012 13:38:36 -0400 Message-ID: <20120509173836.GB9094@phenom.dumpdata.com> References: <4FA268CD02000078000814E4@nat28.tlf.novell.com> <4FA792C00200007800081F06@nat28.tlf.novell.com> <20120508000813.GA29587@phenom.dumpdata.com> <20120509003510.GA19643@phenom.dumpdata.com> <20120509131153.GA13956@andromeda.dapyr.net> <4FAA8E96020000780008288F@nat28.tlf.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <4FAA8E96020000780008288F@nat28.tlf.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich , law@redhat.com Cc: Jeremy Fitzhardinge , Tim.Deegan@citrix.com, weidong.han@intel.com, haitao.shan@intel.com, stefan.bader@canonical.com, xen-devel , Konrad Rzeszutek Wilk List-Id: xen-devel@lists.xenproject.org On Wed, May 09, 2012 at 02:34:46PM +0100, Jan Beulich wrote: > >>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk wrote: > > On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote: > >> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote: > >> > After some digging around, it looks like this is an Ubuntu 11.10 only > > patch: > >> > > > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb > > a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416 > > 0b > >> > >> > >> Ugh. There are looks to be a bug in Fedora as well: > > https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this. > >> > > > > CC-ing Intel. It seems that the userspace programs are crashingon > > Sandybridge machines even if 'xsave=0' is provided on the command line. > > Any advice? > > Going through the bug, all I can see are bogus attempts to pass > "xsave=" (with value 0 or 1) to the kernel. That's a hypervisor > option though, and the only kernel option that's relevant here > is "noxsave" afaik. > > Further, when the hypervisor has xsave support enabled, there's > (in the pv case) nothing the kernel can do to hide the functionality > from applications, as the hardware's CR4.OSXSAVE will be set, and > hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel > command line when "xsave=1" (or xsave enabled by default as in > 4.2) just doesn't make any sense. > > And finally one always has to keep in mind that there is this nice > glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE > when trying to determine whether AVX or FMA is available. It has this: 146 147 if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX) 148 { 149 /* Reset the AVX bit in case OSXSAVE is disabled. */ 150 if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE) != 0 151 && ({ unsigned int xcrlow; 152 unsigned int xcrhigh; 153 asm ("xgetbv" 154 : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0)); 155 (xcrlow & 6) == 6; })) 156 __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable; 157 } And when I ran a little silly C program (attached) to probe the CPUID flags, I got: /root/test-xsave SSE3 CMOV AVX XSAVE Trying XGETBV Illegal instruction (core dumped) Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable is _not_ set. But then looking at the sources I see: # ifdef USE_AS_STRCASECMP_L 102 ENTRY(__strcasecmp) 103 .type __strcasecmp, @gnu_indirect_function 104 cmpl $0, __cpu_features+KIND_OFFSET(%rip) 105 jne 1f 106 call __init_cpu_features 107 1: 108 # ifdef HAVE_AVX_SUPPORT 109 leaq __strcasecmp_avx(%rip), %rax 110 testl $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip) 111 jnz 2f 112 # endif which would imply that the AVX bit is sampled here instead of the YMM one. Perhaps this is needed: --- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig 2012-05-09 13:32:10.832000122 -0400 +++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c 2012-05-09 13:33:31.043000138 -0400 @@ -154,6 +154,8 @@ __init_cpu_features (void) : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0)); (xcrlow & 6) == 6; })) __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable; + else + __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX; } __cpu_features.family = family; ?