From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stefan Bader Subject: Re: Xen HVM regression on certain Intel CPUs Date: Wed, 27 Mar 2013 18:28:35 +0100 Message-ID: <51532C43.8010808@canonical.com> References: <51530F9F.10805@canonical.com> <515315EC.4030803@canonical.com> <20130327160427.GB6688@phenom.dumpdata.com> <5153222B.3030605@canonical.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2517219259483955308==" Return-path: In-Reply-To: <5153222B.3030605@canonical.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: wei.y.yang@intel.com, "xen-devel@lists.xensource.com" , haitao.shan@intel.com, xin.li@intel.com, "H. Peter Anvin" List-Id: xen-devel@lists.xenproject.org This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --===============2517219259483955308== Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="------------enig937FA0125DBAB2E92D1CBF53" This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig937FA0125DBAB2E92D1CBF53 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 27.03.2013 17:45, Stefan Bader wrote: > On 27.03.2013 17:04, Konrad Rzeszutek Wilk wrote: >> On Wed, Mar 27, 2013 at 04:53:16PM +0100, Stefan Bader wrote: >>> On 27.03.2013 16:26, Stefan Bader wrote: >>>> Recently I ran some experiments on newer hardware and realized that = when booting >>>> any kernel newer or equal to v3.5 (Xen version 4.2.1) in 64bit mode = would fail >>>> to bring up any APs (message about CPU Stuck). I was able to normall= y bisect >>>> into a range of realmode changes and then manually drill down to the= following >>>> commit: >>>> >>>> commit cda846f101fb1396b6924f1d9b68ac3d42de5403 >>>> Author: Jarkko Sakkinen >>>> Date: Tue May 8 21:22:46 2012 +0300 >>>> >>>> x86, realmode: read cr4 and EFER from kernel for 64-bit trampoli= ne >>>> >>>> This patch changes 64-bit trampoline so that CR4 and >>>> EFER are provided by the kernel instead of using fixed >>>> values. >>>> >>>> From the Xen debugging console it was possible to gather a bit more = data which >>>> pointed to a failure very close to setting CR4 in startup_32. On thi= s particular >>>> hardware the saved CR4 (about to be set) was 0x1407f0. >>>> >>>> This would set two flags that somehow feel dangerous: PGE (page glob= al enable) >>>> and SMEP (supervisor mode execution protection). SMEP turns out to b= e the main >>>> offender and the following change allows the APs to start: >>>> >>>> --- a/arch/x86/realmode/rm/trampoline_64.S >>>> +++ b/arch/x86/realmode/rm/trampoline_64.S >>>> @@ -93,7 +93,9 @@ ENTRY(startup_32) >>>> movl %edx, %fs >>>> movl %edx, %gs >>>> >>>> - movl pa_tr_cr4, %eax >>>> + movl $X86_CR4_SMEP, %eax >>>> + notl %eax >>>> + andl pa_tr_cr4, %eax >>>> movl %eax, %cr4 # Enable PAE mode >>>> >>>> # Setup trampoline 4 level pagetables >>>> >>>> Now I am not completely convinced that this is really the way to go.= Likely the >>>> Xen hypervisor should not start up the guest with CR4 on the BP cont= aining those >>>> flags. But maybe it still makes sense to mask some dangerous ones of= f in the >>>> realmode code (btw, it seemed that masking the assignments in arch_s= etup or >>>> setup_realmode did not work). >>>> >>>> And finally I am wondering why the SMEP flag in CR4 is set anyway. M= y >>>> understanding would be that this should only be done if cpuid[7].ebx= has bit7 >>>> set. And this does not seem to be the case at least on the one box I= was doing >>>> the bisection on. >>> >>> Seems that I was relying on the wrong source of information when chec= king SMEP >>> support. The cpuid command seems at fail. But /proc/cpuinfo reports i= t. So that >>> at least explains where that comes from... sorry for that. >> >> OK, so if you boot Xen with smep=3D1 (which disables SMEP, kind of cou= nterintuive flag) >> that would work fine? >=20 > Rebooting with smep=3D1 as a hv argument does not fix it. But I would b= e careful > since I just quickly did this without checking whether Xen 4.2.1 undest= ands the > flag already. I will need more time to look into this (and unlikely today) but it feels= like at least the cpuid flags passed on to HVM guest may be not influenced by = the smep boot argument. Probably rather something I could do by masking in th= e config of the guest (which could be another pain as I normally configure = those via libvirt). >=20 > Second using x86info --all on bare metal does show bits set for cpuid[7= ] and > /proc/cpuinfo values are consistent across BP and APs. So I am a tool f= or using > the wrong tool there. >=20 > So I would say the main issue to look at is why reading cr4 as a HVM gu= est > produces the flags on boot. Surely the hypervisor itself has set certai= n things > up but likely there are some epxectations about the initial setup on bo= ot. >=20 >> >> CC-ing the Intel folks who added this in. >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >> >=20 >=20 >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >=20 --------------enig937FA0125DBAB2E92D1CBF53 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCgAGBQJRUyxDAAoJEOhnXe7L7s6jbWYP/jRcpyNhOC2ju9j4rp9iEN9t Uzyo3Cs81SC9ojcCYJ8SwXDPA4IZYKFPIFSLY44fosGu6/9u0hp9dFB0LZgKADuH joF2VD958Xya4J4K6ZFkpYAbFBOkZ0zTjNNMfVv2z40/ByvItHBuFAcvm7seQ8lr kj9/2kEq0K0kmFhsDeLKoCFGxRgyNFEril2FvxSr5uaAVg6SrgQLir8vx1c4E4EH m93XRqacXe3p5np4QLBZPrsO2dZtSrvYbIUpMeg25VOIxW41OlL0ak4YvuAMH7XX mp4ueyH3Lnb65psFDC1bYh4+0uA2VibSIOfhRHMebtCeZf8u+NHBBKOjaGaurDBy QDDKDfvzroms0kGJbZKKelYVwH7aLkmEsTzqbFPg2ezAGaJ+kTTyV3OOG2RQwmCJ mSP5oYHM/ETjEm73e2ZRjxhk2UNLnD4WLMYNpOmBWlNzHt2qg/fLf6mh0fXNLG+8 9VUBhG4urPTNaoqjBM0OMRO/lKK+mIj/Nw9dXojhP7BLBqPTN0xLYzH1vhJCBKVJ VvkdR3+7OO/gsq8fR8zDiLyfB0ferHaFw8NbFy95MplDRLEMyqQ+RXLgKcKbJVvg oyFmgCgXXIrU0DHR5xXLC6KijMTe9zVa8k+PCsyGAUwfTC8WTIAujszS3mgTYAD4 M11RfeZ4Hng/ctKGzuXN =rqzd -----END PGP SIGNATURE----- --------------enig937FA0125DBAB2E92D1CBF53-- --===============2517219259483955308== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2517219259483955308==--