From mboxrd@z Thu Jan 1 00:00:00 1970 From: "H. Peter Anvin" Subject: Re: Xen HVM regression on certain Intel CPUs Date: Wed, 27 Mar 2013 09:09:19 -0700 Message-ID: <515319AF.1050900@zytor.com> References: <51530F9F.10805@canonical.com> <515315EC.4030803@canonical.com> <20130327160427.GB6688@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20130327160427.GB6688@phenom.dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: wei.y.yang@intel.com, "xen-devel@lists.xensource.com" , haitao.shan@intel.com, xin.li@intel.com, Stefan Bader List-Id: xen-devel@lists.xenproject.org On 03/27/2013 09:04 AM, Konrad Rzeszutek Wilk wrote: >>> >>> From the Xen debugging console it was possible to gather a bit more data which >>> pointed to a failure very close to setting CR4 in startup_32. On this particular >>> hardware the saved CR4 (about to be set) was 0x1407f0. >>> >>> This would set two flags that somehow feel dangerous: PGE (page global enable) >>> and SMEP (supervisor mode execution protection). SMEP turns out to be the main >>> offender and the following change allows the APs to start: >>> >>> --- a/arch/x86/realmode/rm/trampoline_64.S >>> +++ b/arch/x86/realmode/rm/trampoline_64.S >>> @@ -93,7 +93,9 @@ ENTRY(startup_32) >>> movl %edx, %fs >>> movl %edx, %gs >>> >>> - movl pa_tr_cr4, %eax >>> + movl $X86_CR4_SMEP, %eax >>> + notl %eax >>> + andl pa_tr_cr4, %eax >>> movl %eax, %cr4 # Enable PAE mode >>> >>> # Setup trampoline 4 level pagetables >>> >>> Now I am not completely convinced that this is really the way to go. Likely the >>> Xen hypervisor should not start up the guest with CR4 on the BP containing those >>> flags. But maybe it still makes sense to mask some dangerous ones off in the >>> realmode code (btw, it seemed that masking the assignments in arch_setup or >>> setup_realmode did not work). >>> >>> And finally I am wondering why the SMEP flag in CR4 is set anyway. My >>> understanding would be that this should only be done if cpuid[7].ebx has bit7 >>> set. And this does not seem to be the case at least on the one box I was doing >>> the bisection on. >> >> Seems that I was relying on the wrong source of information when checking SMEP >> support. The cpuid command seems at fail. But /proc/cpuinfo reports it. So that >> at least explains where that comes from... sorry for that. > > OK, so if you boot Xen with smep=1 (which disables SMEP, kind of counterintuive flag) > that would work fine? > > CC-ing the Intel folks who added this in. > If it is present in /proc/cpuinfo and not in cpuid it means the kernel thinks it has SMEP but the CPU doesn't... an obvious case of fail. However, *where the hell* does the bit come from in the first place? That is what we need to track down. When you say Xen HVM, am I correct in assuming that neither CPUID nor CR4 operations in the main kernel are run through paravirt_ops? -hpa