Xen unstable fails to boot from cold on i386 core duo

Xen unstable fails to boot from cold on i386 core duo

[Daniel P. Berrange]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

Just testing xen-unstable on some of my machines here. I've got a failure
to boot on a Mac Mini with a i386 Core Duo cpu.

If I try to boot xen-unstable changeset 16606 from cold it will immediately
crash with this:

  http://people.redhat.com/berrange/xen-3.2.0-fail.jpg

Sorry for lame camera pic - this being a mac mini there's no serial card,
or even ability to add one in :-(

If I boot into Xen 3.1.0 it works fine. If I warm-reboot from Xen 3.1.0
into Xen-unstable changeset 16606 it also works fine. So it seems only the
boot from cold is problematic.

Attaching the 'xm dmesg' output from a succesfull warm-reboot into 
xen-unstable, and the CPU info. 

Any ideas / suggestions ?

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

[-- Attachment #2: hv.log --]
[-- Type: text/plain, Size: 3685 bytes --]

 __  __            _____  ____    ___              ____                    
 \ \/ /___ _ __   |___ / |___ \  / _ \    _ __ ___|___ \    _ __  _ __ ___ 
  \  // _ \ '_ \    |_ \   __) || | | |__| '__/ __| __) |__| '_ \| '__/ _ \
  /  \  __/ | | |  ___) | / __/ | |_| |__| | | (__ / __/|__| |_) | | |  __/
 /_/\_\___|_| |_| |____(_)_____(_)___/   |_|  \___|_____|  | .__/|_|  \___|
                                                           |_|             
(XEN) Xen version 3.2.0-rc2-pre (berrange@home.berrange.com) (gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)) Sat Dec 15 13:05:31 EST 2007
(XEN) Latest ChangeSet: unavailable
(XEN) Command line: 
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000ede00 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000003e0d5000 (usable)
(XEN)  000000003e0d5000 - 000000003e2d6000 (ACPI NVS)
(XEN)  000000003e2d6000 - 000000003eebf000 (ACPI data)
(XEN)  000000003eebf000 - 000000003eeef000 (ACPI NVS)
(XEN)  000000003eeef000 - 000000003ef00000 (ACPI data)
(XEN)  000000003ef00000 - 0000000040000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed14000 - 00000000fed1a000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ffe00000 - 0000000100000000 (reserved)
(XEN) System RAM: 992MB (1016272kB)
(XEN) Xen heap: 9MB (10080kB)
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) PAE enabled, limit: 16 GB
(XEN) Processor #0 6:14 APIC version 20
(XEN) Processor #1 6:14 APIC version 20
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 1833.347 MHz processor.
(XEN) HVM: VMX enabled
(XEN) CPU0: Intel Genuine Intel(R) CPU            1400  @ 1.83GHz stepping 08
(XEN) Booting processor 1/1 eip 8c000
(XEN) CPU1: Intel Genuine Intel(R) CPU            1400  @ 1.83GHz stepping 08
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer overflows in 14998 jiffies.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 2 CPUs
(XEN) AMD IOMMU: Disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 32-bit, PAE, lsb
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0xc1000000 -> 0xc1401000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000003a000000->000000003c000000 (225331 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c1000000->c1401000
(XEN)  Init. ramdisk: c1401000->c1c61800
(XEN)  Phys-Mach map: c1c62000->c1d460cc
(XEN)  Start info:    c1d47000->c1d47474
(XEN)  Page tables:   c1d48000->c1d5d000
(XEN)  Boot stack:    c1d5d000->c1d5e000
(XEN)  TOTAL:         c0000000->c2000000
(XEN)  ENTRY ADDRESS: c1000000
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Initrd len 0x860800, start at 0xc1401000
(XEN) Scrubbing Free RAM: done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 96kB init memory.

[-- Attachment #3: cpu.log --]
[-- Type: text/plain, Size: 971 bytes --]

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 14
model name	: Genuine Intel(R) CPU            1400  @ 1.83GHz
stepping	: 8
cpu MHz		: 1833.347
cache size	: 2048 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 6
wp		: yes
flags		: fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc pni monitor vmx est tm2 xtpr
bogomips	: 3668.66
clflush size	: 64

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 14
model name	: Genuine Intel(R) CPU            1400  @ 1.83GHz
stepping	: 8
cpu MHz		: 1833.347
cache size	: 2048 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 6
wp		: yes
flags		: fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc up pni monitor vmx est tm2 xtpr
bogomips	: 3668.66
clflush size	: 64


[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen unstable fails to boot from cold on i386 core duo

[Daniel P. Berrange]

On Mon, Dec 17, 2007 at 03:42:21PM +0000, Daniel P. Berrange wrote:
> Just testing xen-unstable on some of my machines here. I've got a failure
> to boot on a Mac Mini with a i386 Core Duo cpu.
> 
> If I try to boot xen-unstable changeset 16606 from cold it will immediately
> crash with this:
> 
>   http://people.redhat.com/berrange/xen-3.2.0-fail.jpg
> 
> Sorry for lame camera pic - this being a mac mini there's no serial card,
> or even ability to add one in :-(
> 
> If I boot into Xen 3.1.0 it works fine. If I warm-reboot from Xen 3.1.0
> into Xen-unstable changeset 16606 it also works fine. So it seems only the
> boot from cold is problematic.
> 
> Attaching the 'xm dmesg' output from a succesfull warm-reboot into 
> xen-unstable, and the CPU info. 
> 
> Any ideas / suggestions ?

Converting ff16420d / vmx_cpu_up+0x25d address into a line number with 
eu-addr2line, it appears as if Xen is crashing & burning in the 'wrmsr' 
call in the 2nd half of this conditional from arch/x86/hvm/vmx/vmcs.c:

    if ( eax & IA32_FEATURE_CONTROL_MSR_LOCK )
    {
        if ( !(eax & (IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX |
                      IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX)) )
        {
            printk("CPU%d: VMX disabled by BIOS.\n", cpu);
            return 0;
        }
    }
    else
    {
        eax = (IA32_FEATURE_CONTROL_MSR_LOCK |
               IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX |
               IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX);
        wrmsr(IA32_FEATURE_CONTROL_MSR, eax, 0);
    }


In Xen 3.1.0 (which works for me), the 'vmx_cpu_up' method does not exist,
but the 'start_vmx' function which does more of less the same thing
writes a slightly different msr:

    if ( eax & IA32_FEATURE_CONTROL_MSR_LOCK )
    {
        if ( (eax & IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON) == 0x0 )
        {
            printk("VMX disabled by Feature Control MSR.\n");
            return 0;
        }
    }
    else
    {
        wrmsr(IA32_FEATURE_CONTROL_MSR,
              IA32_FEATURE_CONTROL_MSR_LOCK |
              IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON, 0);
    }


I guess this explains why Xen-unstable works if I warm re-boot from xen-3.1.0,
because the 3.1.0 hypervisor will have already enabled and lock VMX msr.

Is it possible that older CPUs / BIOS don't play nice with the extra bit

   IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX

...if they don't have SMX support ?

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Xen unstable fails to boot from cold on i386 core duo

[Daniel P. Berrange]

On Tue, Dec 18, 2007 at 12:00:57AM +0000, Daniel P. Berrange wrote:
> Converting ff16420d / vmx_cpu_up+0x25d address into a line number with 
> eu-addr2line, it appears as if Xen is crashing & burning in the 'wrmsr' 
> call in the 2nd half of this conditional from arch/x86/hvm/vmx/vmcs.c:
> 
>     if ( eax & IA32_FEATURE_CONTROL_MSR_LOCK )
>     {
>         if ( !(eax & (IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX |
>                       IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX)) )
>         {
>             printk("CPU%d: VMX disabled by BIOS.\n", cpu);
>             return 0;
>         }
>     }
>     else
>     {
>         eax = (IA32_FEATURE_CONTROL_MSR_LOCK |
>                IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX |
>                IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX);
>         wrmsr(IA32_FEATURE_CONTROL_MSR, eax, 0);
>     }

[snip]

> Is it possible that older CPUs / BIOS don't play nice with the extra bit
> 
>    IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX
> 
> ...if they don't have SMX support ?

I have confirmed that if I modify Xen with this patch:

--- xen-unstable-16606.orig/xen/arch/x86/hvm/vmx/vmcs.c	2007-12-13 04:31:03.000000000 -0500
+++ xen-unstable-16606/xen/arch/x86/hvm/vmx/vmcs.c	2007-12-17 20:21:43.000000000 -0500
@@ -265,8 +265,7 @@ int vmx_cpu_up(void)
 
     if ( eax & IA32_FEATURE_CONTROL_MSR_LOCK )
     {
-        if ( !(eax & (IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX |
-                      IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX)) )
+        if ( !(eax & (IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX)))
         {
             printk("CPU%d: VMX disabled by BIOS.\n", cpu);
             return 0;
@@ -275,8 +274,7 @@ int vmx_cpu_up(void)
     else
     {
         eax = (IA32_FEATURE_CONTROL_MSR_LOCK |
-               IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX |
-               IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_INSIDE_SMX);
+               IA32_FEATURE_CONTROL_MSR_ENABLE_VMXON_OUTSIDE_SMX);
         wrmsr(IA32_FEATURE_CONTROL_MSR, eax, 0);
     }
 


Everything works nicely again.   Obviously this patch will break stuff
related to SMX so a better solution is required by somebody with the
knowledge in this area....

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Xen unstable fails to boot from cold on i386 core duo

[Keir Fraser]

On 18/12/07 02:03, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> Everything works nicely again.   Obviously this patch will break stuff
> related to SMX so a better solution is required by somebody with the
> knowledge in this area....

Try xen-unstable c/s 16643:35ab2bb25e09.

 -- Keir