Re: Issue with the xen hypervisor crash.

[Keir Fraser <keir.fraser@eu.citrix.com>]

[-- Attachment #1: Type: text/plain, Size: 9812 bytes --]

I think it's a BIOS bug, but you can try the attached work-around patch.

 -- Keir

On 22/02/2010 17:41, "Vasudevan, Swaminathan (PNB Roseville)"
<swaminathan.vasudevan@hp.com> wrote:

> Hi Folks,
> I am having some issues with xen hypervisor crashing because it could not find
> one of the cores in my processor. This is not consistent. It happens once in
> every two or three boots.
> Is this is xen bug or is it something related to BIOS. If it is a xen bug, has
> it been addressed and could you point me to the right patch.
> 
> Here is the xen boot message.
> 
>                  
> The highlighted entry will be booted automatically in 5 seconds.
> Booting 'CentOS (2.6.18-164.11.1.el5xen)'
> 
> root (hd0,0)
>  Filesystem type is ext2fs, partition type 0x83
> kernel /boot/xen.gz-3.4.2 dom0_mem=max:512M com1=9600,8n1 console=vga,com1
>    [Multiboot-elf, <0x100000:0x121770:0x71890>, shtab=0x293078,
> entry=0x100000]
> module /boot/vmlinuz-2.6.18-164.11.1.el5xen ro root=LABEL=dom0a rhgb quiet
> audi
> t=1 xencons=xvc console=xvc
>    [Multiboot-module @ 0x294000, 0x96da30 bytes]
> module /boot/initrd-2.6.18-164.11.1.el5xen.img
>    [Multiboot-module @ 0xc02000, 0x80a400 bytes]
> 
>  __  __            _____ _  _    ____
>  \ \/ /___ _ __   |___ /| || |  |___ \
>   \  // _ \ '_ \    |_ \| || |_   __) |
>   /  \  __/ | | |  ___) |__   _| / __/
>  /_/\_\___|_| |_| |____(_) |_|(_)_____|
>                  
> (XEN) Xen version 3.4.2 (root@hq.avaya.com) (gcc version 4.1.2 20080704 (Red
> Hat
>  4.1.2-46)) Wed Jan 13 22:58:58 MST 2010
> (XEN) Latest ChangeSet: unavailable
> (XEN) Command line: dom0_mem=max:512M com1=9600,8n1 console=vga,com1
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
> (XEN)  EDID info not retrieved because no DDC retrieval method detected
> (XEN) Disc information:
> (XEN)  Found 0 MBR signatures
> (XEN)  Found 1 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)  0000000000000000 - 000000000009ac00 (usable)
> (XEN)  000000000009ac00 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 00000000bbdc2000 (usable)
> (XEN)  00000000bbdc2000 - 00000000bbdec000 (ACPI NVS)
> (XEN)  00000000bbdec000 - 00000000bbdef000 (reserved)
> (XEN)  00000000bbdef000 - 00000000bbdf2000 (ACPI NVS)
> (XEN)  00000000bbdf2000 - 00000000bbe03000 (reserved)
> (XEN)  00000000bbe03000 - 00000000bbe04000 (ACPI data)
> (XEN)  00000000bbe04000 - 00000000bbe32000 (reserved)
> (XEN)  00000000bbe32000 - 00000000bbe33000 (usable)
> (XEN)  00000000bbe33000 - 00000000bbe57000 (reserved)
> (XEN)  00000000bbe57000 - 00000000bbe58000 (ACPI data)
> (XEN)  00000000bbe58000 - 00000000bbe59000 (reserved)
> (XEN)  00000000bbe59000 - 00000000bbe65000 (ACPI data)
> (XEN)  00000000bbe65000 - 00000000bbe6c000 (ACPI NVS)
> (XEN)  00000000bbe6c000 - 00000000bbe6d000 (reserved)
> (XEN)  00000000bbe6d000 - 00000000bbe6f000 (ACPI NVS)
> (XEN)  00000000bbe6f000 - 00000000bbe91000 (reserved)
> (XEN)  00000000bbe91000 - 00000000bbe97000 (ACPI NVS)
> (XEN)  00000000bbe97000 - 00000000bc000000 (usable)
> (XEN)  00000000e0000000 - 00000000f0000000 (reserved)
> (XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
> (XEN)  00000000ffa80000 - 00000000ffc00000 (reserved)
> (XEN)  00000000ffe80000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 0000000240000000 (usable)
> (XEN) System RAM: 8126MB (8321816kB)
> (XEN) ACPI: RSDP 000F0400, 0024 (r2 HP    )
> (XEN) ACPI: XSDT BBE63E18, 006C (r1 HP     ProCurve  6222004 MSFT    10013)
> (XEN) ACPI: FACP BBE62D98, 00F4 (r4 HP     ProCurve  6222004 MSFT    10013)
> (XEN) ACPI: DSDT BBE59018, 8CA7 (r1 HP     ProCurve        0 INTL 20051117)
> (XEN) ACPI: FACS BBE6ED40, 0040
> (XEN) ACPI: APIC BBE62F18, 006C (r2 HP     ProCurve  6222004 MSFT    10013)
> (XEN) ACPI: MCFG BBE64F18, 003C (r1 A M I  GMCH945.  6222004 MSFT       97)
> (XEN) ACPI: HPET BBE64E98, 0038 (r1 A M I  ICH7HPET  6222004 AMI.        3)
> (XEN) ACPI: SSDT BBE57918, 0446 (r1 TrmRef PtidDevc     1000 INTL 20051117)
> (XEN) ACPI: SSDT BBE03698, 0655 (r1  PmRef    CpuPm     3000 INTL 20051117)
> (XEN) ACPI: DMAR BBE62C18, 00E0 (r1                       1             0)
> (XEN) ACPI: SPCR BBE64E18, 0050 (r1 A M I    APTIO  6222004 AMI.        3)
> (XEN) ACPI: SSDT BBE03018, 0636 (r1 SataRe SataTabl     1000 INTL 20051117)
> (XEN) Domain heap initialised
> (XEN) Processor #0 7:7 APIC version 20
> (XEN) Processor #1 7:7 APIC version 20
> (XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) Detected 2533.401 MHz processor.
> (XEN) VMX: Supported advanced features:
> (XEN)  - APIC MMIO access virtualisation
> (XEN)  - APIC TPR shadow
> (XEN)  - Virtual NMI
> (XEN)  - MSR direct-access bitmap
> (XEN) HVM: VMX enabled
> (XEN) spurious 8259A interrupt: IRQ7.
> (XEN) CPU0: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
> (XEN) Booting processor 1/1 eip 8c000
> (XEN) Not responding.
> (XEN) Inquiring remote APIC #1...
> (XEN) ... APIC #1 ID: failed
> (XEN) ... APIC #1 VERSION: failed
> (XEN) ... APIC #1 SPIV: failed
> (XEN) CPU #1 not responding - cannot use it.
> (XEN) Total of 1 processors activated.
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using new ACK method
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Brought up 1 CPUs
> (XEN) I/O virtualisation disabled
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN)  Xen  kernel: 64-bit, lsb, compat32
> (XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 ->
> 0xffffffff8075650c
> (XEN) PHYSICAL MEMORY ARRANGEMENT:
> (XEN)  Dom0 alloc.:   0000000236000000->0000000238000000 (122880 pages to be
> all
> ocated)
> (XEN) VIRTUAL MEMORY ARRANGEMENT:
> (XEN)  Loaded kernel: ffffffff80200000->ffffffff8075650c
> (XEN)  Init. ramdisk: ffffffff80757000->ffffffff80f61400
> (XEN)  Phys-Mach map: ffffffff80f62000->ffffffff81062000
> (XEN)  Start info:    ffffffff81062000->ffffffff810624b4
> (XEN)  Page tables:   ffffffff81063000->ffffffff81070000
> (XEN)  Boot stack:    ffffffff81070000->ffffffff81071000
> (XEN)  TOTAL:         ffffffff80000000->ffffffff81400000
> (XEN)  ENTRY ADDRESS: ffffffff80200000
> (XEN) Dom0 has maximum 1 VCPUs
> (XEN) Scrubbing Free RAM:
> ......................................................
> ......................done.
> (XEN) Xen trace buffers: disabled
> (XEN) Std. Loglevel: Errors and warnings
> (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
> (XEN) Xen is relinquishing VGA console.
> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
> Xen
> )
> (XEN) Freed 128kB init memory.
> kernel direct mapping tables up to 20800000 @ 1070000-127c000
> PCI: Not using MMCONFIG.
> (XEN) ioapic_guest_write: apic=0, pin=4, old_irq=4, new_irq=4
> (XEN) ioapic_guest_write: old_entry=000009f1, new_entry=000109f1
> (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
> (XEN) BUG: CPU1 started up but did not get a callout!
> (XEN) Xen BUG at smpboot.c:362
> (XEN) ----[ Xen-3.4.2  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    1
> (XEN) RIP:    e008:[<ffff828c80155837>] smp_callin+0x1b7/0x210
> (XEN) RFLAGS: 0000000000010086   CONTEXT: hypervisor
> (XEN) rax: 0000000000000000   rbx: ffffffffffffffff   rcx: 0000000000000046
> (XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff828c801e9fe4
> (XEN) rbp: 0000000000000001   rsp: ffff83023fdd7e70   r8:  0000000000000001
> (XEN) r9:  0000000000000001   r10: 00000000fffffffc   r11: ffff828c801256a0
> (XEN) r12: 00000000000000c8   r13: 0000000000000000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 00000000bba54000   cr2: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen stack trace from rsp=ffff83023fdd7e70:
> (XEN)    ffffffffffff8000 ffff83023fdd7f28 0000000000000001 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 ffff828c80155c4c
> (XEN)    ffff828c8011a2e3 000000353e00ea34 0000000100000033 0000000000000246
> (XEN)    00007fff26267d78 000000000000e02b 00000000002217b4 ffff83023fdd7f28
> (XEN)    ffff8300bb8f8000 0000000000000001 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    000000000f78bea0 000000000f78c0c0 00000000ffffffff 000000000f78bee0
> (XEN)    0000000000000000 0000000000000202 0000000000000008 00007fff26267ad0
> (XEN)    000000000f78ccd0 000000000f78b220 ffffffff802060aa ffff880014c83f58
> (XEN)    00007fff26267d30 0000000000000001 0000010000000000 ffffffff802060aa
> (XEN)    000000000000e033 0000000000000246 ffff880014c83e18 000000000000e02b
> (XEN)    308c0184024a2181 23a420ac21240204 03480240068c2000 0248104812840108
> (XEN)    13cc500c00000001 ffff8300bbff8000
> (XEN) Xen call trace:
> (XEN)    [<ffff828c80155837>] smp_callin+0x1b7/0x210
> (XEN)    [<ffff828c80155c4c>] start_secondary+0xac/0x430
> (XEN)    [<ffff828c8011a2e3>] timer_softirq_action+0x193/0x2e0
> (XEN)    
> (XEN) 
> (XEN) ****************************************
> (XEN) Panic on CPU 1:
> (XEN) Xen BUG at smpboot.c:362
> (XEN) ****************************************
> (XEN) 
> (XEN) Reboot in five seconds...
> 
> 
> 
> 
> 
> Thanks.
> 
> Swaminathan Vasudevan
> Systems Software Engineer (TC)
> 
> 
> HP ProCurve 
> Hewlett-Packard
> 8000 Foothills Blvd
> M/S 5541
> Roseville, CA - 95747
> tel: 916.785.0937
> fax: 916.785.1815
> email: swaminathan.vasudevan@hp.com
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


[-- Attachment #2: 00-smpboot --]
[-- Type: application/octet-stream, Size: 614 bytes --]

diff -r 4ba4323889b9 xen/arch/x86/smpboot.c
--- a/xen/arch/x86/smpboot.c	Mon Feb 22 18:47:22 2010 +0000
+++ b/xen/arch/x86/smpboot.c	Tue Feb 23 09:14:06 2010 +0000
@@ -948,14 +948,8 @@
 		cpu_set(cpu, cpu_callout_map);
 		Dprintk("After Callout %d.\n", cpu);
 
-		/*
-		 * Wait 5s total for a response
-		 */
-		for (timeout = 0; timeout < 50000; timeout++) {
-			if (cpu_isset(cpu, cpu_callin_map))
-				break;	/* It has booted */
-			udelay(100);
-		}
+		while (!cpu_isset(cpu, cpu_callin_map))
+			continue;
 
 		if (cpu_isset(cpu, cpu_callin_map)) {
 			/* number CPUs logically, starting from 1 (BSP is 0) */

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel