dom0 crash with unstable

All of lore.kernel.org
 help / color / mirror / Atom feed

* dom0 crash with unstable
@ 2009-09-30 18:05 Bryan D. Payne
  2009-09-30 23:47 ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 14+ messages in thread
From: Bryan D. Payne @ 2009-09-30 18:05 UTC (permalink / raw)
  To: xen-devel

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

I'm trying to install Xen-unstable on a new machine.  At first boot,
I'm getting a dom0 crash.  The crash output is shown below.  The
complete console log is attached.

(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xe)
(XEN) Freed 148kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
(XEN) traps.c:466:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.5-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff819067de>]
(XEN) RFLAGS: 0000000000000212   EM: 1   CONTEXT: pv guest
(XEN) rax: ffffffff865ce000   rbx: ffffffff81001000   rcx: 0000000000000006
(XEN) rdx: 0000000000800000   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffffff81807fa8   rsp: ffffffff81807f68   r8:  000000000000001d
(XEN) r9:  ffffffff81807fd8   r10: 0000000006606000   r11: 00000000818ec000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000919001000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81807f68:
(XEN)    0000000000000006 00000000818ec000 ffffffff819067de 000000010000e030
(XEN)    0000000000010012 ffffffff81807fa8 000000000000e02b ffffffff81906d64
(XEN)    ffffffff81807ff8 ffffffff81906166 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

I pulled from Xen unstable yesterday (bd376919f03a tip), and built it
simply using "make world".  The machine has four Intel E5540
processors and 36GB of RAM.  In the BIOS, I have VT-d disabled, and
VT-x enabled.  Any suggestions on what steps I should take for
debugging this problem and getting dom0 to boot?

Thanks,
bryan

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 18364 bytes --]

root  (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd
kernel  /xen.gz console=com1 com1=115200,8n1
   [Multiboot-elf, <0x100000:0x158c60:0xbd3a0>, shtab=0x316078, entry=0x100000]
module  /vmlinuz-2.6.31.1 root=/dev/md2 ro console=hvc0 earlyprintk=xen
   [Multiboot-module @ 0x317000, 0x427d80 bytes]
module  /initrd.img-2.6.31.1
   [Multiboot-module @ 0x73f000, 0x436000 bytes]
savedefault

 __  __            _____  ____                     _        _     _      
 \ \/ /___ _ __   |___ / | ___|    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \ |___ \ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | ___) |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)____/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

(XEN) Xen version 3.5-unstable (root@XXXXX) (gcc version 4.3.2 (Debi9
(XEN) Latest ChangeSet: Tue Sep 29 11:28:33 2009 +0100 20261:bd376919f03a
(XEN) Command line: console=com1 com1=115200,8n1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009ac00 (usable)
(XEN)  000000000009ac00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000bf780000 (usable)
(XEN)  00000000bf78e000 - 00000000bf790000 type 9
(XEN)  00000000bf790000 - 00000000bf79e000 (ACPI data)
(XEN)  00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
(XEN)  00000000bf7d0000 - 00000000bf7e0000 (reserved)
(XEN)  00000000bf7ec000 - 00000000c0000000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000940000000 (usable)
(XEN) ACPI: RSDP 000FA450, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT BF790100, 008C (r1 NEC             20090409 MSFT       97)
(XEN) ACPI: FACP BF790290, 00F4 (r4 040909 FACP1634 20090409 MSFT       97)
(XEN) ACPI: DSDT BF790670, 58D7 (r2  10600 10600000        0 INTL 20051117)
(XEN) ACPI: FACS BF79E000, 0040
(XEN) ACPI: APIC BF790390, 00D2 (r2 040909 APIC1634 20090409 MSFT       97)
(XEN) ACPI: MCFG BF790470, 003C (r1 040909 OEMMCFG  20090409 MSFT       97)
(XEN) ACPI: SLIC BF7904B0, 0176 (r1 NEC             20090409 MSFT       97)
(XEN) ACPI: ERST BF790630, 0040 (r1 040909 WHEA1634 20090409 MSFT       97)
(XEN) ACPI: OEMB BF79E040, 007B (r1 040909 OEMB1634 20090409 MSFT       97)
(XEN) ACPI: SRAT BF79A670, 01D0 (r1 040909 OEMSRAT         1 INTL        1)
(XEN) ACPI: HPET BF79A840, 0038 (r1 040909 OEMHPET  20090409 MSFT       97)
(XEN) ACPI: SSDT BF7A1E20, 0363 (r1 DpgPmm    CpuPm       12 INTL 20051117)
(XEN) ACPI: EINJ BF79A880, 0130 (r1  AMIER AMI_EINJ 20090409 MSFT       97)
(XEN) ACPI: BERT BF79AA10, 0030 (r1  AMIER AMI_BERT 20090409 MSFT       97)
(XEN) ACPI: ERST BF79AA40, 01B0 (r1  AMIER AMI_ERST 20090409 MSFT       97)
(XEN) ACPI: HEST BF79ABF0, 00A8 (r1  AMIER AMI_HEST 20090409 MSFT       97)
(XEN) SRAT: 0000000000000-000000009ffff
(XEN) SRAT: 0000000100000-00000bfffffff
(XEN) SRAT: 0000100000000-000033fffffff
(XEN) SRAT: 0000340000000-000093fffffff
(XEN) System RAM: 36828MB (37712412kB)
(XEN) NUMA turned off
(XEN) Faking a node at 0000000000000000-0000000940000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]
(XEN) ACPI:                  wakeup_vec[bf79e00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x10] enabled)
(XEN) Processor #16 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
(XEN) Processor #2 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
(XEN) Processor #18 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
(XEN) Processor #4 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled)
(XEN) Processor #20 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] enabled)
(XEN) Processor #6 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x16] enabled)
(XEN) Processor #22 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x01] enabled)
(XEN) Processor #1 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x11] enabled)
(XEN) Processor #17 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x03] enabled)
(XEN) Processor #3 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x13] enabled)
(XEN) Processor #19 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x05] enabled)
(XEN) Processor #5 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x15] enabled)
(XEN) Processor #21 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x07] enabled)
(XEN) Processor #7 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x17] enabled)
(XEN) Processor #23 7:10 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
(XEN) Overriding APIC driver with bigsmp
(XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Phys.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
(XEN) PCI: Not using MMCONFIG.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2533.469 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Thermal monitoring enabled (TM1)
(XEN) I/O virtualisation disabled
(XEN) CPU0: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 1/16 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Thermal monitoring enabled (TM1)
(XEN) CPU1: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 2/2 eip 8c000
(XEN) Initializing CPU#2
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check reporting enabled on CPU#2.
(XEN) CPU2: Thermal monitoring enabled (TM1)
(XEN) CPU2: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 3/18 eip 8c000
(XEN) Initializing CPU#3
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check reporting enabled on CPU#3.
(XEN) CPU3: Thermal monitoring enabled (TM1)
(XEN) CPU3: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 4/4 eip 8c000
(XEN) Initializing CPU#4
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 2
(XEN) Intel machine check reporting enabled on CPU#4.
(XEN) CPU4: Thermal monitoring enabled (TM1)
(XEN) CPU4: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 5/20 eip 8c000
(XEN) Initializing CPU#5
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 2
(XEN) Intel machine check reporting enabled on CPU#5.
(XEN) CPU5: Thermal monitoring enabled (TM1)
(XEN) CPU5: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 6/6 eip 8c000
(XEN) Initializing CPU#6
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 3
(XEN) Intel machine check reporting enabled on CPU#6.
(XEN) CPU6: Thermal monitoring enabled (TM1)
(XEN) CPU6: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 7/22 eip 8c000
(XEN) Initializing CPU#7
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 3
(XEN) Intel machine check reporting enabled on CPU#7.
(XEN) CPU7: Thermal monitoring enabled (TM1)
(XEN) CPU7: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 8/1 eip 8c000
(XEN) Initializing CPU#8
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check reporting enabled on CPU#8.
(XEN) CPU8: Thermal monitoring enabled (TM1)
(XEN) CPU8: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 9/17 eip 8c000
(XEN) Initializing CPU#9
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check reporting enabled on CPU#9.
(XEN) CPU9: Thermal monitoring enabled (TM1)
(XEN) CPU9: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 10/3 eip 8c000
(XEN) Initializing CPU#10
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check reporting enabled on CPU#10.
(XEN) CPU10: Thermal monitoring enabled (TM1)
(XEN) CPU10: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 11/19 eip 8c000
(XEN) Initializing CPU#11
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check reporting enabled on CPU#11.
(XEN) CPU11: Thermal monitoring enabled (TM1)
(XEN) CPU11: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 12/5 eip 8c000
(XEN) Initializing CPU#12
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 2
(XEN) Intel machine check reporting enabled on CPU#12.
(XEN) CPU12: Thermal monitoring enabled (TM1)
(XEN) CPU12: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 13/21 eip 8c000
(XEN) Initializing CPU#13
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 2
(XEN) Intel machine check reporting enabled on CPU#13.
(XEN) CPU13: Thermal monitoring enabled (TM1)
(XEN) CPU13: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 14/7 eip 8c000
(XEN) Initializing CPU#14
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 3
(XEN) Intel machine check reporting enabled on CPU#14.
(XEN) CPU14: Thermal monitoring enabled (TM1)
(XEN) CPU14: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Booting processor 15/23 eip 8c000
(XEN) Initializing CPU#15
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 3
(XEN) Intel machine check reporting enabled on CPU#15.
(XEN) CPU15: Thermal monitoring enabled (TM1)
(XEN) CPU15: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz stepping 05
(XEN) Total of 16 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 16 CPUs: passed.
(XEN) Platform timer is 14.318MHz HPET
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) microcode.c:73:d32767 microcode: CPU2 resumed
(XEN) microcode.c:73:d32767 microcode: CPU3 resumed
(XEN) microcode.c:73:d32767 microcode: CPU4 resumed
(XEN) microcode.c:73:d32767 microcode: CPU5 resumed
(XEN) microcode.c:73:d32767 microcode: CPU6 resumed
(XEN) microcode.c:73:d32767 microcode: CPU7 resumed
(XEN) microcode.c:73:d32767 microcode: CPU8 resumed
(XEN) microcode.c:73:d32767 microcode: CPU9 resumed
(XEN) microcode.c:73:d32767 microcode: CPU10 resumed
(XEN) microcode.c:73:d32767 microcode: CPU11 resumed
(XEN) microcode.c:73:d32767 microcode: CPU12 resumed
(XEN) microcode.c:73:d32767 microcode: CPU13 resumed
(XEN) microcode.c:73:d32767 microcode: CPU14 resumed
(XEN) Brought up 16 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU15 resumed
(XEN) HPET: 4 timers in total, 0 timers will be used for broadcast
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x806000
(XEN) elf_parse_binary: phdr: paddr=0x1806000 memsz=0xe4778
(XEN) elf_parse_binary: phdr: paddr=0x18eb000 memsz=0x888
(XEN) elf_parse_binary: phdr: paddr=0x18ec000 memsz=0x15860
(XEN) elf_parse_binary: phdr: paddr=0x1901860 memsz=0x1a37a0
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1aa5000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff81901a60
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81009000
(XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0xffffffff80000000
(XEN)     virt_kstart      = 0xffffffff81000000
(XEN)     virt_kend        = 0xffffffff81aa5000
(XEN)     virt_entry       = 0xffffffff81901a60
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1aa5000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000918000000->0000000920000000 (9266322 pages to be a)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81aa5000
(XEN)  Init. ramdisk: ffffffff81aa5000->ffffffff81edb000
(XEN)  Phys-Mach map: ffffffff81edb000->ffffffff865cd490
(XEN)  Start info:    ffffffff865ce000->ffffffff865ce4b4
(XEN)  Page tables:   ffffffff865cf000->ffffffff86606000
(XEN)  Boot stack:    ffffffff86606000->ffffffff86607000
(XEN)  TOTAL:         ffffffff80000000->ffffffff86800000
(XEN)  ENTRY ADDRESS: ffffffff81901a60
(XEN) Dom0 has maximum 16 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff81806000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81806000 -> 0xffffffff818ea778
(XEN) elf_load_binary: phdr 2 at 0xffffffff818eb000 -> 0xffffffff818eb888
(XEN) elf_load_binary: phdr 3 at 0xffffffff818ec000 -> 0xffffffff81901860
(XEN) elf_load_binary: phdr 4 at 0xffffffff81901860 -> 0xffffffff8197b000
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xe)
(XEN) Freed 148kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
(XEN) traps.c:466:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.5-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff819067de>]
(XEN) RFLAGS: 0000000000000212   EM: 1   CONTEXT: pv guest
(XEN) rax: ffffffff865ce000   rbx: ffffffff81001000   rcx: 0000000000000006
(XEN) rdx: 0000000000800000   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffffff81807fa8   rsp: ffffffff81807f68   r8:  000000000000001d
(XEN) r9:  ffffffff81807fd8   r10: 0000000006606000   r11: 00000000818ec000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000919001000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81807f68:
(XEN)    0000000000000006 00000000818ec000 ffffffff819067de 000000010000e030
(XEN)    0000000000010012 ffffffff81807fa8 000000000000e02b ffffffff81906d64
(XEN)    ffffffff81807ff8 ffffffff81906166 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-09-30 18:05 dom0 crash with unstable Bryan D. Payne
@ 2009-09-30 23:47 ` Jeremy Fitzhardinge
  2009-10-01 13:35   ` Bryan D. Payne
  0 siblings, 1 reply; 14+ messages in thread
From: Jeremy Fitzhardinge @ 2009-09-30 23:47 UTC (permalink / raw)
  To: Bryan D. Payne; +Cc: xen-devel

On 09/30/09 11:05, Bryan D. Payne wrote:
> I'm trying to install Xen-unstable on a new machine.  At first boot,
> I'm getting a dom0 crash.  The crash output is shown below.  The
> complete console log is attached.
>
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xe)
> (XEN) Freed 148kB init memory.
> mapping kernel into physical memory
> Xen: setup ISA identity maps
> (XEN) traps.c:466:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-3.5-unstable  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e033:[<ffffffff819067de>]
> (XEN) RFLAGS: 0000000000000212   EM: 1   CONTEXT: pv guest
> (XEN) rax: ffffffff865ce000   rbx: ffffffff81001000   rcx: 0000000000000006
> (XEN) rdx: 0000000000800000   rsi: 00000000deadbeef   rdi: 00000000deadbeef
> (XEN) rbp: ffffffff81807fa8   rsp: ffffffff81807f68   r8:  000000000000001d
> (XEN) r9:  ffffffff81807fd8   r10: 0000000006606000   r11: 00000000818ec000
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 0000000919001000   cr2: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
> (XEN) Guest stack trace from rsp=ffffffff81807f68:
> (XEN)    0000000000000006 00000000818ec000 ffffffff819067de 000000010000e030
> (XEN)    0000000000010012 ffffffff81807fa8 000000000000e02b ffffffff81906d64
> (XEN)    ffffffff81807ff8 ffffffff81906166 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000001 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000
> (XEN) Domain 0 crashed: rebooting machine in 5 seconds.
>
> I pulled from Xen unstable yesterday (bd376919f03a tip), and built it
> simply using "make world".  The machine has four Intel E5540
> processors and 36GB of RAM.  In the BIOS, I have VT-d disabled, and
> VT-x enabled.  Any suggestions on what steps I should take for
> debugging this problem and getting dom0 to boot?
>   

It looks like something has hit a BUG_ON.  The first step is to try to
identify which one:

$ gdb vmlinux
(gdb) x/i 0xffffffff819067de
(gdb) x/i 0xffffffff81906d64
(gdb) x/i 0xffffffff81906166

should give a first clue.

Thanks,
    J

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-09-30 23:47 ` Jeremy Fitzhardinge
@ 2009-10-01 13:35   ` Bryan D. Payne
  2009-10-01 21:01     ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 14+ messages in thread
From: Bryan D. Payne @ 2009-10-01 13:35 UTC (permalink / raw)
  To: Jeremy Fitzhardinge; +Cc: xen-devel

> $ gdb vmlinux
> (gdb) x/i 0xffffffff819067de
> (gdb) x/i 0xffffffff81906d64
> (gdb) x/i 0xffffffff81906166

(gdb) x/i 0xffffffff819067de
0xffffffff819067de <xen_fix_mfn_list+38>:	ud2a
(gdb) x/i 0xffffffff81906d64
0xffffffff81906d64 <xen_ident_map_ISA+128>:	leaveq
(gdb) x/i 0xffffffff81906166
0xffffffff81906166 <xen_start_kernel+1343>:	
    mov    0xa028b(%rip),%rax        # 0xffffffff819a63f8 <xen_start_info>

Thanks,
-bryan

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-01 13:35   ` Bryan D. Payne
@ 2009-10-01 21:01     ` Jeremy Fitzhardinge
  2009-10-06 15:13       ` Bryan D. Payne
  0 siblings, 1 reply; 14+ messages in thread
From: Jeremy Fitzhardinge @ 2009-10-01 21:01 UTC (permalink / raw)
  To: Bryan D. Payne; +Cc: xen-devel

On 10/01/09 06:35, Bryan D. Payne wrote:
> (gdb) x/i 0xffffffff819067de
> 0xffffffff819067de <xen_fix_mfn_list+38>:	ud2a
> (gdb) x/i 0xffffffff81906d64
> 0xffffffff81906d64 <xen_ident_map_ISA+128>:	leaveq
> (gdb) x/i 0xffffffff81906166
> 0xffffffff81906166 <xen_start_kernel+1343>:	
>     mov    0xa028b(%rip),%rax        # 0xffffffff819a63f8 <xen_start_info>
>   

It looks like its encountering a pfn (page number) that's greater than
the total number of pages given to the domain. 

Aaah, 36GB memory.  What happens if you configure
CONFIG_XEN_MAX_DOMAIN_MEMORY larger to match, or set dom0_mem to less
than 32GB?

(It shouldn't crash regardless, but it will confirm the diagnosis.)

    J

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-01 21:01     ` Jeremy Fitzhardinge
@ 2009-10-06 15:13       ` Bryan D. Payne
  2009-10-06 16:31         ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 14+ messages in thread
From: Bryan D. Payne @ 2009-10-06 15:13 UTC (permalink / raw)
  To: Jeremy Fitzhardinge; +Cc: xen-devel

> It looks like its encountering a pfn (page number) that's greater than
> the total number of pages given to the domain.
>
> Aaah, 36GB memory.  What happens if you configure
> CONFIG_XEN_MAX_DOMAIN_MEMORY larger to match, or set dom0_mem to less
> than 32GB?
>
> (It shouldn't crash regardless, but it will confirm the diagnosis.)

Ok, so I tried setting dom0_mem to a variety of values less than 32GB.
 I'm still getting a crash, but now it is more random.  Basically, I'm
watching the boot process via a serial line, and instead of seeing the
dom0 crash output that I posted before, I'm simply seeing the output
stop, and the boot process hanging, in a different spot with each
boot.  Removing the dom0_mem value beings me back to the behavior I
had before, where the dom0 crash output shows up reliably each time.

-bryan

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 15:13       ` Bryan D. Payne
@ 2009-10-06 16:31         ` Jeremy Fitzhardinge
  2009-10-06 17:23           ` Bryan D. Payne
  0 siblings, 1 reply; 14+ messages in thread
From: Jeremy Fitzhardinge @ 2009-10-06 16:31 UTC (permalink / raw)
  To: Bryan D. Payne; +Cc: xen-devel

On 10/06/09 08:13, Bryan D. Payne wrote:
> Ok, so I tried setting dom0_mem to a variety of values less than 32GB.
>  I'm still getting a crash, but now it is more random.  Basically, I'm
> watching the boot process via a serial line, and instead of seeing the
> dom0 crash output that I posted before, I'm simply seeing the output
> stop, and the boot process hanging, in a different spot with each
> boot.  Removing the dom0_mem value beings me back to the behavior I
> had before, where the dom0 crash output shows up reliably each time.
>   

That's mysterious.  My first thought is that this is a separate problem.

Is it ever stable?  What happens if you set dom0_mem to 4G or less?

Is the console responsive when the kernel hangs?  That is, can you type
"Ctrl-A Ctrl-A Ctrl-A" to get Xen, then enter debug keys?  '0' (zero)
should dump the context for dom0 and give some clue about where it is dying.

Thanks,
    J

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 16:31         ` Jeremy Fitzhardinge
@ 2009-10-06 17:23           ` Bryan D. Payne
  2009-10-06 17:54             ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 14+ messages in thread
From: Bryan D. Payne @ 2009-10-06 17:23 UTC (permalink / raw)
  To: Jeremy Fitzhardinge; +Cc: xen-devel

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

> That's mysterious.  My first thought is that this is a separate problem.
I agree... and thanks for your help in diagnosing this issue.

> Is it ever stable?  What happens if you set dom0_mem to 4G or less?
It still crashes, even with 2G of dom0 memory.

> Is the console responsive when the kernel hangs?  That is, can you type
> "Ctrl-A Ctrl-A Ctrl-A" to get Xen, then enter debug keys?  '0' (zero)
> should dump the context for dom0 and give some clue about where it is dying.
Yes, this seems to work.  Typing 0 dumped info for each of the cores
(i.e., 15 vcpu states).  I've attached the output.

-bryan

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 30712 bytes --]

(XEN) '0' pressed -> dumping Dom0's registers
(XEN) *** Dumping Dom0 vcpu#0 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc90000003e58   rsp: ffffc90000003e30   r8:  ffffffff81806000
(XEN) r9:  0000000000fffb6e   r10: ffffc90000003e98   r11: 0000000000000246
(XEN) r12: ffff8800bf005120   r13: ffffc900000cfb58   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000000000008   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc90000003e30:
(XEN)    ffff8800bf005120 0000000000000080 ffffffff8102f98c ffffc90000003e58
(XEN)    ffffc900000cfb30 ffffc90000003e88 ffffffff8108ff63 ffff8800bf022d80
(XEN)    ffff8800bf005120 000000000000001a 000000000000001a ffffc90000003e98
(XEN)    ffffffff8102fba5 ffffc90000003ed8 ffffffff810ad608 000000000000fa30
(XEN)    ffff8800bf022d80 ffff8800bf022de4 000000000000001a 0000000000000100
(XEN)    0000000000000000 ffffc90000003f08 ffffffff810aee38 ffffc90000003f08
(XEN)    000000000000001a 0000000000000000 0000000000012350 ffffc90000003f28
(XEN)    ffffffff8103667e ffffc90000003f28 0000000000000001 ffffc90000003fa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffffffff81807e48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc9000000b060 ffffc9000000b068 0000000000000001 0000000000000000
(XEN)    6db6db6db6db6db7 ffffffff8193efc0 0000000000000000 ffffffff81807e48
(XEN)    ffffffff81034ebe ffffffff81807e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#1 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000001   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc9000001be58   rsp: ffffc9000001be30   r8:  ffff8800bf086000
(XEN) r9:  0000000000000000   r10: ffffc9000001be98   r11: 0000000000000246
(XEN) r12: ffff8800bf005300   r13: ffffc900000cfb58   r14: 0000000000000001
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc9000001be30:
(XEN)    ffff8800bf005300 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc9000001be88 ffffffff8108ff63 ffff8800bf0e0180
(XEN)    ffff8800bf005300 000000000000001f 000000000000001f ffffc9000001be98
(XEN)    ffffffff8102fba5 ffffc9000001bed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf0e0180 ffff8800bf0e01e4 000000000000001f 0000000000000100
(XEN)    0000000000000200 ffffc9000001bf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    000000000000001f 0000000000000000 0000000000012350 ffffc9000001bf28
(XEN)    ffffffff8103667e ffffc9000001bf28 0000000000000001 ffffc9000001bfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf087e48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc90000023060 ffffc90000023068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf087e48
(XEN)    ffffffff81034ebe ffff8800bf087e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#2 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000002   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc90000033e58   rsp: ffffc90000033e30   r8:  ffff8800bf0a8000
(XEN) r9:  0000000000000000   r10: ffffc90000033e98   r11: 0000000000000246
(XEN) r12: ffff8800bf0054e0   r13: ffffc900000cfb58   r14: 0000000000000002
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc90000033e30:
(XEN)    ffff8800bf0054e0 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc90000033e88 ffffffff8108ff63 ffff8800bf0e0780
(XEN)    ffff8800bf0054e0 0000000000000024 0000000000000024 ffffc90000033e98
(XEN)    ffffffff8102fba5 ffffc90000033ed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf0e0780 ffff8800bf0e07e4 0000000000000024 0000000000000100
(XEN)    0000000000000400 ffffc90000033f08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000024 0000000000000000 0000000000012350 ffffc90000033f28
(XEN)    ffffffff8103667e ffffc90000033f28 0000000000000001 ffffc90000033fa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0a9e48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc9000003b060 ffffc9000003b068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0a9e48
(XEN)    ffffffff81034ebe ffff8800bf0a9e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#3 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000003   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc9000004be58   rsp: ffffc9000004be30   r8:  ffff8800bf0aa000
(XEN) r9:  0000000000000000   r10: ffffc9000004be98   r11: 0000000000000246
(XEN) r12: ffff8800bf0056c0   r13: ffffc900000cfb58   r14: 0000000000000003
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc9000004be30:
(XEN)    ffff8800bf0056c0 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc9000004be88 ffffffff8108ff63 ffff8800bf0e0d80
(XEN)    ffff8800bf0056c0 0000000000000029 0000000000000029 ffffc9000004be98
(XEN)    ffffffff8102fba5 ffffc9000004bed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf0e0d80 ffff8800bf0e0de4 0000000000000029 0000000000000100
(XEN)    0000000000000600 ffffc9000004bf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000029 0000000000000000 0000000000012350 ffffc9000004bf28
(XEN)    ffffffff8103667e ffffc9000004bf28 0000000000000001 ffffc9000004bfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0abe48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc90000053060 ffffc90000053068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0abe48
(XEN)    ffffffff81034ebe ffff8800bf0abe48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#4 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000004   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc90000063e58   rsp: ffffc90000063e30   r8:  ffff8800bf0ac000
(XEN) r9:  0000000000000000   r10: ffffc90000063e98   r11: 0000000000000246
(XEN) r12: ffff8800bf0058a0   r13: ffffc900000cfb58   r14: 0000000000000004
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc90000063e30:
(XEN)    ffff8800bf0058a0 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc90000063e88 ffffffff8108ff63 ffff8800bf0e1380
(XEN)    ffff8800bf0058a0 000000000000002e 000000000000002e ffffc90000063e98
(XEN)    ffffffff8102fba5 ffffc90000063ed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf0e1380 ffff8800bf0e13e4 000000000000002e 0000000000000100
(XEN)    0000000000000800 ffffc90000063f08 ffffffff810aee38 ffffffff8102f38f
(XEN)    000000000000002e 0000000000000000 0000000000012350 ffffc90000063f28
(XEN)    ffffffff8103667e ffffc90000063f28 0000000000000001 ffffc90000063fa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0ade48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc9000006b060 ffffc9000006b068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0ade48
(XEN)    ffffffff81034ebe ffff8800bf0ade48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#5 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000005   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc9000007be58   rsp: ffffc9000007be30   r8:  ffff8800bf0ae000
(XEN) r9:  0000000000000000   r10: ffffc9000007be98   r11: 0000000000000246
(XEN) r12: ffff8800bf005a80   r13: ffffc900000cfb58   r14: 0000000000000005
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc9000007be30:
(XEN)    ffff8800bf005a80 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc9000007be88 ffffffff8108ff63 ffff8800bf0e1980
(XEN)    ffff8800bf005a80 0000000000000033 0000000000000033 ffffc9000007be98
(XEN)    ffffffff8102fba5 ffffc9000007bed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf0e1980 ffff8800bf0e19e4 0000000000000033 0000000000000100
(XEN)    0000000000000a00 ffffc9000007bf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000033 0000000000000000 0000000000012350 ffffc9000007bf28
(XEN)    ffffffff8103667e ffffc9000007bf28 0000000000000001 ffffc9000007bfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0afe48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc90000083060 ffffc90000083068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0afe48
(XEN)    ffffffff81034ebe ffff8800bf0afe48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#6 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000006   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc90000093e58   rsp: ffffc90000093e30   r8:  ffff8800bf0b0000
(XEN) r9:  0000000000000000   r10: ffffc90000093e98   r11: 0000000000000246
(XEN) r12: ffff8800bf005c60   r13: ffffc900000cfb58   r14: 0000000000000006
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc90000093e30:
(XEN)    ffff8800bf005c60 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc90000093e88 ffffffff8108ff63 ffff8800bf12a000
(XEN)    ffff8800bf005c60 0000000000000038 0000000000000038 ffffc90000093e98
(XEN)    ffffffff8102fba5 ffffc90000093ed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf12a000 ffff8800bf12a064 0000000000000038 0000000000000100
(XEN)    0000000000000c00 ffffc90000093f08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000038 0000000000000000 0000000000012350 ffffc90000093f28
(XEN)    ffffffff8103667e ffffc90000093f28 0000000000000001 ffffc90000093fa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0b1e48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc9000009b060 ffffc9000009b068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0b1e48
(XEN)    ffffffff81034ebe ffff8800bf0b1e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#7 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000007   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc900000abe58   rsp: ffffc900000abe30   r8:  ffff8800bf0b2000
(XEN) r9:  0000000000000000   r10: ffffc900000abe98   r11: 0000000000000246
(XEN) r12: ffff8800bf005e40   r13: ffffc900000cfb58   r14: 0000000000000007
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc900000abe30:
(XEN)    ffff8800bf005e40 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc900000abe88 ffffffff8108ff63 ffff8800bf12a600
(XEN)    ffff8800bf005e40 000000000000003d 000000000000003d ffffc900000abe98
(XEN)    ffffffff8102fba5 ffffc900000abed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf12a600 ffff8800bf12a664 000000000000003d 0000000000000100
(XEN)    0000000000000e00 ffffc900000abf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    000000000000003d 0000000000000000 0000000000012350 ffffc900000abf28
(XEN)    ffffffff8103667e ffffc900000abf28 0000000000000001 ffffc900000abfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0b3e48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc900000b3060 ffffc900000b3068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0b3e48
(XEN)    ffffffff81034ebe ffff8800bf0b3e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#8 state: ***
(XEN) RIP:    e033:[<ffffffff811fea48>]
(XEN) RFLAGS: 0000000000000246   EM: 0   CONTEXT: pv guest
(XEN) rax: 00000000001427d8   rbx: 0000000000000008   rcx: 0000000060a3e652
(XEN) rdx: 0000000060a3e652   rsi: 0000000060a3e652   rdi: 000000000026a84e
(XEN) rbp: ffff8800bf0837a8   rsp: ffff8800bf083758   r8:  0000000000000000
(XEN) r9:  ffffc900000cfb60   r10: 0000000000000000   r11: 0000000000018600
(XEN) r12: 0000000000000008   r13: 00000000608fbe7a   r14: 000000000026a84e
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000003
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffff8800bf083758:
(XEN)    ffffc900000cfb60 0000000000000000 000000000026a85c 000000006103b7ee
(XEN)    000000000026a85c 0000000000018600 0000000000000000 000000000000e739
(XEN)    ffff8800bf088000 ffff8800bf088000 ffff8800bf0837b8 ffffffff811feb0e
(XEN)    ffff8800bf0838a8 ffffffff81564911 0000003000000008 ffff8800bf0838b8
(XEN)    ffff8800bf0837e8 ffff8800bf088000 ffff8800bf088000 0000000000000000
(XEN)    ffff8800bf001880 ffff8800bf083868 ffff8800bf088000 ffffffff8173f414
(XEN)    ffff8800bf090010 0000000000000000 ffff8800bf083838 ffffffff810b060e
(XEN)    ffff8800bf083878 ffffffff8106c290 0000000000000040 ffffffff81566927
(XEN)    ffff8800bf0838a8 ffffffff81073d96 ffff8800bf083868 ffff8800bf083868
(XEN)    ffff8800bf080218 0000000000000009 ffff8800bf088000 ffff8800bf088000
(XEN)    ffffffff818356f0 ffff8800bf088000 ffff8800bf083908 ffffffff8106d9ab
(XEN)    0000000100000009 ffff8800bf080218 ffff8800bf0838c8 ffff8800bf0838c8
(XEN)    ffff8800bf0838e8 ffff8800bf083ac8 0000000000000009 ffff8800bf080218
(XEN)    ffff8800bf088000 0000000000000000 ffff8800bf083938 ffffffff815678ef
(XEN)    ffff8800bf088000 0000000000000009 0000000000000003 ffff8800bf083ac8
(XEN)    ffff8800bf083988 ffffffff8105292a ffff8800bf080218 ffff8800bf083bd6
(XEN)    ffffffff8177162d ffff8800bf088000 0000000000000003 ffff8800bf083ac8
(XEN)    0000000000000000 0000000000030001 ffff8800bf083a58 ffffffff81052ad1
(XEN)    0000000000000001 0000000000000004 ffff8800bf0839d0 0000000000000000
(XEN)    ffff880005480248 0000000000000020 ffffffff8177162f 000000200000000f
(XEN) *** Dumping Dom0 vcpu#9 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000009   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc900000dbe58   rsp: ffffc900000dbe30   r8:  ffff8800bf0b6000
(XEN) r9:  0000000000000000   r10: ffffc900000dbe98   r11: 0000000000000246
(XEN) r12: ffff8800bf10a240   r13: ffffc900000cfb58   r14: 0000000000000009
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc900000dbe30:
(XEN)    ffff8800bf10a240 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc900000dbe88 ffffffff8108ff63 ffff8800bf12b200
(XEN)    ffff8800bf10a240 0000000000000047 0000000000000047 ffffc900000dbe98
(XEN)    ffffffff8102fba5 ffffc900000dbed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf12b200 ffff8800bf12b264 0000000000000047 0000000000000100
(XEN)    0000000000001200 ffffc900000dbf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000047 0000000000000000 0000000000012350 ffffc900000dbf28
(XEN)    ffffffff8103667e ffffc900000dbf28 0000000000000001 ffffc900000dbfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0b7e48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc900000e3060 ffffc900000e3068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0b7e48
(XEN)    ffffffff81034ebe ffff8800bf0b7e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#10 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 000000000000000a   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc900000f3e58   rsp: ffffc900000f3e30   r8:  ffff8800bf0b8000
(XEN) r9:  0000000000000040   r10: ffffc900000f3e98   r11: 0000000000000246
(XEN) r12: ffff8800bf10a420   r13: ffffc900000cfb58   r14: 000000000000000a
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc900000f3e30:
(XEN)    ffff8800bf10a420 0000000000000080 ffffffff8102f98c ffffc900000f3e58
(XEN)    ffffc900000cfb30 ffffc900000f3e88 ffffffff8108ff63 ffff8800bf12b800
(XEN)    ffff8800bf10a420 000000000000004c 000000000000004c ffffc900000f3e98
(XEN)    ffffffff8102fba5 ffffc900000f3ed8 ffffffff810ad608 00000000000ffa30
(XEN)    ffff8800bf12b800 ffff8800bf12b864 000000000000004c 0000000000000100
(XEN)    0000000000001400 ffffc900000f3f08 ffffffff810aee38 ffffc900000f3f08
(XEN)    000000000000004c 0000000000000000 0000000000012350 ffffc900000f3f28
(XEN)    ffffffff8103667e ffffc900000f3f28 0000000000000001 ffffc900000f3fa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0b9e48 00000000000f3fa8 0000000000000000 ffffffffff57b000
(XEN)    ffffc900000fb060 ffffc900000fb068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0b9e48
(XEN)    ffffffff81034ebe ffff8800bf0b9e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#11 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 000000000000000b   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc9000010be58   rsp: ffffc9000010be30   r8:  ffff8800bf0ba000
(XEN) r9:  0000000000000040   r10: ffffc9000010be98   r11: 0000000000000246
(XEN) r12: ffff8800bf10a600   r13: ffffc900000cfb58   r14: 000000000000000b
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc9000010be30:
(XEN)    ffff8800bf10a600 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc9000010be88 ffffffff8108ff63 ffff8800bf12be00
(XEN)    ffff8800bf10a600 0000000000000051 0000000000000051 ffffc9000010be98
(XEN)    ffffffff8102fba5 ffffc9000010bed8 ffffffff810ad608 0000000000000001
(XEN)    ffff8800bf12be00 ffff8800bf12be64 0000000000000051 0000000000000100
(XEN)    0000000000001600 ffffc9000010bf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000051 0000000000000000 0000000000012350 ffffc9000010bf28
(XEN)    ffffffff8103667e ffffc9000010bf28 0000000000000001 ffffc9000010bfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0bbe48 000000000010bfa8 0000000000000000 ffffffffff57b000
(XEN)    ffffc90000113060 ffffc90000113068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0bbe48
(XEN)    ffffffff81034ebe ffff8800bf0bbe48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#12 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 000000000000000c   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc90000123e58   rsp: ffffc90000123e30   r8:  ffff8800bf0bc000
(XEN) r9:  0000000000000000   r10: ffffc90000123e98   r11: 0000000000000246
(XEN) r12: ffff8800bf10a7e0   r13: ffffc900000cfb58   r14: 000000000000000c
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc90000123e30:
(XEN)    ffff8800bf10a7e0 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc90000123e88 ffffffff8108ff63 ffff8800bf16c480
(XEN)    ffff8800bf10a7e0 0000000000000056 0000000000000056 ffffc90000123e98
(XEN)    ffffffff8102fba5 ffffc90000123ed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf16c480 ffff8800bf16c4e4 0000000000000056 0000000000000100
(XEN)    0000000000001800 ffffc90000123f08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000056 0000000000000000 0000000000012350 ffffc90000123f28
(XEN)    ffffffff8103667e ffffc90000123f28 0000000000000001 ffffc90000123fa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000140 0000000000000000
(XEN)    ffff8800bf0bde48 0000000000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc9000012b060 ffffc9000012b068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0bde48
(XEN)    ffffffff81034ebe ffff8800bf0bde48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#13 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 000000000000000d   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc9000013be58   rsp: ffffc9000013be30   r8:  ffff8800bf0be000
(XEN) r9:  0000000000000000   r10: ffffc9000013be98   r11: 0000000000000246
(XEN) r12: ffff8800bf10a9c0   r13: ffffc900000cfb58   r14: 000000000000000d
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc9000013be30:
(XEN)    ffff8800bf10a9c0 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc9000013be88 ffffffff8108ff63 ffff8800bf16ca80
(XEN)    ffff8800bf10a9c0 000000000000005b 000000000000005b ffffc9000013be98
(XEN)    ffffffff8102fba5 ffffc9000013bed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf16ca80 ffff8800bf16cae4 000000000000005b 0000000000000101
(XEN)    0000000000001a00 ffffc9000013bf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    000000000000005b 0000000000000001 0000000000012350 ffffc9000013bf28
(XEN)    ffffffff8103667e ffffc9000013bf28 0000000000000002 ffffc9000013bfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000141 0000000000000008
(XEN)    ffff8800bf0bfe48 0000004000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc90000143060 ffffc90000143068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0bfe48
(XEN)    ffffffff81034ebe ffff8800bf0bfe48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#14 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 000000000000000e   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc90000153e58   rsp: ffffc90000153e30   r8:  ffff8800bf0c0000
(XEN) r9:  0000000000000000   r10: ffffc90000153e98   r11: 0000000000000246
(XEN) r12: ffff8800bf10aba0   r13: ffffc900000cfb58   r14: 000000000000000e
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc90000153e30:
(XEN)    ffff8800bf10aba0 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc90000153e88 ffffffff8108ff63 ffff8800bf16d080
(XEN)    ffff8800bf10aba0 0000000000000060 0000000000000060 ffffc90000153e98
(XEN)    ffffffff8102fba5 ffffc90000153ed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf16d080 ffff8800bf16d0e4 0000000000000060 0000000000000101
(XEN)    0000000000001c00 ffffc90000153f08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000060 0000000000000001 0000000000012350 ffffc90000153f28
(XEN)    ffffffff8103667e ffffc90000153f28 0000000000000002 ffffc90000153fa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000141 0000000000000008
(XEN)    ffff8800bf0c1e48 0000004000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc9000015b060 ffffc9000015b068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0c1e48
(XEN)    ffffffff81034ebe ffff8800bf0c1e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) *** Dumping Dom0 vcpu#15 state: ***
(XEN) RIP:    e033:[<ffffffff8100930a>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 000000000000000f   rcx: ffffffff8100930a
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffc9000016be58   rsp: ffffc9000016be30   r8:  ffff8800bf0c2000
(XEN) r9:  0000000000000000   r10: ffffc9000016be98   r11: 0000000000000246
(XEN) r12: ffff8800bf10ad80   r13: ffffc900000cfb58   r14: 000000000000000f
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 0000000921001000   cr2: 0000000000000000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffc9000016be30:
(XEN)    ffff8800bf10ad80 0000000000000080 ffffffff8102f98c 0000000000000264
(XEN)    ffffc900000cfb30 ffffc9000016be88 ffffffff8108ff63 ffff8800bf16d680
(XEN)    ffff8800bf10ad80 0000000000000065 0000000000000065 ffffc9000016be98
(XEN)    ffffffff8102fba5 ffffc9000016bed8 ffffffff810ad608 000000008102f38f
(XEN)    ffff8800bf16d680 ffff8800bf16d6e4 0000000000000065 0000000000000101
(XEN)    0000000000001e00 ffffc9000016bf08 ffffffff810aee38 ffffffff8102f38f
(XEN)    0000000000000065 0000000000000001 0000000000012350 ffffc9000016bf28
(XEN)    ffffffff8103667e ffffc9000016bf28 0000000000000002 ffffc9000016bfa8
(XEN)    ffffffff8125b6f1 0000000000012350 0000000000000141 0000000000000008
(XEN)    ffff8800bf0c3e48 0000004000000000 0000000000000000 ffffffffff57b000
(XEN)    ffffc90000173060 ffffc90000173068 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8800bf0c3e48
(XEN)    ffffffff81034ebe ffff8800bf0c3e48 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000


[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 17:23           ` Bryan D. Payne
@ 2009-10-06 17:54             ` Jeremy Fitzhardinge
  2009-10-06 18:25               ` Bryan D. Payne
  0 siblings, 1 reply; 14+ messages in thread
From: Jeremy Fitzhardinge @ 2009-10-06 17:54 UTC (permalink / raw)
  To: Bryan D. Payne; +Cc: xen-devel

On 10/06/09 10:23, Bryan D. Payne wrote:
>> Is it ever stable?  What happens if you set dom0_mem to 4G or less?
>>     
> It still crashes, even with 2G of dom0 memory.
>   

What console output do you get?

>> Is the console responsive when the kernel hangs?  That is, can you type
>> "Ctrl-A Ctrl-A Ctrl-A" to get Xen, then enter debug keys?  '0' (zero)
>> should dump the context for dom0 and give some clue about where it is dying.
>>     
> Yes, this seems to work.  Typing 0 dumped info for each of the cores
> (i.e., 15 vcpu states).  I've attached the output.
>   

Could you map the RIP values to a symbol with gdb?  From a quick look,
it seems that most or all of the cpus are at the same place.  No, vcpu 8
is doing something else at least.

What happens if you boot dom0 with fewer cpus?

Do you have CONFIG_PARAVIRT_SPINLOCKS enabled?

    J

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 17:54             ` Jeremy Fitzhardinge
@ 2009-10-06 18:25               ` Bryan D. Payne
  2009-10-06 18:56                 ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 14+ messages in thread
From: Bryan D. Payne @ 2009-10-06 18:25 UTC (permalink / raw)
  To: Jeremy Fitzhardinge; +Cc: xen-devel

> What console output do you get?

It's the same as starting dom0 with 4G - 32G of memory, just freezes
at a different place in the boot each time.

> Could you map the RIP values to a symbol with gdb?  From a quick look,
> it seems that most or all of the cpus are at the same place.  No, vcpu 8
> is doing something else at least.

(gdb) x/i 0xffffffff8100930a
0xffffffff8100930a <hypercall_page+778>:	add    %al,(%rax)
(gdb) x/i 0xffffffff811fea48
0xffffffff811fea48 <delay_tsc+62>:	
    cmpq   $0x0,0x632538(%rip)        # 0xffffffff81830f88 <pv_cpu_ops+264>

> What happens if you boot dom0 with fewer cpus?

I tried adding "maxcpus=1" to the linux kernel line in grub.  I used
this in conjunction with the "dom0_mem=2G" option for xen.  Dom0 still
crashes the same as without the maxcpus option.

Just for kicks, I tried a few other options...

* setting mem=4G with dom0_mem=2G seemed still resulted in random dom0 crashing
* setting noapic resulted in a consistent crash within Xen:

(XEN) ----[ Xen-3.5-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48014fe29>] add_pin_to_irq+0x24/0xcc
(XEN) RFLAGS: 0000000000010296   CONTEXT: hypervisor

> Do you have CONFIG_PARAVIRT_SPINLOCKS enabled?

No.

-bryan

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 18:25               ` Bryan D. Payne
@ 2009-10-06 18:56                 ` Jeremy Fitzhardinge
  2009-10-06 19:02                   ` Bryan D. Payne
  0 siblings, 1 reply; 14+ messages in thread
From: Jeremy Fitzhardinge @ 2009-10-06 18:56 UTC (permalink / raw)
  To: Bryan D. Payne; +Cc: xen-devel, Keir Fraser

On 10/06/09 11:25, Bryan D. Payne wrote:
> (gdb) x/i 0xffffffff8100930a
> 0xffffffff8100930a <hypercall_page+778>:	add    %al,(%rax)
>   
778/32 = hypercall 24 = vcpuop.  Probably idling.
> (gdb) x/i 0xffffffff811fea48
> 0xffffffff811fea48 <delay_tsc+62>:	
>     cmpq   $0x0,0x632538(%rip)        # 0xffffffff81830f88 <pv_cpu_ops+264>
>   

That's almost certainly a kernel panic of some kind.  Working out where
it came from will be rather tedious: you need to look through the stack
dump to find code-ish looking addresses then x/i them (they'll be the
same basic format as 0xffffffff8xxxxxxx).

(I really need to work out why they tend not to get printed.)

>> What happens if you boot dom0 with fewer cpus?
>>     
> I tried adding "maxcpus=1" to the linux kernel line in grub.  I used
> this in conjunction with the "dom0_mem=2G" option for xen.  Dom0 still
> crashes the same as without the maxcpus option.
>
> Just for kicks, I tried a few other options...
>
> * setting mem=4G with dom0_mem=2G seemed still resulted in random dom0 crashing
> * setting noapic resulted in a consistent crash within Xen:
>
> (XEN) ----[ Xen-3.5-unstable  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e008:[<ffff82c48014fe29>] add_pin_to_irq+0x24/0xcc
> (XEN) RFLAGS: 0000000000010296   CONTEXT: hypervisor
>   

That shouldn't happen.  Sounds like it might be fall-out from the recent
interrupt changes in Xen.  Did you supply "noapic" to Xen, dom0 or both?

    J

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 18:56                 ` Jeremy Fitzhardinge
@ 2009-10-06 19:02                   ` Bryan D. Payne
  2009-10-06 19:37                     ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 14+ messages in thread
From: Bryan D. Payne @ 2009-10-06 19:02 UTC (permalink / raw)
  To: Jeremy Fitzhardinge; +Cc: xen-devel, Keir Fraser

> That's almost certainly a kernel panic of some kind.  Working out where
> it came from will be rather tedious: you need to look through the stack
> dump to find code-ish looking addresses then x/i them (they'll be the
> same basic format as 0xffffffff8xxxxxxx).

Nice ;-)  I'll let you know what I find out.

>> (XEN) ----[ Xen-3.5-unstable  x86_64  debug=y  Not tainted ]----
>> (XEN) CPU:    0
>> (XEN) RIP:    e008:[<ffff82c48014fe29>] add_pin_to_irq+0x24/0xcc
>> (XEN) RFLAGS: 0000000000010296   CONTEXT: hypervisor
>
> That shouldn't happen.  Sounds like it might be fall-out from the recent
> interrupt changes in Xen.  Did you supply "noapic" to Xen, dom0 or both?

Just Xen.  Should it go to both?

-bryan

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 19:02                   ` Bryan D. Payne
@ 2009-10-06 19:37                     ` Jeremy Fitzhardinge
  2009-10-07 11:47                       ` Jan Beulich
  0 siblings, 1 reply; 14+ messages in thread
From: Jeremy Fitzhardinge @ 2009-10-06 19:37 UTC (permalink / raw)
  To: Bryan D. Payne; +Cc: xen-devel, Keir Fraser

On 10/06/09 12:02, Bryan D. Payne wrote:
> Just Xen.  Should it go to both?
>   

Yeah, they need to agree what interrupt model they're using.  Still
shouldn't crash, regardless.

    J

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-06 19:37                     ` Jeremy Fitzhardinge
@ 2009-10-07 11:47                       ` Jan Beulich
  2009-10-07 19:09                         ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 14+ messages in thread
From: Jan Beulich @ 2009-10-07 11:47 UTC (permalink / raw)
  To: Jeremy Fitzhardinge, Bryan D. Payne; +Cc: xen-devel, Keir Fraser

>>> Jeremy Fitzhardinge <jeremy@goop.org> 06.10.09 21:37 >>>
>On 10/06/09 12:02, Bryan D. Payne wrote:
>> Just Xen.  Should it go to both?
>>   
>
>Yeah, they need to agree what interrupt model they're using.  Still
>shouldn't crash, regardless.

Not really - Xen automatically passes noapic to the kernel when it had
been passed that option (see the dom0_cmdline handling in
__start_xen()).

Jan

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: dom0 crash with unstable
  2009-10-07 11:47                       ` Jan Beulich
@ 2009-10-07 19:09                         ` Jeremy Fitzhardinge
  0 siblings, 0 replies; 14+ messages in thread
From: Jeremy Fitzhardinge @ 2009-10-07 19:09 UTC (permalink / raw)
  To: Jan Beulich; +Cc: xen-devel, Bryan D. Payne, Keir Fraser

On 10/07/09 04:47, Jan Beulich wrote:
> Not really - Xen automatically passes noapic to the kernel when it had
> been passed that option (see the dom0_cmdline handling in
> __start_xen()).
>   


Ah, OK.  I hadn't realized that.

    J

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2009-10-07 19:09 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-09-30 18:05 dom0 crash with unstable Bryan D. Payne
2009-09-30 23:47 ` Jeremy Fitzhardinge
2009-10-01 13:35   ` Bryan D. Payne
2009-10-01 21:01     ` Jeremy Fitzhardinge
2009-10-06 15:13       ` Bryan D. Payne
2009-10-06 16:31         ` Jeremy Fitzhardinge
2009-10-06 17:23           ` Bryan D. Payne
2009-10-06 17:54             ` Jeremy Fitzhardinge
2009-10-06 18:25               ` Bryan D. Payne
2009-10-06 18:56                 ` Jeremy Fitzhardinge
2009-10-06 19:02                   ` Bryan D. Payne
2009-10-06 19:37                     ` Jeremy Fitzhardinge
2009-10-07 11:47                       ` Jan Beulich
2009-10-07 19:09                         ` Jeremy Fitzhardinge

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.