Re: Coredumps and 'crash' utility...

Re: Coredumps and 'crash' utility...

[Dave Anderson]

> I'm using XenSource 4.0 and have the need to debug the
> XenSource-provided hypervisor's coredump file produce at /proc/vmcore.
> I tried using Anderson's "crash" utility to read that file but I'm
> getting the following error:
> 
> crash ./vmlinuz /dom0/proc/vmcore
> 
> crash 4.0-4.7
> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for
> details.
> 
> crash: /dom0/proc/vmcore: not a supported file format
> 
> 
> I examined the core file and this is what it says its type is.
> 
> #file /dom0/proc/vmcore
> /dom0/proc/vmcore: ELF 64-bit LSB core file AMD x86-64, version 1
> (SYSV), SVR4-style
> 
> 
> 1) Has anyone been able to use 'crash' with XenSource's kernels?
> 2) What is the right vmcore file format for an open-source Xen?  Can
> someone show me the output of 'file' on their vmcore so I can compare?

First, you won't get anywhere with the vmlinuz file; you must
use the kernel's vmlinux file, which must have been built with -g.

I don't test current XenSource kernels, but I don't know of any
recent changes from the Red Hat version that would break things.

A "readelf -a vmcore" would be more useful than a file output.
And what is the host machine that you are attempting the crash
session on?  One reason for the "not a supported file format"
would be if you attempt to run an x86 crash executable with
an x86_64 vmlinux/vmcore or vice-versa.

Also, it will probably be more useful to join and take this
up on the crash-utility mailing list:

  https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson

dumpread for core files...

[Roger Cruz]

As I have not found a way to connect a debugger for debugging a
hypervisor crash, I'm looking for the next solution, which is to read
the coredump file generated when the hypervisor crashes (this is with
XenSource 4.0, which does so automatically).

In my research I came across the following post that details a way to
break up the corefile into the hypervisor and domain 0 components and
then use GDB to read them.  This approach seems workable for me, except
that I can't find the utility dumpread.  The links given below are no
longer active.  Does anyone have access to the sources or a newer link?

Thanks

---------------------------------
Hi,

I updated dumpread scripts to extract domain/hypervisor image
from kdump whole-machine dump image.

Changes:
- libelf wrapper contains SWIG generated sources for OSs which has old
SWIG.
- fix scalability problem for dump file size.
- fix some typo.

You can get scripts:
http://people.valinux.co.jp/~moriwaka/dumpread/

and short howto:
http://wiki.xensource.com/xenwiki/XenKdumpAnalysis

Coredumps and 'crash' utility...

[Roger Cruz]

I'm using XenSource 4.0 and have the need to debug the
XenSource-provided hypervisor's coredump file produce at /proc/vmcore.
I tried using Anderson's "crash" utility to read that file but I'm
getting the following error:

crash ./vmlinuz /dom0/proc/vmcore

crash 4.0-4.7
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for
details.

crash: /dom0/proc/vmcore: not a supported file format


I examined the core file and this is what it says its type is.

#file /dom0/proc/vmcore
/dom0/proc/vmcore: ELF 64-bit LSB core file AMD x86-64, version 1
(SYSV), SVR4-style


1) Has anyone been able to use 'crash' with XenSource's kernels?
2) What is the right vmcore file format for an open-source Xen?  Can
someone show me the output of 'file' on their vmcore so I can compare?

Thank you in advance!
Roger Cruz

Re: Coredumps and 'crash' utility...

[Ian Campbell]

On Wed, 2007-10-03 at 12:20 -0400, Roger Cruz wrote:
> I'm using XenSource 4.0 and have the need to debug the
> XenSource-provided hypervisor's coredump file produce at /proc/vmcore.
> I tried using Anderson's "crash" utility to read that file but I'm
> getting the following error:
> 
> crash ./vmlinuz /dom0/proc/vmcore
[snip]
> crash: /dom0/proc/vmcore: not a supported file format
> 
> 
> I examined the core file and this is what it says its type is.
> 
> #file /dom0/proc/vmcore
> /dom0/proc/vmcore: ELF 64-bit LSB core file AMD x86-64, version 1
> (SYSV), SVR4-style
> 

Just a stab in the dark since I've never used crash but is it possible
that because we use a 64 bit hypervisor and 32 bit kernel/userspace in
domain0 you've ended up with a 32 bit version of crash which cannot read
the 64 bit core file?

Ian.

Re: Coredumps and 'crash' utility...

[Akio Takebe]

Hi, Roger

>
>I'm using XenSource 4.0 and have the need to debug the
>XenSource-provided hypervisor's coredump file produce at /proc/vmcore.
>I tried using Anderson's "crash" utility to read that file but I'm
>getting the following error:
>
>crash ./vmlinuz /dom0/proc/vmcore
>
I've never used XenSource 4.0, but is /dom0/proc/vmcore vmcore
created with kdump of xen?
And is ./vmlinuz compiled with CONFIG_DEBUG_INFO=y?
I don't know that crash support gzipped vmlinux,
because I always use vmlinux(not gzipped) compiled with CONFIG_DEBUG_INFO=y.

"readelf -x vmcore" and crash -d7 ./vmlinuz /dom0/proc/vmcore
are better infomations than file command.

Best Regards,

Akio Takebe

RE: Coredumps and 'crash' utility...

[Roger Cruz]

Sorry  about my late reply to the previous posting (included below) but
for some reason, I did not receive any of your replies.  I only came to
know of them when using Google today to find an answer to my problem.

Rather than reply individually and generate a lot of emails, I'm going
to do so in this one here.

>First, you won't get anywhere with the vmlinuz file; you must 
>use the kernel's vmlinux file, which must have been built with -g.

I have confirmed that the domain 0 vmlinux file is compiled with the -g
switch so the symbol info should be there.  It's also not obvious from
my post because I didn't rename the file, but the vmlinuz is the
uncompressed version of the XenSource-provided Domain 0 kernel.

>A "readelf -a vmcore" would be more useful than a file output.

I tried that but it doesn't appear to be like the format either. Here is
the output.

#readelf  -a /proc/vmcore
readelf: Error: Could not locate '/proc/vmcore'.  System error message:
Value too large for defined data type

  ls -l /proc/vmcore
-r-------- 1 root root 4210754772 Oct  9 19:41 /proc/vmcore

>And what is the host machine that you are attempting the crash 
>session on?  One reason for the "not a supported file format" 
>would be if you attempt to run an x86 crash executable with 
>an x86_64 vmlinux/vmcore or vice-versa.

The host machine is an x86_64.  I've been told that the hypervisor
supports 64-bits and that domain 0 is 32-bits but I'm not 100%.  I've
included the serial output of a normal boot using my privately modified
hypervisor.

>Also, it will probably be more useful to join and take this 
>up on the crash-utility mailing list:

Done.

>Just a stab in the dark since I've never used crash but is it possible 
> that because we use a 64 bit hypervisor and 32 bit kernel/userspace in

> domain0 you've ended up with a 32 bit version of crash which cannot
read 
> the 64 bit core file?

This sounds very plausible based on my other experiments.  See results
above of 'readelf' indicating a value too large.

>I've never used XenSource 4.0, but is /dom0/proc/vmcore vmcore 
> created with kdump of xen?

I don't know with 100% certainty.  It is created with kdump but I don't
know if they've modified the file format.  


The other tidbit which may relevant is that I'm mounting the crashing
XenServer as /dom0 in my Linux development station where I have the
sources.  This is so I don't have to copy the 4GB /proc/vmcore locally
so I can see the sources files.  I'm using SSHFS and FUSE to do this.

Thanks to all who've answered.





__  __            _____  _   ___  
 \ \/ /___ _ __   |___ / / | / _ \ 
  \  // _ \ '_ \    |_ \ | || | | |
  /  \  __/ | | |  ___) || || |_| |
 /_/\_\___|_| |_| |____(_)_(_)___/ 
                                   
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.1.0 (root@localdomain) (gcc version 4.1.1 20070105 (Red
Hat 4.1.1-52)) Wed Sep 12 15:32:10 EDT 2007
 Latest ChangeSet: 

(XEN) Command line: /boot/kona-rio-xen-3.1.0.gz com1=115200,8n1
console=com1,tty dom0_mem=752M lowmem_emergency_pool=16M
crashkernel=64M@32M
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 2 seconds
(XEN) Disc information:
(XEN)  Found 3 MBR signatures
(XEN)  Found 3 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f000 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000009f010000 (usable)
(XEN)  000000009f010000 - 000000009f019000 (ACPI NVS)
(XEN)  000000009f019000 - 000000009f028000 (ACPI data)
(XEN)  000000009f028000 - 000000009f071000 (ACPI NVS)
(XEN)  000000009f071000 - 000000009f0bf000 (ACPI data)
(XEN)  000000009f0bf000 - 000000009f0c1000 (ACPI NVS)
(XEN)  000000009f0c1000 - 000000009fb88000 (ACPI data)
(XEN)  000000009fb88000 - 000000009fb9d000 (reserved)
(XEN)  000000009fb9d000 - 000000009fc00000 (ACPI data)
(XEN)  00000000ffe00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000160000000 (usable)
(XEN) Kdump: 64MB (65536kB) at 0x2000000
(XEN) System RAM: 4079MB (4177596kB)
(XEN) Xen heap: 9MB (10056kB)
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) PAE enabled, limit: 16 GB
(XEN) Processor #0 6:15 APIC version 20
(XEN) Processor #6 6:15 APIC version 20
(XEN) Processor #1 6:15 APIC version 20
(XEN) Processor #7 6:15 APIC version 20
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2327.569 MHz processor.
(XEN) HVM: VMX enabled
(XEN) VMX: MSR intercept bitmap enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU            5140  @ 2.33GHz stepping 06
(XEN) Mapping cpu 0 to node 255
(XEN) Booting processor 1/6 eip 90000
(XEN) Mapping cpu 1 to node 255
(XEN) CPU1: Intel(R) Xeon(R) CPU            5140  @ 2.33GHz stepping 06
(XEN) Booting processor 2/1 eip 90000
(XEN) Mapping cpu 2 to node 255
(XEN) CPU2: Intel(R) Xeon(R) CPU            5140  @ 2.33GHz stepping 06
(XEN) Booting processor 3/7 eip 90000
(XEN) Mapping cpu 3 to node 255
(XEN) CPU3: Intel(R) Xeon(R) CPU            5140  @ 2.33GHz stepping 06
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer overflows in 14998 jiffies.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 32-bit, PAE, lsb
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0xc0100000 -> 0xc0440000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000003e000000->000000003f000000 (188416 pages
to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c0100000->c0440000
(XEN)  Init. ramdisk: c0440000->c07cb400
(XEN)  Phys-Mach map: c07cc000->c0888000
(XEN)  Start info:    c0888000->c088846c
(XEN)  Page tables:   c0889000->c0894000
(XEN)  Boot stack:    c0894000->c0895000
(XEN)  TOTAL:         c0000000->c0c00000
(XEN)  ENTRY ADDRESS: c0100000
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Initrd len 0x38b400, start at 0xc0440000
(XEN) Scrubbing Free RAM: ................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen).
(XEN) Freed 92kB init memory.
Linux version 2.6.18-8.1.8.el5.xs4.0.1.125.163xen (root@beefy-2) (gcc
version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Mon Aug 13 09:27:46
EDT 2007
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000002f800000 (usable)
32MB HIGHMEM available.
727MB LOWMEM available.
NX (Execute Disable) protection: active
found SMP MP-table at 000fdad0
DMI 2.3 present.
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24])
IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Enabling APIC mode:  Flat.  Using 2 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at a0000000 (gap: 9fc00000:60200000)
Built 1 zonelists.  Total pages: 194560
Kernel command line: root=LABEL=root-iojicdwe ro console=tty0
console=ttyS0,115200n8
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 16384 bytes)
Xen reported: 2327.518 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Software IO TLB enabled: 
 Aperture:     64 megabytes
 Kernel range: c16dd000 - c56dd000
 Address size: 27 bits
vmalloc area: ee000000-f51fe000, maxmem 2d7fe000
Memory: 687476k/778240k available (2012k kernel code, 82232k reserved,
688k data, 200k init, 32776k highmem)
Checking if this processor honours the WP bit even in supervisor mode...
Ok.
Calibrating delay using timer specific routine.. 4656.74 BogoMIPS
(lpj=23283711)
Security Framework v1.0.0 initialized
Capability LSM initialized
Mount-cache hash table entries: 512
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 4096K
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
ACPI: Core revision 20060707
ENABLING IO-APIC IRQs
SMP alternatives: switching to SMP code
Initializing CPU#1
Initializing CPU#2
Brought up 4 CPUs
Initializing CPU#3
migration_cost=9685
checking if image is initramfs... it is
Freeing initrd memory: 3629k freed
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: BIOS Bug: MCFG area at d0000000 is not E820-reserved
PCI: Not using MMCONFIG.
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [HPCI] (0000:00)
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: Wrong _BBN value, reboot and use option 'pci=noacpi'
ACPI: PCI Root Bridge [PCI0] (0000:00)
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0,
disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
(XEN) ioapic_guest_write: apic=0, pin=4, old_irq=4, new_irq=4
(XEN) ioapic_guest_write: old_entry=000009f1, new_entry=000109f1
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
pnp: PnP ACPI: found 24 devices
xen_mem: Initialising balloon driver.
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a
report
pnp: 00:06: ioport range 0x500-0x53f has been reserved
pnp: 00:06: ioport range 0x400-0x47f could not be reserved
pnp: 00:12: ioport range 0x500-0x53f has been reserved
pnp: 00:12: ioport range 0x400-0x47f could not be reserved
PCI: Failed to allocate mem resource #9:100000@a3100000 for 0000:02:02.0
PCI: Bridge: 0000:02:00.0
  IO window: 7000-7fff
  MEM window: a8300000-a92fffff
  PREFETCH window: a2100000-a30fffff
PCI: Bridge: 0000:02:02.0
  IO window: 6000-6fff
  MEM window: a8200000-a82fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:01:00.0
  IO window: 6000-7fff
  MEM window: a8200000-a92fffff
  PREFETCH window: a2100000-a30fffff
PCI: Bridge: 0000:01:00.3
  IO window: 4000-5fff
  MEM window: a7100000-a81fffff
  PREFETCH window: a1100000-a20fffff
PCI: Bridge: 0000:00:02.0
  IO window: 4000-7fff
  MEM window: a7100000-a93fffff
  PREFETCH window: a1100000-a30fffff
PCI: Bridge: 0000:00:03.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:04.0
  IO window: 3000-3fff
  MEM window: a6100000-a70fffff
  PREFETCH window: a3100000-a40fffff
PCI: Bridge: 0000:00:05.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:06.0
  IO window: 2000-2fff
  MEM window: a5100000-a60fffff
  PREFETCH window: a4100000-a50fffff
PCI: Bridge: 0000:00:07.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: 1000-1fff
  MEM window: a0000000-a10fffff
  PREFETCH window: a9500000-a95fffff
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:02:02.0[A] -> GSI 18 (level, low) -> IRQ 17
ACPI: PCI Interrupt 0000:01:00.3[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Enabling device 0000:00:04.0 (0000 -> 0003)
ACPI: PCI Interrupt 0000:00:04.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:05.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Enabling device 0000:00:06.0 (0000 -> 0003)
ACPI: PCI Interrupt 0000:00:06.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:07.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 17 (level, low) -> IRQ 18
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
loop: loaded (max 128 devices)
Xen virtual console successfully installed as ttyS0
Event-channel device installed.
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 0,0
PNP: PS/2 controller doesn't have KBD irq; using default 1
PNP: PS/2 controller doesn't have AUX irq; using default 12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI No-Shortcut mode
Freeing unused kernel memory: 200k freed
Red Hat nash version 5.1.19.6 starting

Mounting proc filesystem

Mounting sysfs filesystem

Creating /dev

Creating initial device nodes

Setting up hotplug.

input: AT Translated Set 2 keyboard as /class/input/input0
Creating block device nodes.

Loading usbcore.ko module

usbcore: registered new driver usbfs
usbcore: registered new driver hub
Loading uhci-hcd.ko module

USB Universal Host Controller Interface driver v3.0
PCI: Enabling device 0000:00:1d.0 (0000 -> 0001)
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 17 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 18, io base 0x00008080
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
PCI: Enabling device 0000:00:1d.1 (0000 -> 0001)
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 19
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 19, io base 0x00008060
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
PCI: Enabling device 0000:00:1d.2 (0000 -> 0001)
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 17
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 17, io base 0x00008040
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
PCI: Enabling device 0000:00:1d.3 (0000 -> 0001)
ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.3: irq 16, io base 0x00008020
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
input: ImPS/2 Generic Wheel Mouse as /class/input/input1
Loading ohci-hcd.ko module

Loading ehci-hcd.ko module

PCI: Enabling device 0000:00:1d.7 (0000 -> 0002)
ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 17 (level, low) -> IRQ 18
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 5
ehci_hcd 0000:00:1d.7: debug port 1
ehci_hcd 0000:00:1d.7: irq 18, io mem 0xa9400800
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 8 ports detected
end_request: I/O error, dev fd0, sector 0
Loading jbd.ko module

Loading ext3.ko module

Loading scsi_mod.ko module

SCSI subsystem initialized
Loading sd_mod.ko module

Loading libata.ko module

Loading ata_piix.ko module

ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 17
ata1: PATA max UDMA/100 cmd 0x000101f0 ctl 0x000103f6 bmdma 0x000180b0
irq 14
ata2: PATA max UDMA/100 cmd 0x00010170 ctl 0x00010376 bmdma 0x000180b8
irq 15
scsi0 : ata_piix
ata1.00: ATAPI, max UDMA/33
ata1.00: configured for UDMA/33
scsi1 : ata_piix
  Vendor: TEAC      Model: DW-552GA          Rev: R4K5
  Type:   CD-ROM                             ANSI SCSI revision: 05
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 19
ata3: SATA max UDMA/133 cmd 0x000180c8 ctl 0x000180e6 bmdma 0x000180a0
irq 19
ata4: SATA max UDMA/133 cmd 0x000180c0 ctl 0x000180e2 bmdma 0x000180a8
irq 19
scsi2 : ata_piix
ata3.00: ATA-6: WDC WD740GD-00FLC0, 33.08F33, max UDMA/133
ata3.00: 145226112 sectors, multi 16: LBA48 
ata3.01: ATA-7: WDC WD2500YS-01SHB1, 20.06C06, max UDMA/133
ata3.01: 490234752 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata3.00: configured for UDMA/133
ata3.01: configured for UDMA/133
scsi3 : ata_piix
ata4.00: ATA-7: WDC WD400JD-55MSA1, 10.01E01, max UDMA/133
ata4.00: 78165360 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata4.00: configured for UDMA/133
  Vendor: ATA       Model: WDC WD740GD-00FL  Rev: 33.0
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 145226112 512-byte hdwr sectors (74356 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
SCSI device sda: 145226112 512-byte hdwr sectors (74356 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3
sd 2:0:0:0: Attached scsi disk sda
  Vendor: ATA       Model: WDC WD2500YS-01S  Rev: 20.0
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdb: 490234752 512-byte hdwr sectors (251000 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
SCSI device sdb: 490234752 512-byte hdwr sectors (251000 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
 sdb: unknown partition table
sd 2:0:1:0: Attached scsi disk sdb
  Vendor: ATA       Model: WDC WD400JD-55MS  Rev: 10.0
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdc: 78165360 512-byte hdwr sectors (40021 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 78165360 512-byte hdwr sectors (40021 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
 sdc: unknown partition table
sd 3:0:0:0: Attached scsi disk sdc
Loading ide-disk.ko module

Loading ide-generic.ko module

ide0: I/O resource 0x1F0-0x1F7 not free.
ide0: ports already in use, skipping probe
ide1: I/O resource 0x170-0x177 not free.
ide1: ports already in use, skipping probe
Waiting for driver initialization.

Creating root device.

Mounting root filesystem.

end_request: I/O error, dev fd0, sector 0
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.

Setting up new root fs

no fstab.sys, mounting internal defaults

Switching to new root and running init.

unmounting old /dev

unmounting old /proc

unmounting old /sys

  warning: 4GB segment emulation, offending process: modprobe (pid=1161)


INIT: version 2.86 booting


		Welcome to XenServer

		Press 'I' to enter interactive startup.

Real Time Clock Driver v1.12ac
Setting clock  (utc): Wed Oct  3 18:59:07 EDT 2007 [  OK  ]


Starting udev: [  OK  ]


Loading default keymap (us): [  OK  ]


Setting hostname p20xen:  [  OK  ]


ibm_acpi: ec object not found
ACPI: Getting cpuindex for acpiid 0x2
ACPI: Getting cpuindex for acpiid 0x3
ACPI: Getting cpuindex for acpiid 0x4
ACPI: Getting cpuindex for acpiid 0x5
ACPI: Getting cpuindex for acpiid 0x6
ACPI: Getting cpuindex for acpiid 0x7
No devices found

Checking filesystems

Checking all file systems.

[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1 

root-iojicdwe: clean, 25867/501952 files, 471658/1002046 blocks

[  OK  ]


Remounting root filesystem in read-write mode:  [  OK  ]


Mounting local filesystems:  [  OK  ]


Enabling /etc/fstab swaps:  [  OK  ]


INIT: Entering runlevel: 3


Entering non-interactive startup

Loading crash kernel: [  OK  ]


Applying ip6tables firewall rules: [  OK  ]


Applying iptables firewall rules: [  OK  ]


Bringing up loopback interface:  [  OK  ]


Bringing up interface eth0:  [  OK  ]


Bringing up interface eth1:  [  OK  ]


Bringing up interface xenbr0:  

Determining IP information for xenbr0... done.

[  OK  ]


Bringing up interface xenbr1:  [  OK  ]


Starting system logger: [  OK  ]


Starting kernel logger: [  OK  ]


Starting portmap: [  OK  ]


Starting NFS statd: [  OK  ]


Starting RPC idmapd: [  OK  ]


Mounting other filesystems:  [  OK  ]


Unplugging extra VCPUs: cpu1 cpu2 cpu3 [  OK  ]


Setting control domain uuid: [  OK  ]


Starting xenstored: 

Starting sshd: [  OK  ]


ntpd: Synchronizing with time server: [  OK  ]


Starting ntpd: [  OK  ]


Applying Intel IA32 Microcode update... 

^[[60G[  ^[[1;32mOK^[[0;39m  ]




Starting crond: [  OK  ]


Starting xapi SSL: [  OK  ]


Starting xapi: ..start-of-day complete.[  OK  ]


Starting ElxRMSrv:  [  OK  ]




XenServer Host 4.0.1-4249p



System Booted: 2007-10-03 18:59



Your XenServer Host has now finished booting. To manage this

server please use the XenServer XenCenter

application. You can install the XenServer XenCenter

for Windows from the XenServer install media.





You can connect to this system using one of the following network

addresses:



10.40.193.64


-----Original Message-----
From: Roger Cruz 
Sent: Wednesday, October 03, 2007 12:20 PM
To: Roger Cruz; xen-devel@lists.xensource.com
Subject: Coredumps and 'crash' utility...


I'm using XenSource 4.0 and have the need to debug the
XenSource-provided hypervisor's coredump file produce at /proc/vmcore.
I tried using Anderson's "crash" utility to read that file but I'm
getting the following error:

crash ./vmlinuz /dom0/proc/vmcore

crash 4.0-4.7
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for
details.

crash: /dom0/proc/vmcore: not a supported file format


I examined the core file and this is what it says its type is.

#file /dom0/proc/vmcore
/dom0/proc/vmcore: ELF 64-bit LSB core file AMD x86-64, version 1
(SYSV), SVR4-style


1) Has anyone been able to use 'crash' with XenSource's kernels?
2) What is the right vmcore file format for an open-source Xen?  Can
someone show me the output of 'file' on their vmcore so I can compare?

Thank you in advance!
Roger Cruz