xm dump-core and analyzing

xm dump-core and analyzing

[David Pilger]

Hi,

Is there any tool out there that can help me analyze a coredump?

Thanks,
David.

Re: xm dump-core and analyzing

[Akio Takebe]

Hi, David

Yes, you can use crash utility.
See the below.
     http://people.redhat.com/anderson

Best Regards,

Akio Takebe

>Hi,
>
>Is there any tool out there that can help me analyze a coredump?
>
>Thanks,
>David.
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

Re: xm dump-core and analyzing

[David Pilger]

Hi Akio, Thanks for your help.

But I don't think this tool is compatible with xm dump-core's product,
but rather with xen0/xenU kernel coredumps.

Anyone?

On 12/11/06, Akio Takebe <takebe_akio@jp.fujitsu.com> wrote:
> Hi, David
>
> Yes, you can use crash utility.
> See the below.
>     http://people.redhat.com/anderson
>
> Best Regards,
>
> Akio Takebe
>
> >Hi,
> >
> >Is there any tool out there that can help me analyze a coredump?
> >
> >Thanks,
> >David.
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel
>
>

Re: xm dump-core and analyzing

[Dave Anderson]

>
>
>
>
> Hi Akio, Thanks for your help.
>
> But I don't think this tool is compatible with xm dump-core's product,
> but rather with xen0/xenU kernel coredumps.
>
> Anyone?
>

The xm dump-core facility is just a way of forcing a xenU crash,
and uses the same dumpfile format as xenU kernel coredumps.

Xen0 coredumps require the kdump facility since xen0 needs to
be running in order to take a xenU coredump.

Anyway, the crash utility supports xendumps from paravirtualized
x86 and x86_64 xenU kernels, and vmcores from the currently
under development xen0-kdump facility.

Dave Anderson


>
> On 12/11/06, Akio Takebe <takebe_akio@jp.fujitsu.com> wrote:
> > Hi, David
> >
> > Yes, you can use crash utility.
> > See the below.
> >     http://people.redhat.com/anderson
> >
> > Best Regards,
> >
> > Akio Takebe
> >
> > >Hi,
> > >
> > >Is there any tool out there that can help me analyze a coredump?
> > >
> > >Thanks,
> > >David.
> > >

Re: xm dump-core and analyzing

[Keir Fraser]

On 11/12/06 15:34, "Dave Anderson" <anderson@redhat.com> wrote:

>> Hi Akio, Thanks for your help.
>> 
>> But I don't think this tool is compatible with xm dump-core's product,
>> but rather with xen0/xenU kernel coredumps.
>> 
>> Anyone?
> 
> The xm dump-core facility is just a way of forcing a xenU crash,
> and uses the same dumpfile format as xenU kernel coredumps.

I just want to check that you aren't under the impression that 'xm
dump-core' emits an Elf core file. It doesn't -- the format is custom and as
far as I know is only interpreted by a set of gdbserver patches that we ship
in our repository. Does the crash utility really support this special
format?

 -- Keir

Re: xm dump-core and analyzing

[Dave Anderson]

Keir Fraser wrote:

> On 11/12/06 15:34, "Dave Anderson" <anderson@redhat.com> wrote:
>
> >> Hi Akio, Thanks for your help.
> >>
> >> But I don't think this tool is compatible with xm dump-core's product,
> >> but rather with xen0/xenU kernel coredumps.
> >>
> >> Anyone?
> >
> > The xm dump-core facility is just a way of forcing a xenU crash,
> > and uses the same dumpfile format as xenU kernel coredumps.
>
> I just want to check that you aren't under the impression that 'xm
> dump-core' emits an Elf core file. It doesn't -- the format is custom and as
> far as I know is only interpreted by a set of gdbserver patches that we ship
> in our repository. Does the crash utility really support this special
> format?
>

It does -- at least for para-virtualized x86 and x86_64 xenU kernels that
have writeable page tables.  It's not clear to me whether it works on
those xenU kernels with shadow page tables -- I have not personally
seen it do so since we (Red Hat) ship x86/x86_64 xenU kernels
with writeable page tables.

With respect to fully-virtualized kernels, it does not work due to the
skipping of uninstantiated pseudo-pages in the xendump page list,
the issue that John Levon reported here:

http://lists.xensource.com/archives/html/xen-devel/2006-11/msg01226.html

And in addition, I have been working all along with Magnus Damm with
xen0/hypervisor vmcore support as well.

Dave

>
>  -- Keir

Re: xm dump-core and analyzing

[Keir Fraser]

On 11/12/06 15:59, "Dave Anderson" <anderson@redhat.com> wrote:

> It does -- at least for para-virtualized x86 and x86_64 xenU kernels that
> have writeable page tables.  It's not clear to me whether it works on
> those xenU kernels with shadow page tables -- I have not personally
> seen it do so since we (Red Hat) ship x86/x86_64 xenU kernels
> with writeable page tables.
> 
> With respect to fully-virtualized kernels, it does not work due to the
> skipping of uninstantiated pseudo-pages in the xendump page list,
> the issue that John Levon reported here:

Cool! :-)  We'd like to work on this in the Xen-3.0.5 timeframe so the xenU
dumps use a less brain-dead format (in fact, using Elf would be a good
idea!). The current format is an ancient, non-extensible and largely
unmaintained hack; since format compatibility has to be broken we may as
well fix it properly. I already discussed this a bit with John Levon in the
email thread you referenced: we should emit an Elf section for each
contiguous region of (pseudo-physical) address space and then also we will
need Elf notes for non-shadow-pagetable guests to provide the phys-machine
relationship.

 -- Keir

> http://lists.xensource.com/archives/html/xen-devel/2006-11/msg01226.html
> 
> And in addition, I have been working all along with Magnus Damm with
> xen0/hypervisor vmcore support as well.

Re: xm dump-core and analyzing

[Dave Anderson]

Keir Fraser wrote:

> On 11/12/06 15:59, "Dave Anderson" <anderson@redhat.com> wrote:
>
> > It does -- at least for para-virtualized x86 and x86_64 xenU kernels that
> > have writeable page tables.  It's not clear to me whether it works on
> > those xenU kernels with shadow page tables -- I have not personally
> > seen it do so since we (Red Hat) ship x86/x86_64 xenU kernels
> > with writeable page tables.
> >
> > With respect to fully-virtualized kernels, it does not work due to the
> > skipping of uninstantiated pseudo-pages in the xendump page list,
> > the issue that John Levon reported here:
>
> Cool! :-)  We'd like to work on this in the Xen-3.0.5 timeframe so the xenU
> dumps use a less brain-dead format (in fact, using Elf would be a good
> idea!). The current format is an ancient, non-extensible and largely
> unmaintained hack; since format compatibility has to be broken we may as
> well fix it properly. I already discussed this a bit with John Levon in the
> email thread you referenced: we should emit an Elf section for each
> contiguous region of (pseudo-physical) address space and then also we will
> need Elf notes for non-shadow-pagetable guests to provide the phys-machine
> relationship.
>

Cool right back at you!

Nothing would make me happier than to see the xendump format
replaced by an ELF format vmcore -- as long as I can make the
p2m translations.

I "get by" now with paravirtualized x86/x86_64 writeable page table kernels
because even though there are holes in the array of mfn's with respect
to their associated pfn's in the xendump file, I can:

(1) take the machine address of the cr3 from the xendump header,
(2) walk the (writeable) page tables to find the "phys_to_machine_mapping" symbol,
(3) read what's there, and then re-create the phys_to_machine_mapping[] array of the
     dumped kernel.

And from that point on, all p2m translations can be made by looking at that
re-created table for the mfn associated with any pfn, and then looking up
the mfn in the xendump corefile.

But ELF would be a hell of lot nicer...

Note that with Magnus Damm's latest hypervisor/xen0 kexec/kdump "combination"
vmcore format -- which has the additional XEN_ELFNOTE_CRASH_INFO ELF
note containing the dom0 pfn_to_mfn_frame_list_list value-- I can run a crash
session on the vmcore from the xen0 kernel perspective, as in:

 $ crash xen0-vmlinux vmcore

Additionally, Itsuro Oda from VAlinux has also provided a patch so that a crash
session can be run on the xen-syms binary as well, as in:

 $ crash xen-syms vmcore

When that occurs, the crash utility veers off and offers up a different
set of commands appropriate to the hypervisor, i.e., instead of commands
relevant to the vmlinux kernel.

And lastly, from the the xen-syms crash session I can easily find the
pfn_to_mfn_frame_list_list value for any other xenU guest domain,
and then be able to run a crash session on any guest that was
running at the time of the dom0/hypervisor crash:

  $ crash --p2m_mfn <mfn-value> xenU-vmlinux vmcore

Pretty nifty...

Dave

Re: xm dump-core and analyzing

[John Levon]

On Mon, Dec 11, 2006 at 03:48:43PM +0000, Keir Fraser wrote:

> I just want to check that you aren't under the impression that 'xm
> dump-core' emits an Elf core file. It doesn't -- the format is custom and as
> far as I know is only interpreted by a set of gdbserver patches that we ship

And us. (Though we're applying a version of the fix previously discussed
so the file is actually meaningful).

regards
john

Re: xm dump-core and analyzing

[John Levon]

On Mon, Dec 11, 2006 at 11:47:25AM -0500, Dave Anderson wrote:

> Cool right back at you!
> 
> Nothing would make me happier than to see the xendump format
> replaced by an ELF format vmcore -- as long as I can make the
> p2m translations.

Seconded (thirded?). ELF is a perfect format for this since it's
extendable and naturally understood by the debuggers we all have.

> I "get by" now with paravirtualized x86/x86_64 writeable page table kernels
> because even though there are holes in the array of mfn's with respect
> to their associated pfn's in the xendump file, I can:
> 
> (1) take the machine address of the cr3 from the xendump header,
> (2) walk the (writeable) page tables to find the "phys_to_machine_mapping" symbol,
> (3) read what's there, and then re-create the phys_to_machine_mapping[] array of the
>      dumped kernel.
> 
> And from that point on, all p2m translations can be made by looking at that
> re-created table for the mfn associated with any pfn, and then looking up
> the mfn in the xendump corefile.

How do you know what address that symbol is at? It's a requirement for
us that the dump is completely stand-alone. Ideally we would get this
issue fixed for 3.0.4, but I haven't found time to work on the full fix
that Keir suggested :/

regards
john

Re: xm dump-core and analyzing

[David Pilger]

On 12/11/06, Keir Fraser <keir@xensource.com> wrote:
> The current format is an ancient, non-extensible and largely
> unmaintained hack;

For the extensibility part, may I suggest that you turn
coredump_header.xch_nr_vcpus length into 16 bits and use the other 16
bits that left as an offset for a new header (PE style). I don't know
how old software will handle xch_nr_vcpus > 64K, as it always dumps
just the context for the first CPU.

But I understood that you are going to work on an ELF dump format...
good luck with that, it is surely needed!

Re: xm dump-core and analyzing

[John Levon]

On Tue, Dec 12, 2006 at 08:15:22AM +0200, David Pilger wrote:

> On 12/11/06, Keir Fraser <keir@xensource.com> wrote:
> >The current format is an ancient, non-extensible and largely
> >unmaintained hack;
> 
> For the extensibility part, may I suggest that you turn
> coredump_header.xch_nr_vcpus length into 16 bits and use the other 16
> bits that left as an offset for a new header (PE style). I don't know

Please let's not fiddle with the current format at all. In fact I'm
forced to rely on its exact formatting to work out whether it's a 64-bit
or 32-bit dump.

> how old software will handle xch_nr_vcpus > 64K, as it always dumps
> just the context for the first CPU.

What do you mean? All vcpu contexts are dumped.

regards
john

Re: xm dump-core and analyzing

[Dave Anderson]

John Levon wrote:

> On Mon, Dec 11, 2006 at 11:47:25AM -0500, Dave Anderson wrote:
>
> > Cool right back at you!
> >
> > Nothing would make me happier than to see the xendump format
> > replaced by an ELF format vmcore -- as long as I can make the
> > p2m translations.
>
> Seconded (thirded?). ELF is a perfect format for this since it's
> extendable and naturally understood by the debuggers we all have.
>
> > I "get by" now with paravirtualized x86/x86_64 writeable page table kernels
> > because even though there are holes in the array of mfn's with respect
> > to their associated pfn's in the xendump file, I can:
> >
> > (1) take the machine address of the cr3 from the xendump header,
> > (2) walk the (writeable) page tables to find the "phys_to_machine_mapping" symbol,
> > (3) read what's there, and then re-create the phys_to_machine_mapping[] array of the
> >      dumped kernel.
> >
> > And from that point on, all p2m translations can be made by looking at that
> > re-created table for the mfn associated with any pfn, and then looking up
> > the mfn in the xendump corefile.
>
> How do you know what address that symbol is at? It's a requirement for
> us that the dump is completely stand-alone. Ideally we would get this
> issue fixed for 3.0.4, but I haven't found time to work on the full fix
> that Keir suggested :/

>From the vmlinux file -- the crash utility is invoked similarly to gdb:

  $ crash vmlinux dumpfile

Dave

Re: xm dump-core and analyzing

[David Pilger]

On 12/12/06, John Levon <levon@movementarian.org> wrote:
>
> > how old software will handle xch_nr_vcpus > 64K, as it always dumps
> > just the context for the first CPU.
>
> What do you mean? All vcpu contexts are dumped.
>

My mistake...