capturing windows crash dumps

capturing windows crash dumps

[James Harper]

Windows has provision for a driver to get called when a bug check occurs
(KeRegisterBugCheckCallback etc), and also to get called every time some
data is to be written to the dump file.

I would like to be able to capture that information into Dom0 somehow,
as a few of the bug check's I've had in GPLPV in the past have rendered
the block device driver inoperable and therefore unable to write out the
dump file.

Is there an existing communication channel that could be used for this
or do I need to invent write my own? The more low level the better (as
in xenstore may not be accessible depending on the circumstances of the
crash).

Thanks

James

Re: capturing windows crash dumps

[Keir Fraser]

On 18/08/2009 07:47, "James Harper" <james.harper@bendigoit.com.au> wrote:

> Windows has provision for a driver to get called when a bug check occurs
> (KeRegisterBugCheckCallback etc), and also to get called every time some
> data is to be written to the dump file.
> 
> I would like to be able to capture that information into Dom0 somehow,
> as a few of the bug check's I've had in GPLPV in the past have rendered
> the block device driver inoperable and therefore unable to write out the
> dump file.
> 
> Is there an existing communication channel that could be used for this
> or do I need to invent write my own? The more low level the better (as
> in xenstore may not be accessible depending on the circumstances of the
> crash).

Could it be as simple as write the guest physical address of a pre-allocated
buffer into xenstore (from GPLPV drivers)? Then dump that area of guest
memory on guest bugcheck, even as a manual operation from dom0?

 -- Keir

RE: capturing windows crash dumps

[James Harper]

> On 18/08/2009 07:47, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > Windows has provision for a driver to get called when a bug check
occurs
> > (KeRegisterBugCheckCallback etc), and also to get called every time
some
> > data is to be written to the dump file.
> >
> > I would like to be able to capture that information into Dom0
somehow,
> > as a few of the bug check's I've had in GPLPV in the past have
rendered
> > the block device driver inoperable and therefore unable to write out
the
> > dump file.
> >
> > Is there an existing communication channel that could be used for
this
> > or do I need to invent write my own? The more low level the better
(as
> > in xenstore may not be accessible depending on the circumstances of
the
> > crash).
> 
> Could it be as simple as write the guest physical address of a
pre-allocated
> buffer into xenstore (from GPLPV drivers)? Then dump that area of
guest
> memory on guest bugcheck, even as a manual operation from dom0?
> 

The memory in question could be anything from 64kb (minidump) to the
size of DomU memory (full dump) or somewhere inbetween (kernel dump).
I'm normally just interested in the kernel dump which is normally a much
more respectable size, but still not the size of memory you can easily
find when the system has gone belly up.

James

Re: capturing windows crash dumps

[Keir Fraser]

On 18/08/2009 11:42, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> Could it be as simple as write the guest physical address of a
> pre-allocated
>> buffer into xenstore (from GPLPV drivers)? Then dump that area of
> guest
>> memory on guest bugcheck, even as a manual operation from dom0?
> 
> The memory in question could be anything from 64kb (minidump) to the
> size of DomU memory (full dump) or somewhere inbetween (kernel dump).
> I'm normally just interested in the kernel dump which is normally a much
> more respectable size, but still not the size of memory you can easily
> find when the system has gone belly up.

Hmm, okay. Could you dump it out over the emulated serial line?

 -- Keir

RE: capturing windows crash dumps

[James Harper]

> On 18/08/2009 11:42, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> >> Could it be as simple as write the guest physical address of a
> > pre-allocated
> >> buffer into xenstore (from GPLPV drivers)? Then dump that area of
> > guest
> >> memory on guest bugcheck, even as a manual operation from dom0?
> >
> > The memory in question could be anything from 64kb (minidump) to the
> > size of DomU memory (full dump) or somewhere inbetween (kernel
dump).
> > I'm normally just interested in the kernel dump which is normally a
much
> > more respectable size, but still not the size of memory you can
easily
> > find when the system has gone belly up.
> 
> Hmm, okay. Could you dump it out over the emulated serial line?
> 

I was thinking about that. It could work, then I could just do 'xm
console <domu> >dump' and parse the dump from there or something.

James

RE: capturing windows crash dumps

[Stefano Stabellini]

On Tue, 18 Aug 2009, James Harper wrote:
> > On 18/08/2009 11:42, "James Harper" <james.harper@bendigoit.com.au>
> wrote:
> > 
> > >> Could it be as simple as write the guest physical address of a
> > > pre-allocated
> > >> buffer into xenstore (from GPLPV drivers)? Then dump that area of
> > > guest
> > >> memory on guest bugcheck, even as a manual operation from dom0?
> > >
> > > The memory in question could be anything from 64kb (minidump) to the
> > > size of DomU memory (full dump) or somewhere inbetween (kernel
> dump).
> > > I'm normally just interested in the kernel dump which is normally a
> much
> > > more respectable size, but still not the size of memory you can
> easily
> > > find when the system has gone belly up.
> > 
> > Hmm, okay. Could you dump it out over the emulated serial line?
> > 
> 
> I was thinking about that. It could work, then I could just do 'xm
> console <domu> >dump' and parse the dump from there or something.
> 

Good idea, also keep in mind that qemu can emulate multiple serials if
needed.
Stubdoms support multiple serials too, so if you make the changes to add
another serial to qemu please test with stubdoms too :)

RE: capturing windows crash dumps

[James Harper]

> On Tue, 18 Aug 2009, James Harper wrote:
> > > On 18/08/2009 11:42, "James Harper"
<james.harper@bendigoit.com.au>
> > wrote:
> > >
> > > >> Could it be as simple as write the guest physical address of a
> > > > pre-allocated
> > > >> buffer into xenstore (from GPLPV drivers)? Then dump that area
of
> > > > guest
> > > >> memory on guest bugcheck, even as a manual operation from dom0?
> > > >
> > > > The memory in question could be anything from 64kb (minidump) to
the
> > > > size of DomU memory (full dump) or somewhere inbetween (kernel
> > dump).
> > > > I'm normally just interested in the kernel dump which is
normally a
> > much
> > > > more respectable size, but still not the size of memory you can
> > easily
> > > > find when the system has gone belly up.
> > >
> > > Hmm, okay. Could you dump it out over the emulated serial line?
> > >
> >
> > I was thinking about that. It could work, then I could just do 'xm
> > console <domu> >dump' and parse the dump from there or something.
> >
> 
> Good idea, also keep in mind that qemu can emulate multiple serials if
> needed.
> Stubdoms support multiple serials too, so if you make the changes to
add
> another serial to qemu please test with stubdoms too :)

How fast do you think that serial port output could be? Is the baud rate
limited by qemu? To put the question another way, the size of the dump
could be up to the size of DomU's memory so could be many gigabytes,
although would typically be <500MB, would that dump out in reasonable
time?

Thinking about it some more, it would be nice to have a mechanism that
could allow PV Linux/BSD/Solaris/etc to dump out similar information in
the event of a complete crash. What would be involved in having a
hypercall that could be called repeatedly to send data to Dom0? 1 page
at a time would be fine, so the parameters could be 'pfn, offset,
length'. I guess the problem wouldn't be in adding the hypercall but in
having the information end up in Dom0... would that be really difficult
and/or messy?

James

RE: capturing windows crash dumps

[James Harper]

> 
> On 18/08/2009 11:42, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> >> Could it be as simple as write the guest physical address of a
> > pre-allocated
> >> buffer into xenstore (from GPLPV drivers)? Then dump that area of
> > guest
> >> memory on guest bugcheck, even as a manual operation from dom0?
> >
> > The memory in question could be anything from 64kb (minidump) to the
> > size of DomU memory (full dump) or somewhere inbetween (kernel
dump).
> > I'm normally just interested in the kernel dump which is normally a
much
> > more respectable size, but still not the size of memory you can
easily
> > find when the system has gone belly up.
> 
> Hmm, okay. Could you dump it out over the emulated serial line?
> 

The more I look into it the less fond I get of this idea... I already
use the serial line for other things.

What about an event channel and a page of memory that could be retrieved
via the existing hvm_get_param mechanism for HVM domains and part of the
start_info page for PV domains? (just like store_mfn and store_evtchn
etc).

James

Re: capturing windows crash dumps

[Paul Durrant]

James Harper wrote:
> 
> The more I look into it the less fond I get of this idea... I already
> use the serial line for other things.
> 
> What about an event channel and a page of memory that could be retrieved
> via the existing hvm_get_param mechanism for HVM domains and part of the
> start_info page for PV domains? (just like store_mfn and store_evtchn
> etc).
> 

What about a 2nd serial line?

-- 
===============================
Paul Durrant, Software Engineer

Citrix Systems (R&D) Ltd.
First Floor, Building 101
Cambridge Science Park
Milton Road
Cambridge CB4 0FY
United Kingdom

TEL: x35957 (+44 1223 225957)
===============================

Re: capturing windows crash dumps

[Stefano Stabellini]

On Wed, 19 Aug 2009, Paul Durrant wrote:
> James Harper wrote:
> > 
> > The more I look into it the less fond I get of this idea... I already
> > use the serial line for other things.
> > 
> > What about an event channel and a page of memory that could be retrieved
> > via the existing hvm_get_param mechanism for HVM domains and part of the
> > start_info page for PV domains? (just like store_mfn and store_evtchn
> > etc).
> > 
> 
> What about a 2nd serial line?
> 

This is why I was mentioning the fact that you can have multiple
serials in qemu, and that stubdoms support it too.

Re: capturing windows crash dumps

[Konrad Rzeszutek Wilk]

> > > The memory in question could be anything from 64kb (minidump) to the
> > > size of DomU memory (full dump) or somewhere inbetween (kernel
> dump).
> > > I'm normally just interested in the kernel dump which is normally a
> much
> > > more respectable size, but still not the size of memory you can
> easily
> > > find when the system has gone belly up.
> > 
> > Hmm, okay. Could you dump it out over the emulated serial line?
> > 
> 
> The more I look into it the less fond I get of this idea... I already
> use the serial line for other things.

Not being familiar with Windows dumps, but only with Linux kdump (which is basically
a mini-Linux distro running in 64MB - that includes the initrd.img and + RAM space)
which can dump the entire memory on to whatever you want (NFS, disk, USB, etc). 

How does Windows natively/normally do its dump? Is it just dumping the entire memory space
on the local disk? Or do you have to set special flags to do so from the driver?

Re: capturing windows crash dumps

[Paul Durrant]

Konrad Rzeszutek Wilk wrote:
> 
> Not being familiar with Windows dumps, but only with Linux kdump (which is basically
> a mini-Linux distro running in 64MB - that includes the initrd.img and + RAM space)
> which can dump the entire memory on to whatever you want (NFS, disk, USB, etc). 
> 
> How does Windows natively/normally do its dump? Is it just dumping the entire memory space
> on the local disk? Or do you have to set special flags to do so from the driver?
> 

Yes, it normally dumps it's memory to the device using a special 
incarnation of the driver stack for that device.

   Paul

-- 
===============================
Paul Durrant, Software Engineer

Citrix Systems (R&D) Ltd.
First Floor, Building 101
Cambridge Science Park
Milton Road
Cambridge CB4 0FY
United Kingdom

===============================

RE: capturing windows crash dumps

[James Harper]

> > How does Windows natively/normally do its dump? Is it just dumping
the
> > entire memory space
> > on the local disk? Or do you have to set special flags to do so from
the
> > driver?
> 
> Yes, it normally dumps it's memory to the device using a special
> incarnation of the driver stack for that device.
> 

That is correct. An additional and completely separate version of the
driver is created at boot time and only activated in the event of a
crash. The driver is called with all interrupts disabled and cannot
allocate memory etc, so it's a very restricted environment. The ISR is
called on a timer. Hibernate is a separate copy of the driver again but
similar setup.

There are a few callbacks you can set in any driver to tell Windows to
call you every time it has some crash dump information to write out, and
the runtime environment is similar (interrupts disabled, very few system
calls at your disposal). It is this that I would like to use to write
out a crash dump to be captured by Dom0.

The two ideas under consideration are:
1. Use the serial port
2. Create an event channel and a mapped page and dump data out that way.
Add them to the start info page for PV and as HVM parameters (retrieved
via hypercall like the xenstore page and pfn) for HVM

I like #2 best for the following reasons:
. qemu needs patching for a second serial port anyway. I don't want to
use the primary as I'm already using it for something else in the
environment I'm thinking of.
. much more efficient
. works with PV (irrelevant though if there are no PV domains that
support writing out a dump file)
. easy for the DomU to tell if there is something waiting to capture the
crash dump (no parameter present = no crash dump)
. less code on the windows side of things, vs trying to access the
serial port hardware directly for which there is no 'blessed' mechanism
to do in a crash dump environment afaik

James

Re: capturing windows crash dumps

[Keir Fraser]

On 20/08/2009 10:56, "James Harper" <james.harper@bendigoit.com.au> wrote:

> I like #2 best for the following reasons:
> . qemu needs patching for a second serial port anyway. I don't want to
> use the primary as I'm already using it for something else in the
> environment I'm thinking of.
> . much more efficient
> . works with PV (irrelevant though if there are no PV domains that
> support writing out a dump file)
> . easy for the DomU to tell if there is something waiting to capture the
> crash dump (no parameter present = no crash dump)
> . less code on the windows side of things, vs trying to access the
> serial port hardware directly for which there is no 'blessed' mechanism
> to do in a crash dump environment afaik

Well, obviously you can knock up #2 yourself, with an accompanying little
dom0 daemon, which you can ship with the GPLPV drivers. The two components
can communicate via xenstore at start-of-day to prime themselves for later
possible crash dumps.

 -- Keir