xm pause <domain>

xm pause <domain>

[Chotu Ram]

Hi All,

I would like to know how does the pause functionality works with xm. I am able to
use it but not sure what exactly it does. The documentation mentions it but 
doesn't really say what does it do or what is it used for. My assumption is it 
stops the CPU cycles provided to the domain and so the domain kind of "freezes".

Also, once the domain is paused, is it possible to do some analysis on the
state... like what processes are running (which were froze now), what files were
open etc. from the hypervisor using the existing tools. Also, if such tools
don't exist, how difficult or easy would it be to develop such tools and which
parts of the code would have to be modified. Where does the code for pause exist?

I would really appreciate any help in this regard.

Thanks,

Chotu



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Mark Williamson]

> I would like to know how does the pause functionality works with xm. I am
> able to use it but not sure what exactly it does. The documentation
> mentions it but doesn't really say what does it do or what is it used for.
> My assumption is it stops the CPU cycles provided to the domain and so the
> domain kind of "freezes".

Yes, it hypercalls into Xen and causes the Xen CPU scheduler to no longer give 
any CPU time to the domain.

> Also, once the domain is paused, is it possible to do some analysis on the
> state... like what processes are running (which were froze now), what files
> were open etc. from the hypervisor using the existing tools.

Not as such.  Kip wrote some patches allowing GDB to analyse the state of a 
domain but this would be debugger level functionality - to do what you 
describe directly requires a bit more OS-specific knowledge.

> Also, if such 
> tools don't exist, how difficult or easy would it be to develop such tools
> and which parts of the code would have to be modified.

The Linux Kernel Crash Dump project (http://lkcd.sf.net) includes 
Linux-specific analysis tools that (given an LKCD-formatted dump file) can 
inspect what processes were running when a dump was taken, etc.  It would be 
nice if we could dump a domain in LKCD format and thus allow these tools to 
be leveraged.

> Where does the code 
> for pause exist?

IIRC, the code in Xen is in xen/common/dom0_ops.c and xen/common/schedule.c.  
The code for calling this from user space is in the Xc libs, which are called 
by Xend on receiving a request from the xm tool.

> I would really appreciate any help in this regard.

HTH,
Mark

>
> Thanks,
>
> Chotu
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Kip Macy]

>
> The Linux Kernel Crash Dump project (http://lkcd.sf.net) includes
> Linux-specific analysis tools that (given an LKCD-formatted dump file) can
> inspect what processes were running when a dump was taken, etc.  It would be
> nice if we could dump a domain in LKCD format and thus allow these tools to
> be leveraged.

I'm about to implement the crashdump support and am thinking a bit about
the format the dump should take - is the format documented anywhere
other than the tools themselves?



			-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Mark Williamson]

On Sunday 23 January 2005 23:09, Kip Macy wrote:
> > The Linux Kernel Crash Dump project (http://lkcd.sf.net) includes
> > Linux-specific analysis tools that (given an LKCD-formatted dump file)
> > can inspect what processes were running when a dump was taken, etc.  It
> > would be nice if we could dump a domain in LKCD format and thus allow
> > these tools to be leveraged.
>
> I'm about to implement the crashdump support and am thinking a bit about
> the format the dump should take - is the format documented anywhere
> other than the tools themselves?

Cool!

Some information here:
http://lkcd.sourceforge.net/doc/linuxworld2000/LKCD.html

There may be some more hanging around somewhere - I suspect that's the paper I 
remembered reading...

Of course, for non-Linux guests these tools won't necessarily be much (any?) 
use, so ELF format core files might prove more generally useful (I don't know 
if the LKCD lcrash tool can support ELF dump files - it'd be real nice if it 
did).

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Kip Macy]

> Cool!

Enthusiasm is always welcome  :-).

>
> Some information here:
> http://lkcd.sourceforge.net/doc/linuxworld2000/LKCD.html
>
> There may be some more hanging around somewhere - I suspect that's the paper I
> remembered reading...
>
> Of course, for non-Linux guests these tools won't necessarily be much (any?)
> use, so ELF format core files might prove more generally useful (I don't know
> if the LKCD lcrash tool can support ELF dump files - it'd be real nice if it
> did).

Having worked _briefly_ with elf coredumps as a starting point for
process checkpointing, I don't think they would be well suited for this.
An elf coredump defines all of a running applications mappings in the
program header. In a VM, there will be MANY mappings, which may be
fragmented, corresponding to many page directory pointers. If all I
cared about were the kernel state, that would be fine, but I think users
would like to be able to see what processes were doing at the time.
Examining the user-state will of course be very OS-specific.
Nonetheless, I think that it is important that the information be there,
so that if someone does choose to develop the tool, the pieces will
already be there.

Any corrections or contrary opinions are welcome ...



				-Kip



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Mark Williamson]

> Having worked _briefly_ with elf coredumps as a starting point for
> process checkpointing, I don't think they would be well suited for this.

You went over to another dump format for the DFly checkpointing, then?  What 
format does DFly / FreeBSD use for crash memory dumps?

> An elf coredump defines all of a running applications mappings in the
> program header. In a VM, there will be MANY mappings, which may be
> fragmented, corresponding to many page directory pointers

OK, makes sense.

> Examining the user-state will of course be very OS-specific.
> Nonetheless, I think that it is important that the information be there,
> so that if someone does choose to develop the tool, the pieces will
> already be there.

Yes.  I'm not sure how capable the LKCD lcrash tool is at analysing user 
process state but it makes sense to be able to get at it.  It's probably not 
too tough to convert between dump formats anyhow - the mkdump people support 
a couple of different formats this way.

> Any corrections or contrary opinions are welcome ...

This would be cool stuff to have both for developing and for debugging crashes 
in the field.  Xend could fairly easy be modified to auto-dump crashed 
domains, and to dump domains on request.  I like :-)

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Kip Macy]

>
> You went over to another dump format for the DFly checkpointing, then?  What
> format does DFly / FreeBSD use for crash memory dumps?
>
Sorry for not responding sooner. I had to refresh my memory a bit. We
store the info for the vnode of the executable, the signal handling
dispositions, and the info for the open file handles right after the
normal coreheader. The offsets for all the standard mappings is updated
to reflect the new information being added in. Thus, GDB still just
works.

I believe that I wrote the following:
/*
	 * put extra cruft for dumping process state here
	 *  - we really want it be before all the program
	 *    mappings
	 *  - we just need to update the offset accordingly
	 *    and GDB will be none the wiser.
	 */
	if (error == 0)
		error = elf_puttextvp(p, target);
	if (error == 0)
		error = elf_putsigs(p, target);
	if (error == 0)
		error = elf_putfiles(p, target);




>
> This would be cool stuff to have both for developing and for debugging crashes
> in the field.  Xend could fairly easy be modified to auto-dump crashed
> domains, and to dump domains on request.  I like :-)

I've written the code for a generic xen guest coredump, I'll write a
separate tool to post-process the header into something either LKCD or
FreeBSD's gdb will recognize.

Now the question is, can you give me some guidance on adding it into
xend - it doesn't look as straightforward as I had hoped.



			-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Mark A. Williamson]

> I've written the code for a generic xen guest coredump, I'll write a
> separate tool to post-process the header into something either LKCD or
> FreeBSD's gdb will recognize.

Cool.

> Now the question is, can you give me some guidance on adding it into
> xend - it doesn't look as straightforward as I had hoped.

You'll need a function in tools/libxc/ that'll do the dump (I guess this is 
what you have already).

Then you'll need to add a Python wrapper for this function in 
tools/python/xen/lowlevel/xc/xc.c.  This shouldn't be too hard, using the 
other functions in there as a template...

Finally you'll need to provide a means to dump using the xm tool.  This 
touches quite a few files, I'm afraid, although none of them require complex 
changes...

It looks like you'll need to edit:
* tools/python/xen/xm/main.py to add a subprogram for doing a dump.
* tools/python/xen/xend/XendClient.py to add a method for requesting a dump 
from the server
* tools/python/xen/xend/server/SrvDomain.py to add an operation for dumping a 
domain (this will get activated when the client makes a request)
* tools/python/xen/xend/XendDomain.py to add a domain_dump() method to be 
called by SrvDomain.py (this will pause the domain, call your Xc dump 
function and unpause it)

You may find it helpful to compare the call paths triggered by a destroy or a 
save, which will be similar to what you want.

For icing on the cake, you could also rig up some kind of dumping in the 
reap() function in XendDomain.py to autodump crashed domains before 
restarting / destroying them.

HTH,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Rolf Neugebauer]

On Wed, 2005-01-26 at 16:23, Mark A. Williamson wrote:
> > I've written the code for a generic xen guest coredump, I'll write a
> > separate tool to post-process the header into something either LKCD or
> > FreeBSD's gdb will recognize.
> 
> Cool.
> 
> > Now the question is, can you give me some guidance on adding it into
> > xend - it doesn't look as straightforward as I had hoped.
> 
> You'll need a function in tools/libxc/ that'll do the dump (I guess this is 
> what you have already).
> 
> Then you'll need to add a Python wrapper for this function in 
> tools/python/xen/lowlevel/xc/xc.c.  This shouldn't be too hard, using the 
> other functions in there as a template...

why do you want to add this to xend? I might be missing something here,
but can't you just provide a stand-alone util to do this. 

having bindings both for C (in libxc) and in python would still be
useful.

Rolf

> Finally you'll need to provide a means to dump using the xm tool.  This 
> touches quite a few files, I'm afraid, although none of them require complex 
> changes...
> 
> It looks like you'll need to edit:
> * tools/python/xen/xm/main.py to add a subprogram for doing a dump.
> * tools/python/xen/xend/XendClient.py to add a method for requesting a dump 
> from the server
> * tools/python/xen/xend/server/SrvDomain.py to add an operation for dumping a 
> domain (this will get activated when the client makes a request)
> * tools/python/xen/xend/XendDomain.py to add a domain_dump() method to be 
> called by SrvDomain.py (this will pause the domain, call your Xc dump 
> function and unpause it)
> 
> You may find it helpful to compare the call paths triggered by a destroy or a 
> save, which will be similar to what you want.
> 
> For icing on the cake, you could also rig up some kind of dumping in the 
> reap() function in XendDomain.py to autodump crashed domains before 
> restarting / destroying them.
> 
> HTH,
> Mark
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Kip Macy]

>
> why do you want to add this to xend? I might be missing something here,
> but can't you just provide a stand-alone util to do this.
>
I'd like it be possible as an option to have xen, before destroying a
domain, dump its core. This would require notifying xend. I'd also like
to add a HYPERVISOR_panic(char *) call so that guests can dump core
before going off into the weeds.



				-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Mark Williamson]

> why do you want to add this to xend? I might be missing something here,
> but can't you just provide a stand-alone util to do this.

I think this is worth having in Xend so that:
a) you can dump core automatically if a domain crashes
b) we preserve the ability to (at least in principle) remotely do this sort of 
thing using the HTTP interface

That said, if the code is already done, it might be worth writing a command 
line tool that does the dump directly for use in the meantime.

Cheers,
Mark

> having bindings both for C (in libxc) and in python would still be
> useful.
>
> Rolf
>
> > Finally you'll need to provide a means to dump using the xm tool.  This
> > touches quite a few files, I'm afraid, although none of them require
> > complex changes...
> >
> > It looks like you'll need to edit:
> > * tools/python/xen/xm/main.py to add a subprogram for doing a dump.
> > * tools/python/xen/xend/XendClient.py to add a method for requesting a
> > dump from the server
> > * tools/python/xen/xend/server/SrvDomain.py to add an operation for
> > dumping a domain (this will get activated when the client makes a
> > request) * tools/python/xen/xend/XendDomain.py to add a domain_dump()
> > method to be called by SrvDomain.py (this will pause the domain, call
> > your Xc dump function and unpause it)
> >
> > You may find it helpful to compare the call paths triggered by a destroy
> > or a save, which will be similar to what you want.
> >
> > For icing on the cake, you could also rig up some kind of dumping in the
> > reap() function in XendDomain.py to autodump crashed domains before
> > restarting / destroying them.
> >
> > HTH,
> > Mark
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> > Tool for open source databases. Create drag-&-drop reports. Save time
> > by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> > Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Daniel Stekloff]

On Sun, 2005-01-23 at 15:34, Kip Macy wrote:
> > Cool!
> 
> Enthusiasm is always welcome  :-).
> 
> >
> > Some information here:
> > http://lkcd.sourceforge.net/doc/linuxworld2000/LKCD.html
> >
> > There may be some more hanging around somewhere - I suspect that's the paper I
> > remembered reading...
> >
> > Of course, for non-Linux guests these tools won't necessarily be much (any?)
> > use, so ELF format core files might prove more generally useful (I don't know
> > if the LKCD lcrash tool can support ELF dump files - it'd be real nice if it
> > did).
> 
> Having worked _briefly_ with elf coredumps as a starting point for
> process checkpointing, I don't think they would be well suited for this.
> An elf coredump defines all of a running applications mappings in the
> program header. In a VM, there will be MANY mappings, which may be
> fragmented, corresponding to many page directory pointers. If all I
> cared about were the kernel state, that would be fine, but I think users
> would like to be able to see what processes were doing at the time.
> Examining the user-state will of course be very OS-specific.
> Nonetheless, I think that it is important that the information be there,
> so that if someone does choose to develop the tool, the pieces will
> already be there.
> 
> Any corrections or contrary opinions are welcome ...


Hi!

Wouldn't ELF header based dump files, like those used for netdump or
diskdump, be useful because you could use gdb to read them? Isn't the
kexec/kdump project doing something in this area? Since kexec/kdump has
quite a bit of backing, wouldn't it be helpful to dump something in the
same format so a single tool like crash could read vmcores dumped by
either method? 

I haven't had the chance to look this up yet, but does a crashdump done
through Xen require the dumping OS to help with the dump? Or, does the
hypervisor do it alone? Obviously, if the hypervisor can do it alone, it
would be better to just have crashdumps with Xen rather than requiring
kexec and invoking another kernel. If, however, the dumping OS is
required to perform the dump through Xen, then kdump may be the best way
to go. 

I'm interested in this area and would like to offer my help testing,
developing, etc. I will definitely look into how the dump is done using
Xen to start. Please let me know if I can help.

Thanks,

Dan



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: xm pause <domain>

[Mark Williamson]

> Wouldn't ELF header based dump files, like those used for netdump or
> diskdump, be useful because you could use gdb to read them?

That was my original thought but (as Kip says) there are good arguments for 
supporting other dump formats as well.

> Isn't the 
> kexec/kdump project doing something in this area? Since kexec/kdump has
> quite a bit of backing, wouldn't it be helpful to dump something in the
> same format so a single tool like crash could read vmcores dumped by
> either method?

Yes, that seems logical.  I suspect that if basic dumping functionality works, 
it won't be very difficult to support various formats.

> I haven't had the chance to look this up yet, but does a crashdump done
> through Xen require the dumping OS to help with the dump?

It can all be done by a userspace program in dom0 mapping the dumpee's memory 
and outputting it in a suitable format.  It shouldn't be necessary to require 
any co-operation (or, in fact, for the guest to even be aware of it).  The 
dump functionality would be used for guests, not for dom0 or Xen itself.

HTH,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl