save image file format? and [RFC] tmem save/restore/migrate

save image file format? and [RFC] tmem save/restore/migrate

[Dan Magenheimer]

Is there any documentation on the format of the file/stream
used for save/restore/migrate?  Or is it just a sequence of
chunks of bytes pre-defined only by the ordering of the
save/restore code?

I found this 2006 xen-devel thread, but it doesn't look like
it (other than hvm additions) ever happened?  E.g. still
no versioning, self-identification, extensibility?

http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00440.html

I'm starting to look at save/restore/migrate for tmem and will
need to communicate the following information via file/stream:

- pool id and characteristics (a small number of bytes of data),
  for some small number of pools
- for some classes of pools, some number of "pages" of data,
  each page consisting of a "handle" (128 bits) and PAGE_SIZE
  bytes of data associated with that handle
- for some of these pages of data, a handle+invalidate (see below)
- (optional) in some cases the pages will be pre-compressed;
  each can be decompressed on the source side and recompressed
  on the destination side, but this seems a sad waste of
  cpu cycles (though necessary if the compression algorithm
  were different on source and destination); if possible,
  save/transmit non-decompressed data

Note that for the pages of data, dirtying during migration
is not possible, however invalidation IS possible.  E.g.
unlike normally addressable pages which may be transmitted
multiple times during a live migration, a transmitted
tmem page (handle and data) will be transmitted only once,
but may be followed at some point with a handle+invalidate.

The ordering of tmem info/pages vs current saved info/data
is flexible, but the number of pages could be very large,
so for live migrate, transmission should NOT be postponed
until the "final pass" of normal page transmission (e.g.
after the domain has been paused on the source machine).

I also need code to verify that the destination has tmem
support and it is enabled.  Only PV domains can use tmem
so no HVM changes should be necessary.

Any pointers or suggestions welcome, especially any thoughts
on changes that might be required above libxc such as in
python code or (heaven forbid) ioemu/qemu.

Thanks for any help!
Dan

Re: save image file format? and [RFC] tmem save/restore/migrate

[John Levon]

On Wed, Jun 17, 2009 at 04:00:30PM -0700, Dan Magenheimer wrote:

> Is there any documentation on the format of the file/stream
> used for save/restore/migrate?  Or is it just a sequence of
> chunks of bytes pre-defined only by the ordering of the
> save/restore code?
>
> http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00440.html

It's still the case, sadly. The HVM changes were hacked in instead.

regards
john

Re: save image file format? and [RFC] tmem save/restore/migrate

[Tim Deegan]

At 00:00 +0100 on 18 Jun (1245283230), Dan Magenheimer wrote:
> Is there any documentation on the format of the file/stream
> used for save/restore/migrate?  Or is it just a sequence of
> chunks of bytes pre-defined only by the ordering of the
> save/restore code?
> 
> I found this 2006 xen-devel thread, but it doesn't look like
> it (other than hvm additions) ever happened?  E.g. still
> no versioning, self-identification, extensibility?

It's still the ad-hoc stream-of-data format, I'm afraid.  The whole
format needs a good kicking.  It's not even portable between 32-bit and
64-bit tools.  Gianluca (Cc'd) is just starting to look at the
save/restore code in the hopes of making it more sane, so now would be a
good time to bring up any suggestions.

The usual way of adding new fields is to grab another negative number in
the length-of-the-next-block-of-frames field.

BTW, the HVM save records supplied by Xen to libxc _are_
self-identifying and extensible (and there's room in the header for a
version number, though by sticking to the principle of only transferring
architectural state we've avoided the need to use it so far).  But
they're just dropped into the stream after the memory pages and before
xend glues on the qemu record.

Cheers, 

Tim.

> http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00440.html
> 
> I'm starting to look at save/restore/migrate for tmem and will
> need to communicate the following information via file/stream:
> 
> - pool id and characteristics (a small number of bytes of data),
>   for some small number of pools
> - for some classes of pools, some number of "pages" of data,
>   each page consisting of a "handle" (128 bits) and PAGE_SIZE
>   bytes of data associated with that handle
> - for some of these pages of data, a handle+invalidate (see below)
> - (optional) in some cases the pages will be pre-compressed;
>   each can be decompressed on the source side and recompressed
>   on the destination side, but this seems a sad waste of
>   cpu cycles (though necessary if the compression algorithm
>   were different on source and destination); if possible,
>   save/transmit non-decompressed data
> 
> Note that for the pages of data, dirtying during migration
> is not possible, however invalidation IS possible.  E.g.
> unlike normally addressable pages which may be transmitted
> multiple times during a live migration, a transmitted
> tmem page (handle and data) will be transmitted only once,
> but may be followed at some point with a handle+invalidate.
> 
> The ordering of tmem info/pages vs current saved info/data
> is flexible, but the number of pages could be very large,
> so for live migrate, transmission should NOT be postponed
> until the "final pass" of normal page transmission (e.g.
> after the domain has been paused on the source machine).
> 
> I also need code to verify that the destination has tmem
> support and it is enabled.  Only PV domains can use tmem
> so no HVM changes should be necessary.
> 
> Any pointers or suggestions welcome, especially any thoughts
> on changes that might be required above libxc such as in
> python code or (heaven forbid) ioemu/qemu.
> 
> Thanks for any help!
> Dan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

Re: save image file format? and [RFC] tmem save/restore/migrate

[John Levon]

On Thu, Jun 18, 2009 at 09:51:58AM +0100, Tim Deegan wrote:

> It's still the ad-hoc stream-of-data format, I'm afraid.  The whole
> format needs a good kicking.  It's not even portable between 32-bit and
> 64-bit tools.  Gianluca (Cc'd) is just starting to look at the
> save/restore code in the hopes of making it more sane, so now would be a
> good time to bring up any suggestions.

One possibility Anthony Liguori brought up before is using all of qemu's
container format. I'm not sure it's ever feasible to make it fully
compatible, but it certainly seems like an obvious choice to use as a
base. Unfortunately, I think it's defined-by-the-code as well.

regards
john

Re: save image file format? and [RFC] tmem save/restore/migrate

[Stefano Stabellini]

John Levon wrote:

> On Thu, Jun 18, 2009 at 09:51:58AM +0100, Tim Deegan wrote:
> 
>> It's still the ad-hoc stream-of-data format, I'm afraid.  The whole
>> format needs a good kicking.  It's not even portable between 32-bit and
>> 64-bit tools.  Gianluca (Cc'd) is just starting to look at the
>> save/restore code in the hopes of making it more sane, so now would be a
>> good time to bring up any suggestions.
> 
> One possibility Anthony Liguori brought up before is using all of qemu's
> container format. I'm not sure it's ever feasible to make it fully
> compatible, but it certainly seems like an obvious choice to use as a
> base. Unfortunately, I think it's defined-by-the-code as well.
> 

At least in theory is certainly feasible: just a matter or registering
another savevm function for a record called "cpu" or "vcpu", that would
take care of saving the guest memory using xc_domain_save.

The qemu people are also maintaing save record compatibility now, so we
are safe from that perspective.

Re: save image file format? and [RFC] tmem save/restore/migrate

[Tim Deegan]

At 13:02 +0100 on 18 Jun (1245330161), Stefano Stabellini wrote:
> At least in theory is certainly feasible: just a matter or registering
> another savevm function for a record called "cpu" or "vcpu", that would
> take care of saving the guest memory using xc_domain_save.

Yuck.  You still need to add a bunch of metadata from xend that qemu
doesn't know about, so you'll have to wrap the qemu output in another
file format already.  Wrapping the memory image inside qemu's image is
just layering for its own sake.  (Also, using qemu to save a PV guests 
would be pretty wierd).

> The qemu people are also maintaing save record compatibility now, so we
> are safe from that perspective.

Yes, but their code-defines-format model is rubbish.  I've written a
couple of tools that parse HVM save file info based on the Xen public
headers; doing the same for a qemu-style implicit format is a PITA.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

Re: save image file format? and [RFC] tmem save/restore/migrate

[John Levon]

On Thu, Jun 18, 2009 at 01:05:58PM +0100, Tim Deegan wrote:

> At 13:02 +0100 on 18 Jun (1245330161), Stefano Stabellini wrote:
> > At least in theory is certainly feasible: just a matter or registering
> > another savevm function for a record called "cpu" or "vcpu", that would
> > take care of saving the guest memory using xc_domain_save.
> 
> Yuck.  You still need to add a bunch of metadata from xend that qemu
> doesn't know about, so you'll have to wrap the qemu output in another
> file format already.

Why? The qemu format is extensible, no? Why isn't this just an .sxp
SaveStateEntry ?

Think about it from the point of view of a save-file inspector. If they
can write the container-unwrapping code just once, and have specific
methods for Xen-particular or kvm-particular sections, that seems like a
clear win.

> just layering for its own sake.  (Also, using qemu to save a PV guests 
> would be pretty wierd).

Why? We already use qemu for PV guests and that's only going to become
more common. But heck, if you like, make libxc write out qemu format.

> > The qemu people are also maintaing save record compatibility now, so we
> > are safe from that perspective.
> 
> Yes, but their code-defines-format model is rubbish.

Maybe now is the time to help them fix that? It's really no worse than
Xen's code-defines-format model, headers or not.

If we do need a separate container format, let's use ELF like the core
files (slightly extended as I mentioned). Just not yet another format
when there's no need.

regards
john

Re: save image file format? and [RFC] tmem save/restore/migrate

[Tim Deegan]

At 13:26 +0100 on 18 Jun (1245331580), John Levon wrote:
> > just layering for its own sake.  (Also, using qemu to save a PV guests 
> > would be pretty wierd).
> 
> Why? We already use qemu for PV guests and that's only going to become
> more common.

Is it?  AIUI the only thing we use qemu for in PV guests is the pvfb
backend, which is just because nobody's put in a proper library
interface to that code.

> > > The qemu people are also maintaing save record compatibility now, so we
> > > are safe from that perspective.
> > 
> > Yes, but their code-defines-format model is rubbish.
> 
> Maybe now is the time to help them fix that? It's really no worse than
> Xen's code-defines-format model, headers or not.

Qemu's code-defines-format model is actually much better than the
xend/libxc code-defines-format model. :)  But that's not to say it's the
thing we should be copying.

> If we do need a separate container format, let's use ELF like the core
> files (slightly extended as I mentioned). Just not yet another format
> when there's no need.

ELF could work; it gives us easy access to unpacking code in several
languages.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

Re: save image file format? and [RFC] tmem save/restore/migrate

[John Levon]

On Thu, Jun 18, 2009 at 01:43:05PM +0100, Tim Deegan wrote:

> At 13:26 +0100 on 18 Jun (1245331580), John Levon wrote:
> > > just layering for its own sake.  (Also, using qemu to save a PV guests 
> > > would be pretty wierd).
> > 
> > Why? We already use qemu for PV guests and that's only going to become
> > more common.
> 
> Is it?  AIUI the only thing we use qemu for in PV guests is the pvfb
> backend, which is just because nobody's put in a proper library
> interface to that code.

The console as well if xenconsoled isn't running.

> > > Yes, but their code-defines-format model is rubbish.
> > 
> > Maybe now is the time to help them fix that? It's really no worse than
> > Xen's code-defines-format model, headers or not.
> 
> Qemu's code-defines-format model is actually much better than the
> xend/libxc code-defines-format model. :)  But that's not to say it's the
> thing we should be copying.

Well, given we're agreed that we should have a document of /some/
specification, why not qemu's? Presumably we need to document the state
that qemu /does/ save for HVM guests anyway, so why not the whole thing?

> > If we do need a separate container format, let's use ELF like the core
> > files (slightly extended as I mentioned). Just not yet another format
> > when there's no need.
> 
> ELF could work; it gives us easy access to unpacking code in several
> languages.

I admit ELF is easier to work with. It's a shame qemu (and Xen!) didn't
use it from the start.

regards
john

Re: save image file format? and [RFC] tmem save/restore/migrate

[Tim Deegan]

At 13:47 +0100 on 18 Jun (1245332839), John Levon wrote:
> Well, given we're agreed that we should have a document of /some/
> specification, why not qemu's? Presumably we need to document the state
> that qemu /does/ save for HVM guests anyway, so why not the whole thing?

If we have our own format (using ELF or indeed any other container
format to break it into typed sections) then we get to use our own
typespace rather than asking for a slice of theirs. :)

More seriously, I don't think it gets us enough real advantage to
comensate for its disadvantages: it's an ad-hoc, evolved, implicitly
defined format, which is the sort of thing we're trying to avoid; It
encourages further aggregation of unrelated dom0 tools into qemu (I
understand that some people see that as a good thing, but I don't); and
it's tied down to the needs and history of qemu, which are not entirely
aligned with Xen's.

Anyway, I think we've probably made our positions clear; Gianluca will
presumably have his own ideas.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

Re: save image file format? and [RFC] tmem save/restore/migrate

[Gianluca Guida]

Hello,

xen-devel calling, of course during days off! :-)

On Thu, Jun 18, 2009 at 4:18 PM, Tim Deegan<Tim.Deegan@citrix.com> wrote:
> More seriously, I don't think it gets us enough real advantage to
> comensate for its disadvantages: it's an ad-hoc, evolved, implicitly
> defined format, which is the sort of thing we're trying to avoid; It
> encourages further aggregation of unrelated dom0 tools into qemu (I
> understand that some people see that as a good thing, but I don't); and
> it's tied down to the needs and history of qemu, which are not entirely
> aligned with Xen's.
>
> Anyway, I think we've probably made our positions clear; Gianluca will
> presumably have his own ideas.

Thanks everyone for the comments on the subject!

I admit I am not actually 'passionate' about file formats, but I do
agree with those who said that using *qemu* to save a VM is ugly. I
personally think that it would be yet another dangerous precedent for
the idea that qemu should be used for stuff which are completely
unrelated with hardware virtualization.

As for the file formats, I think ELF idea is cute: modular,
extendible, versioned, and easily writable by libelf.

Let me know if you have further comments, and please note that I am
just starting to work on a general overhaul of save/restore code, so
it will take a while for me to get some code out of this talk.

Thanks,
Gianluca

-- 
It was a type of people I did not know, I found them very strange and
they did not inspire confidence at all. Later I learned that I had been
introduced to electronic engineers.
                                                  E. W. Dijkstra

Re: save image file format? and [RFC] tmem save/restore/migrate

[Stefano Stabellini]

Tim Deegan wrote:

> At 13:26 +0100 on 18 Jun (1245331580), John Levon wrote:
>>> just layering for its own sake.  (Also, using qemu to save a PV guests 
>>> would be pretty wierd).
>> Why? We already use qemu for PV guests and that's only going to become
>> more common.
> 
> Is it?  AIUI the only thing we use qemu for in PV guests is the pvfb
> backend, which is just because nobody's put in a proper library
> interface to that code.


We also use it for the console backend when a vfb is given to the guest.

 
>>>> The qemu people are also maintaing save record compatibility now, so we
>>>> are safe from that perspective.
>>> Yes, but their code-defines-format model is rubbish.
>> Maybe now is the time to help them fix that? It's really no worse than
>> Xen's code-defines-format model, headers or not.
> 
> Qemu's code-defines-format model is actually much better than the
> xend/libxc code-defines-format model. :)  But that's not to say it's the
> thing we should be copying.

It is more about reusing than copying.

Re: save image file format? and [RFC] tmem save/restore/migrate

[Stefano Stabellini]

Tim Deegan wrote:

> At 13:02 +0100 on 18 Jun (1245330161), Stefano Stabellini wrote:
>> At least in theory is certainly feasible: just a matter or registering
>> another savevm function for a record called "cpu" or "vcpu", that would
>> take care of saving the guest memory using xc_domain_save.
> 
> Yuck.  You still need to add a bunch of metadata from xend that qemu
> doesn't know about, so you'll have to wrap the qemu output in another
> file format already.  Wrapping the memory image inside qemu's image is
> just layering for its own sake.  (Also, using qemu to save a PV guests 
> would be pretty wierd).


Most (but unfortunately not all) the information saved by xend are qemu
command line options or easy to get from them.
Certainly being backward compatible is going to be harder this way than
involving xend.

 
>> The qemu people are also maintaing save record compatibility now, so we
>> are safe from that perspective.
> 
> Yes, but their code-defines-format model is rubbish.  I've written a
> couple of tools that parse HVM save file info based on the Xen public
> headers; doing the same for a qemu-style implicit format is a PITA.
> 

We can always write an rfc that documents the format and request them to
keep it up to date.
We can also reuse exiting loadvm qemu code to write tools, even though
it wouldn't be as elegant as using xen public headers.

RE: save image file format? and [RFC] tmem save/restore/migrate

[Dan Magenheimer]

> BTW, the HVM save records supplied by Xen to libxc _are_
> self-identifying and extensible (and there's room in the header for a
> version number, though by sticking to the principle of only 
> transferring
> architectural state we've avoided the need to use it so far).  But
> they're just dropped into the stream after the memory pages and before
> xend glues on the qemu record.

Semi-related, does this mean that HVM live migrate
is fully-compatible between versions of Xen, e.g.
either way between a Xen 3.1 machine and a Xen 3.4
machine and back again, even though new fields
(TSS? EPT?) are in the save record?

Thanks,
Dan

Re: save image file format? and [RFC] tmem save/restore/migrate

[Keir Fraser]

On 19/06/2009 00:25, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

>> BTW, the HVM save records supplied by Xen to libxc _are_
>> self-identifying and extensible (and there's room in the header for a
>> version number, though by sticking to the principle of only
>> transferring
>> architectural state we've avoided the need to use it so far).  But
>> they're just dropped into the stream after the memory pages and before
>> xend glues on the qemu record.
> 
> Semi-related, does this mean that HVM live migrate
> is fully-compatible between versions of Xen, e.g.
> either way between a Xen 3.1 machine and a Xen 3.4
> machine and back again, even though new fields
> (TSS? EPT?) are in the save record?

No, that would be kind of hard since there may be new virtual hardware and
other interfaces we expose in newer versions of Xen which simply do not
exist in older versions. We only support migration to the same or newer
versions of Xen.

 -- Keir

Re: save image file format? and [RFC] tmem save/restore/migrate

[Stefano Stabellini]

Dan Magenheimer wrote:

> Any pointers or suggestions welcome, especially any thoughts
> on changes that might be required above libxc such as in
> python code or (heaven forbid) ioemu/qemu.
> 


The qemu record is saved by savevm.c:qemu_savevm_state that calls all
the functions registered with register_savevm by the emulated devices.