What runtime states to be preserved across save / restore?

What runtime states to be preserved across save / restore?

[Wei Liu]

Hi all

I'm trying to determine what runtime states to be preserved across
save / restore.

Things that are randomly generated if not set, definitely want to
preserve:
1. guest uuid
2. mac address
3. vtpm uuid

Things might change when guest is running, and seem to be worthy of
preserving:
1. max memory size
2. target memory size
3. CDROM state

As for other things, I think using the stored configurations and let the
remote end make its own decision is sufficient.

Thoughts?

Wei.

Re: What runtime states to be preserved across save / restore?

[Ian Campbell]

On Mon, 2014-04-28 at 14:10 +0100, Wei Liu wrote:
> Hi all
> 
> I'm trying to determine what runtime states to be preserved across
> save / restore.
> 
> Things that are randomly generated if not set, definitely want to
> preserve:
> 1. guest uuid
> 2. mac address
> 3. vtpm uuid
> 
> Things might change when guest is running, and seem to be worthy of
> preserving:
> 1. max memory size
> 2. target memory size
> 3. CDROM state
> 
> As for other things, I think using the stored configurations and let the
> remote end make its own decision is sufficient.
> 
> Thoughts?

I think the above looks pretty sensible. 

Other things I can think of right now:

      * The devid of each device -- for some classes these can be
        automatically assigned, I'm not really sure if that needs
        preserving or not.
      * Hotplug of devices more generally

The selection of the block device backend is also interesting. If the
user didn't select then you want to remember that so the other end can
make its own choice about what is best, but if the user asked for
something explicit it needs preserving.

But more generally by making a baseline working version of this stuff
work at all you are also adding the moving parts which we would need to
fix any bugs which we later discover because we've not preserved
something.

Ian.

Re: What runtime states to be preserved across save / restore?

[Wei Liu]

On Mon, Apr 28, 2014 at 02:42:13PM +0100, Ian Campbell wrote:
> On Mon, 2014-04-28 at 14:10 +0100, Wei Liu wrote:
> > Hi all
> > 
> > I'm trying to determine what runtime states to be preserved across
> > save / restore.
> > 
> > Things that are randomly generated if not set, definitely want to
> > preserve:
> > 1. guest uuid
> > 2. mac address
> > 3. vtpm uuid
> > 
> > Things might change when guest is running, and seem to be worthy of
> > preserving:
> > 1. max memory size
> > 2. target memory size
> > 3. CDROM state
> > 
> > As for other things, I think using the stored configurations and let the
> > remote end make its own decision is sufficient.
> > 
> > Thoughts?
> 
> I think the above looks pretty sensible. 
> 
> Other things I can think of right now:
> 
>       * The devid of each device -- for some classes these can be
>         automatically assigned, I'm not really sure if that needs
>         preserving or not.

I don't think so. The receiving end should make its own decision.

>       * Hotplug of devices more generally
> 

Scripts? If so, they, with the things you mentioned below ...

> The selection of the block device backend is also interesting. If the
> user didn't select then you want to remember that so the other end can
> make its own choice about what is best, but if the user asked for
> something explicit it needs preserving.
> 

... are already in the config file if user ever specifies them, so
they are already preserved.

> But more generally by making a baseline working version of this stuff
> work at all you are also adding the moving parts which we would need to
> fix any bugs which we later discover because we've not preserved
> something.
> 

Sure. Then I will commence with my current list.

Wei.

> Ian.

Re: What runtime states to be preserved across save / restore?

[Ian Campbell]

On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:

> > Other things I can think of right now:
> > 
> >       * The devid of each device -- for some classes these can be
> >         automatically assigned, I'm not really sure if that needs
> >         preserving or not.
> 
> I don't think so. The receiving end should make its own decision.

Having eth0 become eth1 over migrate would be surprising though,
wouldn't it?

> >       * Hotplug of devices more generally
> > 
> 
> Scripts? If so, they, with the things you mentioned below ...

Not script, I meant e.g. xl block-attach and stuff like that.

> > The selection of the block device backend is also interesting. If the
> > user didn't select then you want to remember that so the other end can
> > make its own choice about what is best, but if the user asked for
> > something explicit it needs preserving.
> > 
> 
> ... are already in the config file if user ever specifies them, so
> they are already preserved.

Great. But the config file doesn't contain hotplugged devices I think.

Ian.

Re: What runtime states to be preserved across save / restore?

[Wei Liu]

On Mon, Apr 28, 2014 at 03:16:17PM +0100, Ian Campbell wrote:
> On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:
> 
> > > Other things I can think of right now:
> > > 
> > >       * The devid of each device -- for some classes these can be
> > >         automatically assigned, I'm not really sure if that needs
> > >         preserving or not.
> > 
> > I don't think so. The receiving end should make its own decision.
> 
> Having eth0 become eth1 over migrate would be surprising though,
> wouldn't it?
> 

I see your point. It's fairly to preserve anyway -- just read from
xenstore and update a field and that's it.

But even if we don't preserve it we're in a place no worse then where we
are before. I shall say we have no way to preserve it with current
infrastructure so I won't count it as a regression. ;-)

> > >       * Hotplug of devices more generally
> > > 
> > 
> > Scripts? If so, they, with the things you mentioned below ...
> 
> Not script, I meant e.g. xl block-attach and stuff like that.
> 

I see. This is a bit tricky though, as the relavent information is
broken down to different place or even impossible to resynthesis.

But with the nice new infrastructure we can now update the saved
configurations as we attach / detach different devices. Just more coding
is required.

Wei.

> > > The selection of the block device backend is also interesting. If the
> > > user didn't select then you want to remember that so the other end can
> > > make its own choice about what is best, but if the user asked for
> > > something explicit it needs preserving.
> > > 
> > 
> > ... are already in the config file if user ever specifies them, so
> > they are already preserved.
> 
> Great. But the config file doesn't contain hotplugged devices I think.
> 
> Ian.

Re: What runtime states to be preserved across save / restore?

[Daniel Kiper]

On Mon, Apr 28, 2014 at 03:16:17PM +0100, Ian Campbell wrote:
> On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:
>
> > > Other things I can think of right now:
> > >
> > >       * The devid of each device -- for some classes these can be
> > >         automatically assigned, I'm not really sure if that needs
> > >         preserving or not.
> >
> > I don't think so. The receiving end should make its own decision.
>
> Having eth0 become eth1 over migrate would be surprising though,
> wouldn't it?
>
> > >       * Hotplug of devices more generally
> > >
> >
> > Scripts? If so, they, with the things you mentioned below ...
>
> Not script, I meant e.g. xl block-attach and stuff like that.

Do not forget about memory hotplug and in general memory != maxmem.
This stuff is also important in migration case.

Daniel

Re: What runtime states to be preserved across save / restore?

[Wei Liu]

On Tue, Apr 29, 2014 at 09:29:10AM +0200, Daniel Kiper wrote:
> On Mon, Apr 28, 2014 at 03:16:17PM +0100, Ian Campbell wrote:
> > On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:
> >
> > > > Other things I can think of right now:
> > > >
> > > >       * The devid of each device -- for some classes these can be
> > > >         automatically assigned, I'm not really sure if that needs
> > > >         preserving or not.
> > >
> > > I don't think so. The receiving end should make its own decision.
> >
> > Having eth0 become eth1 over migrate would be surprising though,
> > wouldn't it?
> >
> > > >       * Hotplug of devices more generally
> > > >
> > >
> > > Scripts? If so, they, with the things you mentioned below ...
> >
> > Not script, I meant e.g. xl block-attach and stuff like that.
> 
> Do not forget about memory hotplug and in general memory != maxmem.
> This stuff is also important in migration case.
> 

I think max memory and target memory are easy -- I can retrieve them
from xenstore. But where can I get information about memory hotplug?
If the relevant bits are missing in libxl then I think we need another
series to address this problem.

Wei.

> Daniel

Re: What runtime states to be preserved across save / restore?

[Daniel Kiper]

On Tue, Apr 29, 2014 at 10:05:26AM +0100, Wei Liu wrote:
> On Tue, Apr 29, 2014 at 09:29:10AM +0200, Daniel Kiper wrote:
> > On Mon, Apr 28, 2014 at 03:16:17PM +0100, Ian Campbell wrote:
> > > On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:
> > >
> > > > > Other things I can think of right now:
> > > > >
> > > > >       * The devid of each device -- for some classes these can be
> > > > >         automatically assigned, I'm not really sure if that needs
> > > > >         preserving or not.
> > > >
> > > > I don't think so. The receiving end should make its own decision.
> > >
> > > Having eth0 become eth1 over migrate would be surprising though,
> > > wouldn't it?
> > >
> > > > >       * Hotplug of devices more generally
> > > > >
> > > >
> > > > Scripts? If so, they, with the things you mentioned below ...
> > >
> > > Not script, I meant e.g. xl block-attach and stuff like that.
> >
> > Do not forget about memory hotplug and in general memory != maxmem.
> > This stuff is also important in migration case.
> >
>
> I think max memory and target memory are easy -- I can retrieve them
> from xenstore. But where can I get information about memory hotplug?
> If the relevant bits are missing in libxl then I think we need another
> series to address this problem.

I think that target and max memory should be sufficient. I mentioned
about that because when I tested migration with xm a few years ago
I was not able to migrate machine which had hotplugged memory. IIRC
it happened because migration process refered to config file instead
of checking relevant values in xenstore. Acording to my knowledge save
and restore machinery is (or was) used for migration. Correct me
if I am wrong.

Daniel

Re: What runtime states to be preserved across save / restore?

[Wei Liu]

On Tue, Apr 29, 2014 at 12:40:15PM +0200, Daniel Kiper wrote:
> On Tue, Apr 29, 2014 at 10:05:26AM +0100, Wei Liu wrote:
> > On Tue, Apr 29, 2014 at 09:29:10AM +0200, Daniel Kiper wrote:
> > > On Mon, Apr 28, 2014 at 03:16:17PM +0100, Ian Campbell wrote:
> > > > On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:
> > > >
> > > > > > Other things I can think of right now:
> > > > > >
> > > > > >       * The devid of each device -- for some classes these can be
> > > > > >         automatically assigned, I'm not really sure if that needs
> > > > > >         preserving or not.
> > > > >
> > > > > I don't think so. The receiving end should make its own decision.
> > > >
> > > > Having eth0 become eth1 over migrate would be surprising though,
> > > > wouldn't it?
> > > >
> > > > > >       * Hotplug of devices more generally
> > > > > >
> > > > >
> > > > > Scripts? If so, they, with the things you mentioned below ...
> > > >
> > > > Not script, I meant e.g. xl block-attach and stuff like that.
> > >
> > > Do not forget about memory hotplug and in general memory != maxmem.
> > > This stuff is also important in migration case.
> > >
> >
> > I think max memory and target memory are easy -- I can retrieve them
> > from xenstore. But where can I get information about memory hotplug?
> > If the relevant bits are missing in libxl then I think we need another
> > series to address this problem.
> 
> I think that target and max memory should be sufficient. I mentioned

Cool!

> about that because when I tested migration with xm a few years ago
> I was not able to migrate machine which had hotplugged memory. IIRC
> it happened because migration process refered to config file instead
> of checking relevant values in xenstore. Acording to my knowledge save
> and restore machinery is (or was) used for migration. Correct me
> if I am wrong.
> 

You're right about this. ;-)

Wei.

> Daniel

Re: What runtime states to be preserved across save / restore?

[George Dunlap]

On Tue, Apr 29, 2014 at 11:42 AM, Wei Liu <wei.liu2@citrix.com> wrote:
> On Tue, Apr 29, 2014 at 12:40:15PM +0200, Daniel Kiper wrote:
>> On Tue, Apr 29, 2014 at 10:05:26AM +0100, Wei Liu wrote:
>> > On Tue, Apr 29, 2014 at 09:29:10AM +0200, Daniel Kiper wrote:
>> > > On Mon, Apr 28, 2014 at 03:16:17PM +0100, Ian Campbell wrote:
>> > > > On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:
>> > > >
>> > > > > > Other things I can think of right now:
>> > > > > >
>> > > > > >       * The devid of each device -- for some classes these can be
>> > > > > >         automatically assigned, I'm not really sure if that needs
>> > > > > >         preserving or not.
>> > > > >
>> > > > > I don't think so. The receiving end should make its own decision.
>> > > >
>> > > > Having eth0 become eth1 over migrate would be surprising though,
>> > > > wouldn't it?
>> > > >
>> > > > > >       * Hotplug of devices more generally
>> > > > > >
>> > > > >
>> > > > > Scripts? If so, they, with the things you mentioned below ...
>> > > >
>> > > > Not script, I meant e.g. xl block-attach and stuff like that.
>> > >
>> > > Do not forget about memory hotplug and in general memory != maxmem.
>> > > This stuff is also important in migration case.
>> > >
>> >
>> > I think max memory and target memory are easy -- I can retrieve them
>> > from xenstore. But where can I get information about memory hotplug?
>> > If the relevant bits are missing in libxl then I think we need another
>> > series to address this problem.
>>
>> I think that target and max memory should be sufficient. I mentioned
>
> Cool!
>
>> about that because when I tested migration with xm a few years ago
>> I was not able to migrate machine which had hotplugged memory. IIRC
>> it happened because migration process refered to config file instead
>> of checking relevant values in xenstore. Acording to my knowledge save
>> and restore machinery is (or was) used for migration. Correct me
>> if I am wrong.
>>
>
> You're right about this. ;-)

Hmm, speaking of which, you may want to check the Xen-enforced limit
as well.  Some toolstacks (such as xapi, and I believe xl) will
optionally lower a VM's memory limit in Xen, so that a VM cannot
balloon back up once it's ballooned down.  However, other toolstacks
(such as Oracle's, as I understand it) allow the VMs to balloon
themselves up if they want, and so don't set the limit.  Whatever the
limit is should probably be copied across by default.

 -George

Re: What runtime states to be preserved across save / restore?

[Wei Liu]

On Tue, May 06, 2014 at 10:32:09AM +0100, George Dunlap wrote:
> On Tue, Apr 29, 2014 at 11:42 AM, Wei Liu <wei.liu2@citrix.com> wrote:
> > On Tue, Apr 29, 2014 at 12:40:15PM +0200, Daniel Kiper wrote:
> >> On Tue, Apr 29, 2014 at 10:05:26AM +0100, Wei Liu wrote:
> >> > On Tue, Apr 29, 2014 at 09:29:10AM +0200, Daniel Kiper wrote:
> >> > > On Mon, Apr 28, 2014 at 03:16:17PM +0100, Ian Campbell wrote:
> >> > > > On Mon, 2014-04-28 at 14:57 +0100, Wei Liu wrote:
> >> > > >
> >> > > > > > Other things I can think of right now:
> >> > > > > >
> >> > > > > >       * The devid of each device -- for some classes these can be
> >> > > > > >         automatically assigned, I'm not really sure if that needs
> >> > > > > >         preserving or not.
> >> > > > >
> >> > > > > I don't think so. The receiving end should make its own decision.
> >> > > >
> >> > > > Having eth0 become eth1 over migrate would be surprising though,
> >> > > > wouldn't it?
> >> > > >
> >> > > > > >       * Hotplug of devices more generally
> >> > > > > >
> >> > > > >
> >> > > > > Scripts? If so, they, with the things you mentioned below ...
> >> > > >
> >> > > > Not script, I meant e.g. xl block-attach and stuff like that.
> >> > >
> >> > > Do not forget about memory hotplug and in general memory != maxmem.
> >> > > This stuff is also important in migration case.
> >> > >
> >> >
> >> > I think max memory and target memory are easy -- I can retrieve them
> >> > from xenstore. But where can I get information about memory hotplug?
> >> > If the relevant bits are missing in libxl then I think we need another
> >> > series to address this problem.
> >>
> >> I think that target and max memory should be sufficient. I mentioned
> >
> > Cool!
> >
> >> about that because when I tested migration with xm a few years ago
> >> I was not able to migrate machine which had hotplugged memory. IIRC
> >> it happened because migration process refered to config file instead
> >> of checking relevant values in xenstore. Acording to my knowledge save
> >> and restore machinery is (or was) used for migration. Correct me
> >> if I am wrong.
> >>
> >
> > You're right about this. ;-)
> 
> Hmm, speaking of which, you may want to check the Xen-enforced limit
> as well.  Some toolstacks (such as xapi, and I believe xl) will
> optionally lower a VM's memory limit in Xen, so that a VM cannot
> balloon back up once it's ballooned down.  However, other toolstacks
> (such as Oracle's, as I understand it) allow the VMs to balloon
> themselves up if they want, and so don't set the limit.  Whatever the
> limit is should probably be copied across by default.
> 

I grep'ed libxl code. If I'm not mistaken, the "Xen-enforced limit" is
done by setting max memory target, which is already preserved in current
scheme.

Wei.

>  -George