A migration framework for external devices

A migration framework for external devices

[Stefan Berger]

[-- Attachment #1.1: Type: text/plain, Size: 1561 bytes --]

Hello!

 As part of our off-list discussion on how to migrate the virtual TPM in 
support of virtual machine migration (live migration), we came up with the 
idea of having a migration framework that could be used for general 
migration of 'external devices' such as disk images and possibly 
serialized state of device models. I was going to look into this now and 
was wondering whether a framework like this is the right approach, 
particularly since it would exist next to XenD, which I believe is 
handling live migration ?
 To be a bit clearer on the idea of the framework: It would consist of a 
deamon running on the target machine whose different plug-ins know how to 
handle the migration of different pieces of state information and know how 
to de-serialize them (which mere 'scp' would not do). Clients on the 
source machine would communicate with that daemon and transfer the state. 
The clients would have to be triggered by XenD after a partition is not 
scheduled anymore and be given the IP address of the target machine. 
Afterwards there needs to be some synchronization on resuming the 
scheduling on the target machine after all state has been deserialized.
  The plugable deamon itself would handle the communication sockets, a 
low-level protocol which the plugins and clients would use, have support 
for timing and protocol time-outs and provide threading. The plugins would 
have to do the rest of what's necessary to communicate with the 
infrastructure and the higher-level protocol shared with the clients. 
  Comments?

  Stefan

[-- Attachment #1.2: Type: text/html, Size: 1832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: A migration framework for external devices

[Muli Ben-Yehuda]

On Wed, Feb 08, 2006 at 03:16:37PM -0500, Stefan Berger wrote:

>  To be a bit clearer on the idea of the framework: It would consist of a 
> deamon running on the target machine whose different plug-ins know how to 
> handle the migration of different pieces of state information and know how 
> to de-serialize them (which mere 'scp' would not do).

So an underlying assumption would be that the local and remove real or
virtual devices support serialize/deserialize of state, correct?

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

Re: A migration framework for external devices

[Stefan Berger]

[-- Attachment #1.1: Type: text/plain, Size: 609 bytes --]

Muli Ben-Yehuda <mulix@mulix.org> wrote on 02/08/2006 04:28:17 PM:

> On Wed, Feb 08, 2006 at 03:16:37PM -0500, Stefan Berger wrote:
> 
> >  To be a bit clearer on the idea of the framework: It would consist of 
a 
> > deamon running on the target machine whose different plug-ins know how 
to 
> > handle the migration of different pieces of state information and know 
how 
> > to de-serialize them (which mere 'scp' would not do).
> 
> So an underlying assumption would be that the local and remove real or
> virtual devices support serialize/deserialize of state, correct?

Yes, that's correct.

  Stefan

[-- Attachment #1.2: Type: text/html, Size: 813 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: A migration framework for external devices

[Mike D. Day]

Stefan Berger wrote:
>>>  To be a bit clearer on the idea of the framework: It would consist of 
> a 
>>> deamon running on the target machine whose different plug-ins know how 
> to 
>>> handle the migration of different pieces of state information and know 
> how 
>>> to de-serialize them (which mere 'scp' would not do).
>> So an underlying assumption would be that the local and remove real or
>> virtual devices support serialize/deserialize of state, correct?
> 
> Yes, that's correct.

Would also  be very cool if the framework had pre-migration hooks. For 
example, allow someone to write a plugin to verify that the target host 
is qualified and in a good state to accept a migration  before it 
actually starts.

Also, some physical devices may need to be configured/re-configured on 
the target prior to a migration (infiniband, fiber channel, iscsi) and 
it would be good to have plugins for that as well.

Mike

Re: A migration framework for external devices

[Stefan Berger]

[-- Attachment #1.1: Type: text/plain, Size: 1322 bytes --]

ncmike@us.ltcfwd.linux.ibm.com wrote on 02/08/2006 05:32:07 PM:

> Stefan Berger wrote:
> >>>  To be a bit clearer on the idea of the framework: It would consist 
of 
> > a 
> >>> deamon running on the target machine whose different plug-ins know 
how 
> > to 
> >>> handle the migration of different pieces of state information and 
know 
> > how 
> >>> to de-serialize them (which mere 'scp' would not do).
> >> So an underlying assumption would be that the local and remove real 
or
> >> virtual devices support serialize/deserialize of state, correct?
> > 
> > Yes, that's correct.
> 
> Would also  be very cool if the framework had pre-migration hooks. For 
> example, allow someone to write a plugin to verify that the target host 
> is qualified and in a good state to accept a migration  before it 
> actually starts.

It should be possible to do that as long as it is definable what 'good 
state' means.
> 
> Also, some physical devices may need to be configured/re-configured on 
> the target prior to a migration (infiniband, fiber channel, iscsi) and 
> it would be good to have plugins for that as well.

Again, this also sounds good to me. 

Another question is whether an extensible protocol exists that could be 
recycled for this purpose or we have to build something from ground up.

  Stefan

> 
> Mike

[-- Attachment #1.2: Type: text/html, Size: 1783 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: A migration framework for external devices

[Mike D. Day]

Stefan Berger wrote:
> 
> It should be possible to do that as long as it is definable what 'good 
> state' means.

Yes, good point. In this case the framework should not define policy, 
but the plugin should. In other words, the plugin could do whatever it 
wants to on the target, the framework only cares about the status 
returned by the plugin. If the plugin says "ok to proceed" then the 
migration continues.

> 
> Another question is whether an extensible protocol exists that could be 
> recycled for this purpose or we have to build something from ground up.

Unfortunately the best thing I can think of is a protocol defined using 
xml. As you mentioned, this idea includes existing bulk transfer but 
adds event and procedural semantics. xml encodings are already defined 
for everything we need including bulk data transfers, event semantics, 
and remote procedure calls. However, I don't think we should use xml 
encodings for bulk data transfer. We could keep that part as it is now.

I don't think xml would dominate the migration protocol. We could use it 
for the plugin semantics and to control the important checkpoints in pre 
and post -migration phases. But the migration itself should be encoded 
in the existing file transfer protocol.

Mike

Re: A migration framework for external devices

[Daniel Veillard]

On Thu, Feb 09, 2006 at 07:34:35AM -0500, Mike D. Day wrote:
> adds event and procedural semantics. xml encodings are already defined 
> for everything we need including bulk data transfers, event semantics, 
> and remote procedure calls. However, I don't think we should use xml 
> encodings for bulk data transfer. We could keep that part as it is now.

  XML for transporting random data is a really bad fit, you need basically
to encode it with base64 (or else), it becomes bulky, loose the stucture
aspects of XML, and gets really inefficient. 
  Another common XML related design error is to embbed XML along with
other data in a stream without markers, you end up having to precompute
the size of the XML instance which makes streaming impossible or force 
some unclean processing at the XML level (as an XML instance has no end
marker in itself, the end must be provided by the container or the code
driving the parser).
  So how do you plan to glue the XML and the other parts together ?

Daniel

-- 
Daniel Veillard      | Red Hat http://redhat.com/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: A migration framework for external devices

[Mike D. Day]

Daniel Veillard wrote:
>   Another common XML related design error is to embbed XML along with
> other data in a stream without markers, you end up having to precompute
> the size of the XML instance which makes streaming impossible or force 
> some unclean processing at the XML level (as an XML instance has no end
> marker in itself, the end must be provided by the container or the code
> driving the parser).

Yes, totally agree

>   So how do you plan to glue the XML and the other parts together ?

One way is to have two channels (like ftp). Have an xml session channel 
and a tcp data channel. That gives the benefits of xml-rpc for 
triggering events and invoking plugins. You could use xml-rpc to 
negotiate the port for the migration data transfer, for example. Then 
use the data channel (scp/tcp) for the migration.

I'm not convinced this approach is better than starting from scratch tho.

Mike

-- 
LTC Open Hypervisor Project:
1) Ensure linux has a good GPL hypervisor
2) Ensure Xen exploits xSeries Platforms
3) Provide value-add Xen Extensions for IBM customers.

Re: A migration framework for external devices

[Daniel Veillard]

On Thu, Feb 09, 2006 at 11:10:17AM -0500, Mike D. Day wrote:
> Daniel Veillard wrote:
> >  So how do you plan to glue the XML and the other parts together ?
> 
> One way is to have two channels (like ftp). Have an xml session channel 

  Urgh, way more complex, you start hitting synchronization problems
requiring more round trips etc ... Maybe the convenience still makes
sense though.

> and a tcp data channel. That gives the benefits of xml-rpc for 
> triggering events and invoking plugins. You could use xml-rpc to 
> negotiate the port for the migration data transfer, for example. Then 
> use the data channel (scp/tcp) for the migration.
> 
> I'm not convinced this approach is better than starting from scratch tho.

  I can't tell :-) I was just raising a problem which could happen in
the proposal, I still have much to read/learn on the current state...

Daniel

-- 
Daniel Veillard      | Red Hat http://redhat.com/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: A migration framework for external devices

[Stefan Berger]

[-- Attachment #1.1: Type: text/plain, Size: 2670 bytes --]

xen-devel-bounces@lists.xensource.com wrote on 02/09/2006 07:34:35 AM:

> Stefan Berger wrote:
> > 
> > It should be possible to do that as long as it is definable what 'good 

> > state' means.
> 
> Yes, good point. In this case the framework should not define policy, 
> but the plugin should. In other words, the plugin could do whatever it 
> wants to on the target, the framework only cares about the status 
> returned by the plugin. If the plugin says "ok to proceed" then the 
> migration continues.

Exactly. The framework itself should have no knowledge about the 
individual requirements of those technologies that need to be supported on 
the target system. It only would provide a communication channel, a socket 
to talk to on a system, and push all the intelligence into the plugins.

Also what the plugins would need to do is lock the system so that once the 
migration of VM A is starting no other migrated virtual machine B has 
altered the state into a contradictory state that prevents migration of A. 
I think one of the aspects of migration is to look at the target system 
first and see whether one can migrate into it, then lock it to its state 
(or do that at the same time) and then start the migration. Basically 
migration needs to be 'atomic'.

> 
> > 
> > Another question is whether an extensible protocol exists that could 
be 
> > recycled for this purpose or we have to build something from ground 
up.
> 
> Unfortunately the best thing I can think of is a protocol defined using 
> xml. As you mentioned, this idea includes existing bulk transfer but 
> adds event and procedural semantics. xml encodings are already defined 
> for everything we need including bulk data transfers, event semantics, 
> and remote procedure calls. However, I don't think we should use xml 
> encodings for bulk data transfer. We could keep that part as it is now.

I would have let the plugins implement their own protocol that suits their 
needs and the framework just provides a lower level transport protocol for 
dispatching messages to the plugins. At least that was my initial 
thinking. Not sure whether RPC is necessary here.

> 
> I don't think xml would dominate the migration protocol. We could use it 

> for the plugin semantics and to control the important checkpoints in pre 

> and post -migration phases. But the migration itself should be encoded 
> in the existing file transfer protocol.

I mixture of binary and xml-based protocol would certainly be helpful.

  Stefan
> 
> Mike
> 

> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 3250 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: A migration framework for external devices

[Mike D. Day]

Stefan Berger wrote:
> Also what the plugins would need to do is lock the system so that once the 
> migration of VM A is starting no other migrated virtual machine B has 
> altered the state into a contradictory state that prevents migration of A. 
> I think one of the aspects of migration is to look at the target system 
> first and see whether one can migrate into it, then lock it to its state 
> (or do that at the same time) and then start the migration. Basically 
> migration needs to be 'atomic'.

Yes this is the key point. With the framework in place it would be a 
nice thing to do "dry runs" on the target. That is, do everything but 
actually migrate the domain. That way you could check that all the 
devices can be configured correctly, etc. In other words, verify the 
platform ahead of time. Get a high confidence in the target before 
attempting the migration.

> 
> I would have let the plugins implement their own protocol that suits their 
> needs and the framework just provides a lower level transport protocol for 
> dispatching messages to the plugins. At least that was my initial 
> thinking. Not sure whether RPC is necessary here.

Yeah, I'm not sure either the more I consider things. If it can save 
time and work, I'm for it.

Mike

Re: A migration framework for external devices

[Anthony Liguori]

Hi Stefan,

Thanks for sending this note out.  Comments inlined.

Stefan Berger wrote:
>
> Hello!
>
>  As part of our off-list discussion on how to migrate the virtual TPM 
> in support of virtual machine migration (live migration), we came up 
> with the idea of having a migration framework that could be used for 
> general migration of 'external devices' such as disk images and 
> possibly serialized state of device models. I was going to look into 
> this now and was wondering whether a framework like this is the right 
> approach, particularly since it would exist next to XenD, which I 
> believe is handling live migration ?
I'm immediately wary of any framework for migration.  There's been talk 
(at least) of moving to a push/pull migration model which would avoid 
having to constantly listen for incoming migrations (and also add a bit 
of security too).  Any sort of framework seems like its just going to 
make those sort of changes harder.
>  To be a bit clearer on the idea of the framework: It would consist of 
> a deamon running on the target machine whose different plug-ins know 
> how to handle the migration of different pieces of state information 
> and know how to de-serialize them (which mere 'scp' would not do).
Plug-ins concern me even more as it implies a stable interface.  There's 
a lot of churning going on in Xend right now and we're just not there 
yet.  Also, I'd like to see us move away from using a daemon for 
migration anyway.

I'm assuming TPM migration requires bidirectional communication right?  
Is it static throughout the lifetime of the domain or does it change?  
How much state are we talking about migrating?

I think simple patches that introduce TPM migration would be a 
preferable start.  If we end up having a lot of code that handles 
individual device migration than we can abstract it.

Regards,

Anthony Liguori
> Clients on the source machine would communicate with that daemon and 
> transfer the state. The clients would have to be triggered by XenD 
> after a partition is not scheduled anymore and be given the IP address 
> of the target machine. Afterwards there needs to be some 
> synchronization on resuming the scheduling on the target machine after 
> all state has been deserialized.
>   The plugable deamon itself would handle the communication sockets, a 
> low-level protocol which the plugins and clients would use, have 
> support for timing and protocol time-outs and provide threading. The 
> plugins would have to do the rest of what's necessary to communicate 
> with the infrastructure and the higher-level protocol shared with the 
> clients.
>   Comments?
>
>   Stefan
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

Re: A migration framework for external devices

[Stefan Berger]

[-- Attachment #1.1: Type: text/plain, Size: 4758 bytes --]

aliguori@us.ltcfwd.linux.ibm.com wrote on 02/09/2006 10:05:37 AM:

> Hi Stefan,
> 
> Thanks for sending this note out.  Comments inlined.
> 
> Stefan Berger wrote:
> >
> > Hello!
> >
> >  As part of our off-list discussion on how to migrate the virtual TPM 
> > in support of virtual machine migration (live migration), we came up 
> > with the idea of having a migration framework that could be used for 
> > general migration of 'external devices' such as disk images and 
> > possibly serialized state of device models. I was going to look into 
> > this now and was wondering whether a framework like this is the right 
> > approach, particularly since it would exist next to XenD, which I 
> > believe is handling live migration ?
> I'm immediately wary of any framework for migration.  There's been talk 
> (at least) of moving to a push/pull migration model which would avoid 
> having to constantly listen for incoming migrations (and also add a bit 
> of security too).  Any sort of framework seems like its just going to 
> make those sort of changes harder.

The problem that I would see with a framework living outside XenD is that 
you now have two different entities taking care of migration. Certainly it 
should be one piece of code that does everything.
I don't understand your argument about a push/pull migration model. I mean 
in certain ways everything is push/pull and push is certainly what we have 
today with a command like "xm migrate <DomID> <Host>", which effectively 
pushes the vm onto another machine. What would be different in the new 
model?


> >  To be a bit clearer on the idea of the framework: It would consist of 

> > a deamon running on the target machine whose different plug-ins know 
> > how to handle the migration of different pieces of state information 
> > and know how to de-serialize them (which mere 'scp' would not do).

> Plug-ins concern me even more as it implies a stable interface.  There's 


Plug-ins  will need to exist in some form on another since knowledge is 
needed about how to migarte a specific technology and prepare the target 
system for it - and maybe check the target system first whether that 
technology is supported there or migration requirements can be met. In a 
way they do exist today with classes like 
xen/xend/{pciif,netif,blkif,usbif,tpmif}.py which are all implementing 
technology-specific code - not for migration, though. 

> a lot of churning going on in Xend right now and we're just not there 
> yet.  Also, I'd like to see us move away from using a daemon for 
> migration anyway.

It depends on what you understand under 'daemon'. The target system for 
migration will have to have something running that listens for an incoming 
migration-request. I would call that a 'daemon'.

> 
> I'm assuming TPM migration requires bidirectional communication right? 

Yes, it will.

> Is it static throughout the lifetime of the domain or does it change? 

The TPM state itself is not static throughout the lifetime of a domain. It 
does change - if that's what you mean.

> How much state are we talking about migrating?

It's not going to be much in terms of kilobytes or so, but it might end up 
being the first device that lives outside a domain an needs to be 
migrated.

> 
> I think simple patches that introduce TPM migration would be a 
> preferable start.  If we end up having a lot of code that handles 
> individual device migration than we can abstract it.

My gut feeling is that we need to design a flexible migration protocol 
that is is extensibile. So I am just looking around what other poeple 
think, although I am doing some coding as well :-).

   Stefan

> 
> Regards,
> 
> Anthony Liguori
> > Clients on the source machine would communicate with that daemon and 
> > transfer the state. The clients would have to be triggered by XenD 
> > after a partition is not scheduled anymore and be given the IP address 

> > of the target machine. Afterwards there needs to be some 
> > synchronization on resuming the scheduling on the target machine after 

> > all state has been deserialized.
> >   The plugable deamon itself would handle the communication sockets, a 

> > low-level protocol which the plugins and clients would use, have 
> > support for timing and protocol time-outs and provide threading. The 
> > plugins would have to do the rest of what's necessary to communicate 
> > with the infrastructure and the higher-level protocol shared with the 
> > clients.
> >   Comments?
> >
> >   Stefan
> > 
------------------------------------------------------------------------
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> > 
> 

[-- Attachment #1.2: Type: text/html, Size: 6042 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: A migration framework for external devices

[Anthony Liguori]

Stefan Berger wrote:
> The problem that I would see with a framework living outside XenD is 
> that you now have two different entities taking care of migration. 
> Certainly it should be one piece of code that does everything.
> I don't understand your argument about a push/pull migration model. I 
> mean in certain ways everything is push/pull and push is certainly 
> what we have today with a command like "xm migrate <DomID> <Host>", 
> which effectively pushes the vm onto another machine. What would be 
> different in the new model?
Sorry, I should have been more specific.  You still have an xm migrate 
<Dom> <Host> command, but instead of always having a daemon running on 
Host to receive the migration, it instead uses something like ssh to 
execute an "xm migrate-incoming <port>" command on the host.  Locally, 
you would use an "xm migrate-outgoing <Dom> <Host> <port>" command.

Since migration doesn't actually do anything when not migrating there's 
no point in just having an idle thread in Xend (or any idle daemon at 
all).  It also allows you to do clever things like vary the port which 
should add to the security of migration.
> Plug-ins  will need to exist in some form on another since knowledge 
> is needed about how to migarte a specific technology and prepare the 
> target system for it - and maybe check the target system first whether 
> that technology is supported there or migration requirements can be 
> met. In a way they do exist today with classes like 
> xen/xend/{pciif,netif,blkif,usbif,tpmif}.py which are all implementing 
> technology-specific code - not for migration, though.
Why do plugins have to exist?  The only reason to have a plugin 
mechanism is to be able to maintain plugins outside of the Xend tree 
which would require a stable plugin interface.  I don't think we're at a 
point where we can do that.
>
> > Is it static throughout the lifetime of the domain or does it change?  
>
> The TPM state itself is not static throughout the lifetime of a 
> domain. It does change - if that's what you mean.
>
> > How much state are we talking about migrating?
>
> It's not going to be much in terms of kilobytes or so, but it might 
> end up being the first device that lives outside a domain an needs to 
> be migrated.
How many round trips would it require?  If the data is dynamic, it has 
to be transferred (or at least finalized) during the final stage of 
migration which is performance critical.
> My gut feeling is that we need to design a flexible migration protocol 
> that is is extensibile. So I am just looking around what other poeple 
> think, although I am doing some coding as well :-).
This all sounds like it's going to add complexity.  The tools are 
already far too complex.

Regards,

Anthony Liguori
>    Stefan
>
> >
> > Regards,
> >
> > Anthony Liguori
> > > Clients on the source machine would communicate with that daemon and
> > > transfer the state. The clients would have to be triggered by XenD
> > > after a partition is not scheduled anymore and be given the IP 
> address
> > > of the target machine. Afterwards there needs to be some
> > > synchronization on resuming the scheduling on the target machine 
> after
> > > all state has been deserialized.
> > >   The plugable deamon itself would handle the communication 
> sockets, a
> > > low-level protocol which the plugins and clients would use, have
> > > support for timing and protocol time-outs and provide threading. The
> > > plugins would have to do the rest of what's necessary to communicate
> > > with the infrastructure and the higher-level protocol shared with the
> > > clients.
> > >   Comments?
> > >
> > >   Stefan
> > > 
> ------------------------------------------------------------------------
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > >  
> >

Re: A migration framework for external devices

[Stefan Berger]

[-- Attachment #1.1: Type: text/plain, Size: 4828 bytes --]

xen-devel-bounces@lists.xensource.com wrote on 02/09/2006 12:05:46 PM:

> Stefan Berger wrote:
> > The problem that I would see with a framework living outside XenD is 
> > that you now have two different entities taking care of migration. 
> > Certainly it should be one piece of code that does everything.
> > I don't understand your argument about a push/pull migration model. I 
> > mean in certain ways everything is push/pull and push is certainly 
> > what we have today with a command like "xm migrate <DomID> <Host>", 
> > which effectively pushes the vm onto another machine. What would be 
> > different in the new model?
> Sorry, I should have been more specific.  You still have an xm migrate 
> <Dom> <Host> command, but instead of always having a daemon running on 
> Host to receive the migration, it instead uses something like ssh to 
> execute an "xm migrate-incoming <port>" command on the host.  Locally, 
> you would use an "xm migrate-outgoing <Dom> <Host> <port>" command.
> 
> Since migration doesn't actually do anything when not migrating there's 
> no point in just having an idle thread in Xend (or any idle daemon at 
> all).  It also allows you to do clever things like vary the port which 
> should add to the security of migration.

I don't think that adds to security. How would you find out the current 
port number if it's a moving target? You have to either have a shared 
algorithm that calculates the current port number based on some known 
parameters or you ask a daemon/lookup service where that port currently 
is.

> > Plug-ins  will need to exist in some form on another since knowledge 
> > is needed about how to migarte a specific technology and prepare the 
> > target system for it - and maybe check the target system first whether 

> > that technology is supported there or migration requirements can be 
> > met. In a way they do exist today with classes like 
> > xen/xend/{pciif,netif,blkif,usbif,tpmif}.py which are all implementing 

> > technology-specific code - not for migration, though.
> Why do plugins have to exist?  The only reason to have a plugin 
> mechanism is to be able to maintain plugins outside of the Xend tree 
> which would require a stable plugin interface.  I don't think we're at a 

> point where we can do that.
> >
> > > Is it static throughout the lifetime of the domain or does it 
change? 
> >
> > The TPM state itself is not static throughout the lifetime of a 
> > domain. It does change - if that's what you mean.
> >
> > > How much state are we talking about migrating?
> >
> > It's not going to be much in terms of kilobytes or so, but it might 
> > end up being the first device that lives outside a domain an needs to 
> > be migrated.
> How many round trips would it require?  If the data is dynamic, it has 
> to be transferred (or at least finalized) during the final stage of 
> migration which is performance critical.

The transfer could happen in parallel of transferring the last few (dirty) 
pages. I cannot say about the number of exchanges.

> > My gut feeling is that we need to design a flexible migration protocol 

> > that is is extensibile. So I am just looking around what other poeple 
> > think, although I am doing some coding as well :-).
> This all sounds like it's going to add complexity.  The tools are 
> already far too complex.

That goes hand-in-hand with added functionality.

   Stefan

> 
> Regards,
> 
> Anthony Liguori
> >    Stefan
> >
> > >
> > > Regards,
> > >
> > > Anthony Liguori
> > > > Clients on the source machine would communicate with that daemon 
and
> > > > transfer the state. The clients would have to be triggered by XenD
> > > > after a partition is not scheduled anymore and be given the IP 
> > address
> > > > of the target machine. Afterwards there needs to be some
> > > > synchronization on resuming the scheduling on the target machine 
> > after
> > > > all state has been deserialized.
> > > >   The plugable deamon itself would handle the communication 
> > sockets, a
> > > > low-level protocol which the plugins and clients would use, have
> > > > support for timing and protocol time-outs and provide threading. 
The
> > > > plugins would have to do the rest of what's necessary to 
communicate
> > > > with the infrastructure and the higher-level protocol shared with 
the
> > > > clients.
> > > >   Comments?
> > > >
> > > >   Stefan
> > > > 
> > 
------------------------------------------------------------------------
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > > 
> > >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 6250 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: A migration framework for external devices

[Mike D. Day]

Anthony Liguori wrote:
> all).  It also allows you to do clever things like vary the port which 
> should add to the security of migration.

Allowing the target to choose the port is good practice but not added 
security.

> Why do plugins have to exist?  The only reason to have a plugin 
> mechanism is to be able to maintain plugins outside of the Xend tree 
> which would require a stable plugin interface.  I don't think we're at a 
> point where we can do that.

No, you are missing an important point. The plugin mechanism is 
necessary to isolate specialized device migration code from 
general-purpose migration code. I don't think it has anything to do with 
where the plugins are maintained.

> This all sounds like it's going to add complexity.  The tools are 
> already far too complex.

I think the complexity is already there in the usage scenarios. So this 
is not adding complexity. Rather it is dealing with complexity that 
exists. Right now the tools ignore most of the complex scenarios.

Mike

Re: A migration framework for external devices

[Anthony Liguori]

Mike D. Day wrote:
> Anthony Liguori wrote:
>> all).  It also allows you to do clever things like vary the port 
>> which should add to the security of migration.
>
> Allowing the target to choose the port is good practice but not added 
> security.
If the Xend is always listening for migrations on a well-known port, it 
is trivially easy to start migrating domains to that host.  If the port 
number isn't decided until the time of migration (and better yet, is 
decided through a secure channel like SSH), it makes it difficult to 
determine when a port is open to migrate to and which port that is.

It's not perfect, but no security mechanism is.  It's definitely better 
than what we have now as it means you have to authenticate to a host 
before you can migrate to it.
>> Why do plugins have to exist?  The only reason to have a plugin 
>> mechanism is to be able to maintain plugins outside of the Xend tree 
>> which would require a stable plugin interface.  I don't think we're 
>> at a point where we can do that.
>
> No, you are missing an important point. The plugin mechanism is 
> necessary to isolate specialized device migration code from 
> general-purpose migration code. I don't think it has anything to do 
> with where the plugins are maintained.
All devices have to have their state migrated in some form.  There's 
already code to handle that in Xend (via the S-Expression configuration 
file).  The only reason TPM migration doesn't just work is that the 
current state migration is unidirectional and TPM requires bidirectional 
state synchronization.

All that's strictly required here is the ability to transfer the TPM 
state.  This is just a little bit of additional code in XendCheckpoint 
that ran after suspend to transfer the TPM state.

You certainly don't need to have a plugin mechanism to handle TPM 
devices.  I don't know of any other devices that need this logic either 
today so it would be creating a lot of complexity with a plugin 
mechanism and extensible protocol when it's not necessary.

Regards,

Anthony Liguori

Re: A migration framework for external devices

[Mike D. Day]

Anthony Liguori wrote:

> If the Xend is always listening for migrations on a well-known port, it 
> is trivially easy to start migrating domains to that host.  If the port 
> number isn't decided until the time of migration (and better yet, is 
> decided through a secure channel like SSH), it makes it difficult to 
> determine when a port is open to migrate to and which port that is.

All you are doing with the dynamic port is making it harder for the 
novice. It is good design but not security. (How many network services 
do you know of that gain security by using variable ports?)

> All devices have to have their state migrated in some form.  There's 
> already code to handle that in Xend (via the S-Expression configuration 
> file).  The only reason TPM migration doesn't just work is that the 
> current state migration is unidirectional and TPM requires bidirectional 
> state synchronization.

I was thinking far beyond the front-end devices themselves and 
considering the physical devices on the target machine, many of which 
will be dual-ported storage, vlans, and other things that need physical 
configuration in order to support the migrated virtual devices.

That is still a missing piece in the migration solution.

> All that's strictly required here is the ability to transfer the TPM 
> state.  This is just a little bit of additional code in XendCheckpoint 
> that ran after suspend to transfer the TPM state.

Sorry, I disagree. A lot more is required for migration to be a useful 
solution.

Mike