VM Migration on a NUMA server?

VM Migration on a NUMA server?

[Kun Cheng]

[-- Attachment #1.1: Type: text/plain, Size: 698 bytes --]

Hi all,

I'm sorry for taking your time and I'd like to make an enquery about the
status of VM migration support on a NUMA server. Currently It looks like
when a vm is migrated only its vcpus are moved to the other node but not
its memory. So, is anyone trying to fix that issue? If I want to do it
myself, it seems like two major problems are ahead of me:

1) How to specify the target node for memory migration? I'll be grateful if
anyone can give me  some hints.

2) Memory Migration. Looks like it can be done by leveraging the existing
migration related functions on Xen.

I don't have much practical experience on NUMA machines. And I'll be
appreciated for any reply & help.

Thanks,
Kenneth

[-- Attachment #1.2: Type: text/html, Size: 873 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Kun Cheng]

[-- Attachment #1.1: Type: text/plain, Size: 970 bytes --]

Let me be more specific, such idea is for the NUMA scheduling as currently
xen only migrate vcpus but leave memory at the previous node.

Kun Cheng <chengkunck@gmail.com>于2015年7月31日 周五10:32写道：

> Hi all,
>
> I'm sorry for taking your time and I'd like to make an enquery about the
> status of VM migration support on a NUMA server. Currently It looks like
> when a vm is migrated only its vcpus are moved to the other node but not
> its memory. So, is anyone trying to fix that issue? If I want to do it
> myself, it seems like two major problems are ahead of me:
>
> 1) How to specify the target node for memory migration? I'll be grateful
> if anyone can give me  some hints.
>
> 2) Memory Migration. Looks like it can be done by leveraging the existing
> migration related functions on Xen.
>
> I don't have much practical experience on NUMA machines. And I'll be
> appreciated for any reply & help.
>
> Thanks,
> Kenneth
>

[-- Attachment #1.2: Type: text/html, Size: 1315 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Wei Liu]

On Fri, Jul 31, 2015 at 11:41:11AM +0000, Kun Cheng wrote:
> Let me be more specific, such idea is for the NUMA scheduling as currently
> xen only migrate vcpus but leave memory at the previous node.
> 

Xen doesn't support memory migration at the moment. However Xen scheduler
won't migrate vcpus out of its NUMA node in most cases.

I will let Dario (CC'ed) confirm. He might also able to give you
pointers if you want to work on such feature.

Wei.

> Kun Cheng <chengkunck@gmail.com>于2015年7月31日 周五10:32写道：
> 
> > Hi all,
> >
> > I'm sorry for taking your time and I'd like to make an enquery about the
> > status of VM migration support on a NUMA server. Currently It looks like
> > when a vm is migrated only its vcpus are moved to the other node but not
> > its memory. So, is anyone trying to fix that issue? If I want to do it
> > myself, it seems like two major problems are ahead of me:
> >
> > 1) How to specify the target node for memory migration? I'll be grateful
> > if anyone can give me  some hints.
> >
> > 2) Memory Migration. Looks like it can be done by leveraging the existing
> > migration related functions on Xen.
> >
> > I don't have much practical experience on NUMA machines. And I'll be
> > appreciated for any reply & help.
> >
> > Thanks,
> > Kenneth
> >

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3948 bytes --]

On Fri, 2015-07-31 at 02:32 +0000, Kun Cheng wrote:
> Hi all,
>
Hi,
> 
> I'm sorry for taking your time and I'd like to make an enquery about
> the status of VM migration support on a NUMA server. 
>
Status is: it's not there, and won't happen soon. I've started working
on it, but then got preempted by other issues, and concentrated on
making Xen do the best possible _without_ moving the memory (e.g., with
NUMA-aware scheduling, now achieved through per-vcpu soft affinities).

Moving memory around is really really tricky. It's probably at least
doable for HVM guests, while, for PV, I'm not even so sure it can be
done! :-/

> Currently It looks like when a vm is migrated only its vcpus are moved
> to the other node but not its memory. So, is anyone trying to fix that
> issue? 
>
What do you mean with "when a vm is migrated"? If soft affinity for a VM
is specified in the config file (or afterwards, but I'd recommend to do
it in the config file, if you're interested in NUMA effects), memory is
allocated from the NUMA node that such affinity spans, and the Xen
scheduler (provided you're using Credit1, our default scheduler), will
try as hard as it can to schedule the vcpus of the vm on the pcpus of
that same node (or set of nodes).

If it's not possible, because all those pcpus are busy, the vcpus are
allowed to run on some other pcpu, outside of the NUMA node(s) the vm
has affinity with, on the basis that some execution, even with slow
memory access, is better than no execution at all.

If you're interested in having the vcpus of the vm _only_ running on the
pcpus of the node to which the memory is attached, I'd suggest using
hard affinity, instead than soft (still specifying it in the config
file).

Support for soft affinity in Credit2 is being worked on. For other
schedulers (ARINC and RTDS), it's not that useful.

> If I want to do it myself, it seems like two major problems are ahead
> of me:
>  
> 1) How to specify the target node for memory migration? I'll be
> grateful if anyone can give me  some hints.
>
Not sure I get. In my mind, if we will have this in place at some point,
migration will happen, either:
 - automatically, upon load balancing considerations
 - manually, with dedicated libxl interfaces and xl command

at that point, for the latter case, there will be a way of specifying a
target node, and that will most likely be an integer, or a list of
integers...

> 2) Memory Migration. Looks like it can be done by leveraging the
> existing migration related functions on Xen.
>
Mmmm... Maybe I see what you mean now. So, you want to perform a local
migration, and use that as a way of actually moving the guest to another
node, is this correct? If yes, it did work, last time I checked.

If doing this like that, it's true that you don't have any way for
specifying a target node. Therefore, what happens is, either:
 - if no soft or hard affinity is specified in the config file, the
   automatic NUMA placement code will run, and it most likely will
   choose a different node for the target vm, but not in a way that you
   can control easily
 - if any affinity is set, the vm will be re-created in the same exact 
   node.

That is why, a way to workaround this, and actually use local migration
as a memory-migration mechanism, is to leverage `xl config-update'. In
fact, you can do as follows:

# xl create vm.cfg 'cpus_soft="node:1'"
# xl config-update <domid> 'cpus_soft="node:0"'
# <do a local migration>

As I said, this all worked last time I tried... Is it not working for
you? Or was it something else you were after?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1114 bytes --]

On Fri, 2015-07-31 at 13:10 +0100, Wei Liu wrote:
> On Fri, Jul 31, 2015 at 11:41:11AM +0000, Kun Cheng wrote:
> > Let me be more specific, such idea is for the NUMA scheduling as currently
> > xen only migrate vcpus but leave memory at the previous node.
> > 
> 
> Xen doesn't support memory migration at the moment. However Xen scheduler
> won't migrate vcpus out of its NUMA node in most cases.
> 
Yes, as I said in my email, if soft affinity is used (and that's the
default, if anything contradicting that is specified in the confing
file) the vcpus are allowed to run outside of the NUMA node where the
memory has been allocated, but they'll try not to do so.

> I will let Dario (CC'ed) confirm. He might also able to give you
> pointers if you want to work on such feature.
> 
Thanks for the ping on this! :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Kun Cheng]

[-- Attachment #1.1: Type: text/plain, Size: 5613 bytes --]

Thank you for such a detailed reply!
Currently what's in my mind is to let memory also be migrated when a vcpu
or a vm has to be migrated to some other node. In that case let's suppose
there would be a simple load balancing mechanism. And I think a NUMA
machine can be some sort of seen as small symmetric 'machines' connected
together. So in my mind I think migrating both vcpu and memory will be
similar to a live migration. Yes that's the 'local migration' in my mind.
But as most time I am in Xen4.1.2, I think I need to learn about the cpu
affinity now and perhaps disscus with you later, if you're convinent. (But
I'm still wondering if it's posdible to add some code to specify the target
node....)

A local migration will be the best if we have a way to specify the target
node. However my plan B is to follow the migration procedures to handle the
memory . In such way maybe I could be able to bypass some tricky part. :)

And if we see it a little further, migrating all memory pages is not
economic as (supposing that) the load is heavy and perhaps a short time
little you have to move a vcpu again. Then given the locality theory maybe
we should only migrate the recent used pages and then others if needed.
However that's my illusions as there's no base for the whole building
now.LOL.

Thank you, I'm really appreciated.

(And it's really weird to type on a ipad screen)

Dario Faggioli <dario.faggioli@citrix.com>于2015年7月31日 周五20:46写道：

> On Fri, 2015-07-31 at 02:32 +0000, Kun Cheng wrote:
> > Hi all,
> >
> Hi,
> >
> > I'm sorry for taking your time and I'd like to make an enquery about
> > the status of VM migration support on a NUMA server.
> >
> Status is: it's not there, and won't happen soon. I've started working
> on it, but then got preempted by other issues, and concentrated on
> making Xen do the best possible _without_ moving the memory (e.g., with
> NUMA-aware scheduling, now achieved through per-vcpu soft affinities).
>
> Moving memory around is really really tricky. It's probably at least
> doable for HVM guests, while, for PV, I'm not even so sure it can be
> done! :-/
>
> > Currently It looks like when a vm is migrated only its vcpus are moved
> > to the other node but not its memory. So, is anyone trying to fix that
> > issue?
> >
> What do you mean with "when a vm is migrated"? If soft affinity for a VM
> is specified in the config file (or afterwards, but I'd recommend to do
> it in the config file, if you're interested in NUMA effects), memory is
> allocated from the NUMA node that such affinity spans, and the Xen
> scheduler (provided you're using Credit1, our default scheduler), will
> try as hard as it can to schedule the vcpus of the vm on the pcpus of
> that same node (or set of nodes).
>
> If it's not possible, because all those pcpus are busy, the vcpus are
> allowed to run on some other pcpu, outside of the NUMA node(s) the vm
> has affinity with, on the basis that some execution, even with slow
> memory access, is better than no execution at all.
>
> If you're interested in having the vcpus of the vm _only_ running on the
> pcpus of the node to which the memory is attached, I'd suggest using
> hard affinity, instead than soft (still specifying it in the config
> file).
>
> Support for soft affinity in Credit2 is being worked on. For other
> schedulers (ARINC and RTDS), it's not that useful.
>
> > If I want to do it myself, it seems like two major problems are ahead
> > of me:
> >
> > 1) How to specify the target node for memory migration? I'll be
> > grateful if anyone can give me  some hints.
> >
> Not sure I get. In my mind, if we will have this in place at some point,
> migration will happen, either:
>  - automatically, upon load balancing considerations
>  - manually, with dedicated libxl interfaces and xl command
>
> at that point, for the latter case, there will be a way of specifying a
> target node, and that will most likely be an integer, or a list of
> integers...
>
> > 2) Memory Migration. Looks like it can be done by leveraging the
> > existing migration related functions on Xen.
> >
> Mmmm... Maybe I see what you mean now. So, you want to perform a local
> migration, and use that as a way of actually moving the guest to another
> node, is this correct? If yes, it did work, last time I checked.
>
> If doing this like that, it's true that you don't have any way for
> specifying a target node. Therefore, what happens is, either:
>  - if no soft or hard affinity is specified in the config file, the
>    automatic NUMA placement code will run, and it most likely will
>    choose a different node for the target vm, but not in a way that you
>    can control easily
>  - if any affinity is set, the vm will be re-created in the same exact
>    node.
>
> That is why, a way to workaround this, and actually use local migration
> as a memory-migration mechanism, is to leverage `xl config-update'. In
> fact, you can do as follows:
>
> # xl create vm.cfg 'cpus_soft="node:1'"
> # xl config-update <domid> 'cpus_soft="node:0"'
> # <do a local migration>
>
> As I said, this all worked last time I tried... Is it not working for
> you? Or was it something else you were after?
>
> Regards,
> Dario
>
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>

[-- Attachment #1.2: Type: text/html, Size: 6394 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Kun Cheng]

[-- Attachment #1.1: Type: text/plain, Size: 1594 bytes --]

I've looked into those settings and vcpu affinities on Xen wiki page.
However what I'm trying to argue about is memory should be migrated when
vcpus are moved to another node. But setting a vcpu's affinity only seems
to allow vcpu migration  between different nodes. As my NUMA server is on
the way I cannot verify that. Anyhow, moving memory really draws my
interest now.

On Fri, Jul 31, 2015 at 8:50 PM Dario Faggioli <dario.faggioli@citrix.com>
wrote:

> On Fri, 2015-07-31 at 13:10 +0100, Wei Liu wrote:
> > On Fri, Jul 31, 2015 at 11:41:11AM +0000, Kun Cheng wrote:
> > > Let me be more specific, such idea is for the NUMA scheduling as
> currently
> > > xen only migrate vcpus but leave memory at the previous node.
> > >
> >
> > Xen doesn't support memory migration at the moment. However Xen scheduler
> > won't migrate vcpus out of its NUMA node in most cases.
> >
> Yes, as I said in my email, if soft affinity is used (and that's the
> default, if anything contradicting that is specified in the confing
> file) the vcpus are allowed to run outside of the NUMA node where the
> memory has been allocated, but they'll try not to do so.
>
> > I will let Dario (CC'ed) confirm. He might also able to give you
> > pointers if you want to work on such feature.
> >
> Thanks for the ping on this! :-)
>
> Regards,
> Dario
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>

[-- Attachment #1.2: Type: text/html, Size: 2099 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1639 bytes --]

On Sat, 2015-08-01 at 06:21 +0000, Kun Cheng wrote:
> I've looked into those settings and vcpu affinities on Xen wiki page.
> However what I'm trying to argue about is memory should be migrated
> when vcpus are moved to another node. 
>
I still struggle a bit to understand what you mean. It's exactly what I
said (or, at least, what I tried to do) that memory is *not* moved right
now.

There's no mean to do that, and putting one together is something really
difficult. There probably would be benefits of having it in place, but
only if it's implemented in the proper way, and applied with the proper
policing, which all should be thought, discussed and implemented.

> But setting a vcpu's affinity only seems to allow vcpu migration
> between different nodes. As my NUMA server is on the way I cannot
> verify that. 
>
Well, that is correct, memory is not moved. The local migration trick,
which I described in my previous email, actually works "only" because
migrating a guest basically means re-creating it from scratch (although,
of course, the "new" guest resumes from where the "old" was when
migration started), and hence re-allocate all it's memory, rather than
moving it.

> Anyhow, moving memory really draws my interest now. 
>
Glad to hear that. Feel free to give it a try, and report here what you
find out. :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Kun Cheng]

[-- Attachment #1.1: Type: text/plain, Size: 3535 bytes --]

On Mon, Aug 3, 2015 at 6:10 PM Dario Faggioli <dario.faggioli@citrix.com>
wrote:

> On Sat, 2015-08-01 at 06:21 +0000, Kun Cheng wrote:
> > I've looked into those settings and vcpu affinities on Xen wiki page.
> > However what I'm trying to argue about is memory should be migrated
> > when vcpus are moved to another node.
> >
> I still struggle a bit to understand what you mean. It's exactly what I
> said (or, at least, what I tried to do) that memory is *not* moved right
> now.
>
OK I get it, memory is not moved right now. What I said was discussing the
necessity of doing that.  As according to the locality theory (e.g.
programs always prone to access the data that is recently accessed) and
potential memory access delay (memory pages are on Node A but vcpus are
moved to Node B), then I suppose moving memory pages along with the vcpus
are necessary. If it can be achieved we should be able to improve the VM
execution performance, I suppose.(But probably it will introduce overhead
as moving memory is really really annoying...) .And If my words still
confuses you, please feel free to tell me which part misleads you :)


>
> There's no mean to do that, and putting one together is something really
> difficult. There probably would be benefits of having it in place, but
> only if it's implemented in the proper way, and applied with the proper
> policing, which all should be thought, discussed and implemented.
>

What do you mean by saying "There's no mean to do that"? Did the underlying
implementation (e.g. functions or other things) have no or incomplete
support for moving memory pages (I seldom explore the mechanism which deals
with memory)? Or did you mean it's too tricky & difficult to complete such
a goal?


> > But setting a vcpu's affinity only seems to allow vcpu migration
> > between different nodes. As my NUMA server is on the way I cannot
> > verify that.
> >
> Well, that is correct, memory is not moved. The local migration trick,
> which I described in my previous email, actually works "only" because
> migrating a guest basically means re-creating it from scratch (although,
> of course, the "new" guest resumes from where the "old" was when
> migration started), and hence re-allocate all it's memory, rather than
> moving it.
>

Yes 'local migration' cannot be seen as an actual 'memory migration'. But
from the VM user's view, their VMs get scheduled to another node and
relevant memory also 'appear' there so that's a fake 'move', strictly
speaking.

I'm also thinking about the plan B I mentioned. Basically that is following
the VM migration procedures to learn how to move a page. I think both share
something in common as moving a page is allocating a new one and copy old
data to it (I'm still exploring how to let the vcpu access the new pages,
remapping?), at the same time dirty pages etc. should be dealt. Am I
correct?


>
> > Anyhow, moving memory really draws my interest now.
> >
> Glad to hear that. Feel free to give it a try, and report here what you
> find out. :-)
>

I'm evaluating the feasibility now and I wish (if I'm going to do it) it
(or phase 1, maybe it should be split into multiple multiple stages to
reduce the difficulty & complexity) can be completed in 6-8 months.


>
> Regards,
> Dario
>
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>

[-- Attachment #1.2: Type: text/html, Size: 4777 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VM Migration on a NUMA server?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2496 bytes --]

On Mon, 2015-08-03 at 12:23 +0000, Kun Cheng wrote:

> On Mon, Aug 3, 2015 at 6:10 PM Dario Faggioli
> <dario.faggioli@citrix.com> wrote:
> 
>         On Sat, 2015-08-01 at 06:21 +0000, Kun Cheng wrote:
>         > I've looked into those settings and vcpu affinities on Xen
>         wiki page.
>         > However what I'm trying to argue about is memory should be
>         migrated
>         > when vcpus are moved to another node.
>         >
>         I still struggle a bit to understand what you mean. It's
>         exactly what I
>         said (or, at least, what I tried to do) that memory is *not*
>         moved right
>         now.
> OK I get it, memory is not moved right now. 
>
Right.

> What I said was discussing the necessity of doing that.  As according
> to the locality theory (e.g. programs always prone to access the data
> that is recently accessed) and potential memory access delay (memory
> pages are on Node A but vcpus are moved to Node B), then I suppose
> moving memory pages along with the vcpus are necessary. 
>
Ok, then, yes, being able to move memory would be a good thing in my
opinion too.

Of course, given the particular workload we are talking about (virtual
machines), it remains to be seen whether and how much benefit we get
from that, which is really hard to tell without being able to
try/benchmark.

For instance, locality principles applies to single programs, but,
inside a VM, there are multiple programs running, which may limit the
reliability of such principles (or require them to be adapted
accordingly).

> If it can be achieved we should be able to improve the VM execution
> performance, I suppose.(But probably it will introduce overhead as
> moving memory is really really annoying...) .
>
Exactly. So, yes, nce we are able to do it, we'll need to think to what
pages we want to move, and at how much and how often we want to move
them... not just start moving pages around like mad, just because a vcpu
happen to have been scheduled on a "foreign node" for in instance!


> And If my words still confuses you, please feel free to tell me which
> part misleads you :)
>
No, I think it's clear now.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel