Re: how to move the memory of a domain from one NUMAnode to another?

Re: how to move the memory of a domain from one NUMAnode to another?

[butine]

[-- Attachment #1.1: Type: text/plain, Size: 2917 bytes --]

hello,Dario,


linux provides the function move_pages ,which can move individual pages of a process to another node,as follows:

long move_pages(int pid, unsigned long count, void **pages,
    const int *nodes, int *status, int flags);

Therefore,if xen can provide the similar approach,then we recognize the hot memory pages,and preferentially migrate the hot memory pages.Then the efficiecy of VM's migration will be very high.

And how to improve about the new version of memory migration?  

 

======= 2013-08-07 19:15:21 From your letter：=======

>On mer, 2013-08-07 at 18:55 +0800, butine@zju.edu.cn wrote:
>> hello,Dario,
>>
>Hi,
>
>> I have been studying in your patch about "memory migration",
>>
>Oh, you are? Cool... What do you think about'em? :-)
>
>I'm working on a new version of that series from quite a while now, but
>I keep being sidetracked by various things. Anyway, as of now, It
>shouldn't take much longer to be able to repost something...
>
>> how to recognize the hot memory,
>>
>And do you have any idea of that?
>
>The two options I can think of are performance counters (but they're
>coarse grained, they're not available on all archs, etc.) and 'hinting
>page faults'. That is something Linux does, i.e., it periodically
>removes the _PAGE_PRESENT bit from all the pages and, when the page
>fault occurs, it make a note of where the page is and from where it is
>being accessed (and at that point you can even count how many times,
>etc.).
>
>That being said, the point here is, would that be worth? I mean, for
>Linux is not a big deal to migrate just one (or a bunch of) page(s) at a
>time, for us here, it indeed is quite a big deal (PV guests!!). So I
>honestly don't think we can take the same approach and hence, while it
>would be useful to have some info about the above, I'm not sure it's
>terribly important.
>
>Thoughts?
>
>> and how to gurantee no downtime.
>>
>And again, did you reach any conclusion on this? :-)
>
>I do not think that, for PV guest, zero downtime is almost impossible.
>Right now I'm doing a very quick suspend-resume alike cycle. We can (in
>future) try to turn it into something even more lightweight (some kind
>of quiescence state), but you don't change the PTEs under the feet of a
>running domain! :-O
>
>> At the same time,In the migration process,the P2M need to be locked?
>>
>Well, right now I'm suspending the domain, so I really think it does
>not.
>
>Dario
>
>--
><<This happens because I choose it to happen!>> (Raistlin Majere)
>-----------------------------------------------------------------
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>http://lists.xen.org/xen-devel
>

= = = = = = = = = = = = = = = = = = = =
   

Thanks,

Regards,

Butine Huang

2013-08-08

[-- Attachment #1.2: Type: text/html, Size: 3905 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: how to move the memory of a domain from one NUMAnode to another?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1135 bytes --]

On gio, 2013-08-08 at 13:56 +0800, butine@zju.edu.cn wrote:
> linux provides the function move_pages ,which can move individual
> pages of a process to another node,as follows:
> 
> long move_pages(int pid, unsigned long count, void **pages,
>     const int *nodes, int *status, int flags);
> 
> Therefore,if xen can provide the similar approach,then we recognize
> the hot memory pages,and preferentially migrate the hot memory
> pages.Then the efficiecy of VM's migration will be very high.
> 
Yes, but unfortunately Xen can't provide something like that. It may be
possible for HVM (or, in general, autotranslated) guests, but not for PV
guests. :-(

> And how to improve about the new version of memory migration?  
> 
For now, I'm concentrating on making it work. After that, we'll see how
to make if better, faster, more efficient, etc.

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: how to move the memory of a domain from one NUMAnode to another?

[butine]

[-- Attachment #1.1: Type: text/plain, Size: 1943 bytes --]

hello，Dario

Your approach about memory migration has some problems:

(1)the domain must be suspended,and how long is the downtime(100ms or 1s)?

(2)will the memory migration affect the balloon,TMem mechanism? 

And I want to design to migrate the memory pages,the process is as follows:

(1)drop the reference to the memory pages of domain

(2)lock and copy the memory pages in the new node

(3)update the page tables,the P2M

(4)bind the VCPU to the new node 
This process will not affect the interrupt of the domain.
pls provide the suggestions,thanks!! 

======= 2013-08-08 15:56:15 from your letter：=======

>On gio, 2013-08-08 at 13:56 +0800, butine@zju.edu.cn wrote:
>> linux provides the function move_pages ,which can move individual
>> pages of a process to another node,as follows:
>>
>> long move_pages(int pid, unsigned long count, void **pages,
>>     const int *nodes, int *status, int flags);
>>
>> Therefore,if xen can provide the similar approach,then we recognize
>> the hot memory pages,and preferentially migrate the hot memory
>> pages.Then the efficiecy of VM's migration will be very high.
>>
>Yes, but unfortunately Xen can't provide something like that. It may be
>possible for HVM (or, in general, autotranslated) guests, but not for PV
>guests. :-(
>
>> And how to improve about the new version of memory migration? 
>>
>For now, I'm concentrating on making it work. After that, we'll see how
>to make if better, faster, more efficient, etc.
>
>Dario
>
>--
><<This happens because I choose it to happen!>> (Raistlin Majere)
>-----------------------------------------------------------------
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>http://lists.xen.org/xen-devel
>

= = = = = = = = = = = = = = = = = = = =
Butine Huang   

2013-08-12

[-- Attachment #1.2: Type: text/html, Size: 3436 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: how to move the memory of a domain from one NUMAnode to another?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3774 bytes --]

On lun, 2013-08-12 at 14:22 +0800, butine@zju.edu.cn wrote:
> hello，Dario
> 
Hi,

> Your approach about memory migration has some problems:
> 
It sure does. :-)

> (1)the domain must be suspended,and how long is the downtime(100ms or
> 1s)? 
> 
The downtime is supposed to be very short, and I think it really is
possible to make it very short. However, I don't have any measurements
about it right now.

That being said, I'm not aware of any approach to memory migration that
does not involve suspending the domain (at least for PV), if you have
any idea, please, share it with us. :-)

> (2)will the memory migration affect the balloon,TMem mechanism? 
> 
In general, it will "affect" them for sure... Actually, it affects
almost everything! :-O

However, I think you'd have to clarify what you mean by "affect
ballooning" and "affect Tmem". The guest will have a new node-affinity,
so it should get the memory from the new set of nodes when ballooning up
after memory migration, if that was what you where asking. Also, we have
someone working on NUMA-aware ballooning already. As soon as both the
project will be mature enough, we'll double check the interactions.

About Tmem, Dan kindly replied to my original RFC series, saying that
The Good Thing (TM) to do about Tmem in this case is _do_not_touch_ it,
and I agree and plan to try doing so.

Again, if this don't answer your question, I'm afraid you will have to
try to formulate it a bit better, because I am almost sure I am not
getting it.

> And I want to design to migrate the memory pages,the process is as
> follows:
> 
> (1)drop the reference to the memory pages of domain
> 
Ok, and does "drop the reference" also means deallocate it? If yes,
where do you want to store it temporarily? If no, how do you plan to
have the domain allocate the double of its allowed static_max amount of
memory without the entire Xen ecosystem complaining about it? Is this
something you want to do that without suspending the domain? If yes,
how's that possible for a PV guest?

> (2)lock and copy the memory pages in the new node
> 
See above regarding static_max...

> (3)update the page tables,the P2M
> 
See above regarding the need for suspending/quiesce a PV guest...

> (4)bind the VCPU to the new node 
> This process will not affect the interrupt of the domain.
>
Are you telling or asking? And in any case, what is it that you are
telling or asking?

> pls provide the suggestions,thanks!! 
>
Mmmm... Sorry, but I fail to understand again... The suggestions about
what? All in all, I don't think what you say you would like/want to
implement is much different from what I have in mind and (still
incomplete) in code.

I'm sure there is something, both in the design and in the code, from my
approach that could be improved, and if you're keen on doing that,
you're very welcome on-board. :-D

What I have been able to submit up to now, is in the RFC patches that
you've seen. What I am baking these last weeks, I will release as soon
as coming back from vacations (early in September), but it still follows
more or less the same approach (taking into account, of course, of the
comments that the RFC got).

If there is something that you think it should be done different (and
better), then it is me that am waiting for your suggestions, and believe
me, I'm very open to them, so do not hesitate to speak and/or send your
code! :-P

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel