[RFC][PATCH 0/5] Memory merging driver for Linux

[RFC][PATCH 0/5] Memory merging driver for Linux

[Izik Eidus]

when kvm is used in production servers, many times it run the same 
guests operation systems more than once
the idea of this module is to find the identical pages in diffrent 
guests and to share them so we can save memory,
due to the fact that many guests run identical operation systems, alot 
of data in the ram is equal between the guests

this module find this identical data (pages) and merge them into one 
single page
this new page is write protected so in any case the guest will try to 
write to it do_wp_page will duplicate the page

this module simply go over a list of pages that were registered, and 
find the identical pages (using hash table)
the pages that it scan are anonymous, each time that it find an 
identical pages it create a file mapped
(right now it is just kernel allocated) page that will be the shared page,

as for now i am missing swapping support (will add soon using non-linear 
vmas)
 
this module can be used for every other purpuse and work without kvm
(i used it for qemu)
to make it work for kvm, the mmu notifers sent by andrea should be used

i added 2 new functions to the kernel
one:
page_wrprotect() make the page as read only by setting the ptes point to
it as read only.
second:
replace_page() - replace the pte mapping related to vm area between two 
pages

few numbers:
for started windows i can share almost the whole memory (as it zero all 
the pages),
so i can start much much more windows guests than i have memory (as long 
as no one touch it)

for linux guests i was able to share 800mb+ for 4 centos guests that 
each had 512mb memory allocated to
(again it was without work load, and they ran X)

-- 
woof.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [RFC][PATCH 0/5] Memory merging driver for Linux

[Rik van Riel]

On Mon, 21 Jan 2008 18:05:53 +0200
Izik Eidus <izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:

> i added 2 new functions to the kernel
> one:
> page_wrprotect() make the page as read only by setting the ptes point to
> it as read only.
> second:
> replace_page() - replace the pte mapping related to vm area between two 
> pages

How will this work on CPUs with nested paging support, where the
CPU does the guest -> physical address translation?  (opposed to
having shadow page tables)

Is it sufficient to mark the page read-only in the guest->physical
translation page table?

-- 
All rights reversed.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [RFC][PATCH 0/5] Memory merging driver for Linux

[Andrea Arcangeli]

On Wed, Jan 23, 2008 at 12:05:10PM -0500, Rik van Riel wrote:
> On Mon, 21 Jan 2008 18:05:53 +0200
> Izik Eidus <izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> 
> > i added 2 new functions to the kernel
> > one:
> > page_wrprotect() make the page as read only by setting the ptes point to
> > it as read only.
> > second:
> > replace_page() - replace the pte mapping related to vm area between two 
> > pages
> 
> How will this work on CPUs with nested paging support, where the
> CPU does the guest -> physical address translation?  (opposed to
> having shadow page tables)

sptes resolve guest addresses to host physical addresses (what is
different is only which kind of guest address is being translated).

sptes are faster than nptes for non pte-mangling non-context-switching
memory intensive number crunching workloads infact. (DBMS will
appreciate ntpes instead ;)

> Is it sufficient to mark the page read-only in the guest->physical
> translation page table?

Yes, just like with sptes too. I guess ntpes will also be managed as a
tlb even if they won't require many changes, but the mmu notifier
already firing in those two calls is what will keep both sptes and
nptes in sync with the main linux VM. The serialization against
get_user_pages that refills the spte/npte layer with
nonpresent-nofault case of course happens through the PT lock, just
like for the regular linux page fault against the pte that is pte_none
for a little while but with the lock held (and set to write protect or
new value before releasing it). This infact shows how the mmu
notifiers that connects the linux pte to the spte/npte works for more
than swapping.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [RFC][PATCH 0/5] Memory merging driver for Linux

[Izik Eidus]

Andrea Arcangeli wrote:
> On Wed, Jan 23, 2008 at 12:05:10PM -0500, Rik van Riel wrote:
>   
>> On Mon, 21 Jan 2008 18:05:53 +0200
>> Izik Eidus <izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>>
>>     
>>> i added 2 new functions to the kernel
>>> one:
>>> page_wrprotect() make the page as read only by setting the ptes point to
>>> it as read only.
>>> second:
>>> replace_page() - replace the pte mapping related to vm area between two 
>>> pages
>>>       
>> How will this work on CPUs with nested paging support, where the
>> CPU does the guest -> physical address translation?  (opposed to
>> having shadow page tables)
>>     

thanks for reviewing.

nested page tables are some what diffrent from shadow page tables
instead of keeping another page table like we are doing with the shadow code
we are keeping another layer that translate the physical memory of the 
guest into the
physical memory of the host,
to this new layer we are allowed to add access permission, so we can 
mark the pages that
are shared as readonly and to vmexit on that, so it should work with that.
>
> sptes resolve guest addresses to host physical addresses (what is
> different is only which kind of guest address is being translated).
>
> sptes are faster than nptes for non pte-mangling non-context-switching
> memory intensive number crunching workloads infact. (DBMS will
> appreciate ntpes instead ;)
>
>   
>> Is it sufficient to mark the page read-only in the guest->physical
>> translation page table?
>>     
>
> Yes, just like with sptes too. I guess ntpes will also be managed as a
> tlb even if they won't require many changes, but the mmu notifier
> already firing in those two calls is what will keep both sptes and
> nptes in sync with the main linux VM. The serialization against
> get_user_pages that refills the spte/npte layer with
> nonpresent-nofault case of course happens through the PT lock, just
> like for the regular linux page fault against the pte that is pte_none
> for a little while but with the lock held (and set to write protect or
> new value before releasing it). This infact shows how the mmu
> notifiers that connects the linux pte to the spte/npte works for more
> than swapping.
>   
yea, without mmu notifiers this driver cant work safely and effective 
for kvm
it can only work for normal applications such as qemu without the mmu 
notifers.

-- 
woof.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [RFC][PATCH 0/5] Memory merging driver for Linux

[Avi Kivity]

Rik van Riel wrote:
> On Mon, 21 Jan 2008 18:05:53 +0200
> Izik Eidus <izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>
>   
>> i added 2 new functions to the kernel
>> one:
>> page_wrprotect() make the page as read only by setting the ptes point to
>> it as read only.
>> second:
>> replace_page() - replace the pte mapping related to vm area between two 
>> pages
>>     
>
> How will this work on CPUs with nested paging support, where the
> CPU does the guest -> physical address translation?  (opposed to
> having shadow page tables)
>
>   

Nested page tables are very similar to real-mode shadow paging: both 
translate guest physical addresses to host physical addreses.

In any case, the merge driver is oblivious to the paging method used, it 
works at the Linux pte level and relies on mmu notifiers to keep 
everything in sync.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [RFC][PATCH 0/5] Memory merging driver for Linux

[Chris Wright]

* Izik Eidus (izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org) wrote:
> this module find this identical data (pages) and merge them into one 
> single page
> this new page is write protected so in any case the guest will try to 
> write to it do_wp_page will duplicate the page

What happens if you've merged more pages than you can recover on write
faults?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [RFC][PATCH 0/5] Memory merging driver for Linux

[Avi Kivity]

Chris Wright wrote:
> * Izik Eidus (izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org) wrote:
>   
>> this module find this identical data (pages) and merge them into one 
>> single page
>> this new page is write protected so in any case the guest will try to 
>> write to it do_wp_page will duplicate the page
>>     
>
> What happens if you've merged more pages than you can recover on write
> faults?
>   

You start to swap.  Just like Linux when you start to write on fork()ed 
memory.

A management application may start taking measures, like inflating 
balloons and migrating to other hosts, but swapping is needed as a last 
resort measure.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [RFC][PATCH 0/5] Memory merging driver for Linux

[Izik Eidus]

Avi Kivity wrote:
> Chris Wright wrote:
>> * Izik Eidus (izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org) wrote:
>>  
>>> this module find this identical data (pages) and merge them into one 
>>> single page
>>> this new page is write protected so in any case the guest will try 
>>> to write to it do_wp_page will duplicate the page
>>>     
>>
>> What happens if you've merged more pages than you can recover on write
>> faults?
>>   
>
> You start to swap.  Just like Linux when you start to write on 
> fork()ed memory.
>
> A management application may start taking measures, like inflating 
> balloons and migrating to other hosts, but swapping is needed as a 
> last resort measure.
>
yes, write faults are getting into do_wp_page() that in turn create a 
new anonymous/swappable page
so it is safe.

-- 
woof.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/