[RFC] KVM guest page swap out support - take2

[RFC] KVM guest page swap out support - take2

[Shaohua Li]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

Hi,
This is a updated patch set of swap out kvm guest pages.Changes are:
1. refresh against to kvm-26
2. clean up shadow page handling to make gfn_to_page not be called
within kmap_atomic/kunamp_atomic.
3. variant bug fixing. Now the patch is quite stable in my test.
4. make swap out optional. A compile option can completely close swap
out. There should be no performance impact if swapout isn't enabled.

There are still a lot of unresolved issues:
1. just work for one vm. waiting for kvm hook into scheduler
2. Not work for mp guest and can't swap out pages which shadow page
table point to, this require kvm .flush_tlb can send ipi to vcpu.
3. swapoff isn't supported. We need a hook to sys_swapoff for kvm.

Please review, suggests and comments are welcome.

Thanks,
Shaohua

[-- Attachment #2: export-symbol.patch --]
[-- Type: text/x-patch, Size: 2109 bytes --]

symbols swapout required

Index: 2.6.21-rc7/mm/swap_state.c
===================================================================
--- 2.6.21-rc7.orig/mm/swap_state.c	2007-04-24 02:20:00.000000000 +0800
+++ 2.6.21-rc7/mm/swap_state.c	2007-05-21 10:10:20.000000000 +0800
@@ -207,6 +207,7 @@ void delete_from_swap_cache(struct page 
 	swap_free(entry);
 	page_cache_release(page);
 }
+EXPORT_SYMBOL(delete_from_swap_cache);
 
 /*
  * Strange swizzling function only for use by shmem_writepage
@@ -225,6 +226,7 @@ int move_to_swap_cache(struct page *page
 		INC_CACHE_INFO(exist_race);
 	return err;
 }
+EXPORT_SYMBOL(move_to_swap_cache);
 
 /*
  * Strange swizzling function for shmem_getpage (and shmem_unuse)
@@ -307,6 +309,7 @@ struct page * lookup_swap_cache(swp_entr
 	INC_CACHE_INFO(find_total);
 	return page;
 }
+EXPORT_SYMBOL(lookup_swap_cache);
 
 /* 
  * Locate a page of swap in physical memory, reserving swap cache space
@@ -364,3 +367,4 @@ struct page *read_swap_cache_async(swp_e
 		page_cache_release(new_page);
 	return found_page;
 }
+EXPORT_SYMBOL(read_swap_cache_async);
Index: 2.6.21-rc7/mm/swapfile.c
===================================================================
--- 2.6.21-rc7.orig/mm/swapfile.c	2007-04-24 02:20:00.000000000 +0800
+++ 2.6.21-rc7/mm/swapfile.c	2007-05-21 10:10:20.000000000 +0800
@@ -211,6 +211,7 @@ noswap:
 	spin_unlock(&swap_lock);
 	return (swp_entry_t) {0};
 }
+EXPORT_SYMBOL(get_swap_page);
 
 swp_entry_t get_swap_page_of_type(int type)
 {
@@ -303,6 +304,7 @@ void swap_free(swp_entry_t entry)
 		spin_unlock(&swap_lock);
 	}
 }
+EXPORT_SYMBOL(swap_free);
 
 /*
  * How many references to page are currently swapped out?
Index: 2.6.21-rc7/mm/filemap.c
===================================================================
--- 2.6.21-rc7.orig/mm/filemap.c	2007-04-24 02:20:00.000000000 +0800
+++ 2.6.21-rc7/mm/filemap.c	2007-05-21 10:11:09.000000000 +0800
@@ -465,6 +465,7 @@ int add_to_page_cache_lru(struct page *p
 		lru_cache_add(page);
 	return ret;
 }
+EXPORT_SYMBOL(add_to_page_cache_lru);
 
 #ifdef CONFIG_NUMA
 struct page *__page_cache_alloc(gfp_t gfp)

[-- Attachment #3: kvm-swap.tgz --]
[-- Type: application/x-compressed-tar, Size: 8742 bytes --]

[-- Attachment #4: Type: text/plain, Size: 286 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[-- Attachment #5: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [RFC] KVM guest page swap out support - take2

[Avi Kivity]

Shaohua Li wrote:
> Hi,
> This is a updated patch set of swap out kvm guest pages.Changes are:
> 1. refresh against to kvm-26
> 2. clean up shadow page handling to make gfn_to_page not be called
> within kmap_atomic/kunamp_atomic.
> 3. variant bug fixing. Now the patch is quite stable in my test.
> 4. make swap out optional. A compile option can completely close swap
> out. There should be no performance impact if swapout isn't enabled.
>   

What is the performance impact if swapout is enabled, but there is no 
memory pressure?  I prefer to have this enabled unconditionally (well, 
dependent on CONFIG_SWAP), but have a tunable to control the swapper 
aggressiveness (like /proc/sys/vm/swappiness).


> There are still a lot of unresolved issues:
> 1. just work for one vm. waiting for kvm hook into scheduler
>   

I will implement this soon.

> 2. Not work for mp guest and can't swap out pages which shadow page
> table point to, this require kvm .flush_tlb can send ipi to vcpu.
>   

This, too, will be added soon.  This is part of the apic patchset.

> 3. swapoff isn't supported. We need a hook to sys_swapoff for kvm.
>
> Please review, suggests and comments are welcome.
>   

I'll review it later today.  Meanwhile, it would help if the patches 
were posted inline instead of in a tarball.

I'm also interested in what the s390 folk have to say.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: [RFC] KVM guest page swap out support - take2

[Avi Kivity]

Shaohua Li wrote:
> Hi,
> This is a updated patch set of swap out kvm guest pages.Changes are:
> 1. refresh against to kvm-26
> 2. clean up shadow page handling to make gfn_to_page not be called
> within kmap_atomic/kunamp_atomic.
> 3. variant bug fixing. Now the patch is quite stable in my test.
> 4. make swap out optional. A compile option can completely close swap
> out. There should be no performance impact if swapout isn't enabled.
>
> There are still a lot of unresolved issues:
> 1. just work for one vm. waiting for kvm hook into scheduler
> 2. Not work for mp guest and can't swap out pages which shadow page
> table point to, this require kvm .flush_tlb can send ipi to vcpu.
> 3. swapoff isn't supported. We need a hook to sys_swapoff for kvm.
>
> Please review, suggests and comments are welcome.
>
>   

The patch rmaps read-only guest pages.  While this is of course 
required, it will make unshadowing ptes pointing to shared library pages 
very expensive.  If we have a guest with 200 processes, all mapping 
glibc, then an exit() will have to walk a linear list of size 200 per 
shadowed pte.

Do you know how the Linux rmap implementation handles this?

An idea that comes to mind is to allocate an extra mm_struct for the 
guest and use the normal Linux vma management there.  I don't know how 
practical this is.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: [RFC] KVM guest page swap out support - take2

[Shaohua Li]

On Tue, 2007-06-05 at 22:17 +0800, Avi Kivity wrote:
> Shaohua Li wrote:
> > Hi,
> > This is a updated patch set of swap out kvm guest pages.Changes are:
> > 1. refresh against to kvm-26
> > 2. clean up shadow page handling to make gfn_to_page not be called
> > within kmap_atomic/kunamp_atomic.
> > 3. variant bug fixing. Now the patch is quite stable in my test.
> > 4. make swap out optional. A compile option can completely close
> swap
> > out. There should be no performance impact if swapout isn't enabled.
> >
> > There are still a lot of unresolved issues:
> > 1. just work for one vm. waiting for kvm hook into scheduler
> > 2. Not work for mp guest and can't swap out pages which shadow page
> > table point to, this require kvm .flush_tlb can send ipi to vcpu.
> > 3. swapoff isn't supported. We need a hook to sys_swapoff for kvm.
> >
> > Please review, suggests and comments are welcome.
> >
> >  
> 
> The patch rmaps read-only guest pages.  While this is of course
> required, it will make unshadowing ptes pointing to shared library
> pages
> very expensive.  If we have a guest with 200 processes, all mapping
> glibc, then an exit() will have to walk a linear list of size 200 per
> shadowed pte.
I'm doing performance test currently, will update you soon.
> Do you know how the Linux rmap implementation handles this?
In Linux, you can get the address_space from the page, then get all vmas
of the address_space, then walking the page tables of all the vmas.It's
also a linear list walking, I think.

Thanks,
Shaohua

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: [RFC] KVM guest page swap out support - take2

[Shaohua Li]

On Wed, 2007-06-06 at 09:18 +0800, Shaohua Li wrote:
> On Tue, 2007-06-05 at 22:17 +0800, Avi Kivity wrote:
> > Shaohua Li wrote:
> > > Hi,
> > > This is a updated patch set of swap out kvm guest pages.Changes are:
> > > 1. refresh against to kvm-26
> > > 2. clean up shadow page handling to make gfn_to_page not be called
> > > within kmap_atomic/kunamp_atomic.
> > > 3. variant bug fixing. Now the patch is quite stable in my test.
> > > 4. make swap out optional. A compile option can completely close
> > swap
> > > out. There should be no performance impact if swapout isn't enabled.
> > >
> > > There are still a lot of unresolved issues:
> > > 1. just work for one vm. waiting for kvm hook into scheduler
> > > 2. Not work for mp guest and can't swap out pages which shadow page
> > > table point to, this require kvm .flush_tlb can send ipi to vcpu.
> > > 3. swapoff isn't supported. We need a hook to sys_swapoff for kvm.
> > >
> > > Please review, suggests and comments are welcome.
> > >
> > >  
> > 
> > The patch rmaps read-only guest pages.  While this is of course
> > required, it will make unshadowing ptes pointing to shared library
> > pages
> > very expensive.  If we have a guest with 200 processes, all mapping
> > glibc, then an exit() will have to walk a linear list of size 200 per
> > shadowed pte.
> I'm doing performance test currently, will update you soon.
> > Do you know how the Linux rmap implementation handles this?
> In Linux, you can get the address_space from the page, then get all vmas
> of the address_space, then walking the page tables of all the vmas.It's
> also a linear list walking, I think.
I had some basic test results. The test is under a 1G memory host and
runs a 256M memory guest. There are nearly no swap. the benchmark is to
measure the time compile kernel source. With swap enabled, the
performance degrades about 9%. rmap read-only guest page hasn't any
performance impact. The degradation is caused by 'find_get_page', it
takes a lot of time, looks like it didn't handle big file well, but I
need time to do further check.

Thanks,
Shaohua

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: [RFC] KVM guest page swap out support - take2

[Shaohua Li]

On Thu, 2007-06-07 at 17:10 +0800, Shaohua Li wrote:
> On Wed, 2007-06-06 at 09:18 +0800, Shaohua Li wrote:
> > On Tue, 2007-06-05 at 22:17 +0800, Avi Kivity wrote:
> > > Shaohua Li wrote:
> > > > Hi,
> > > > This is a updated patch set of swap out kvm guest pages.Changes are:
> > > > 1. refresh against to kvm-26
> > > > 2. clean up shadow page handling to make gfn_to_page not be called
> > > > within kmap_atomic/kunamp_atomic.
> > > > 3. variant bug fixing. Now the patch is quite stable in my test.
> > > > 4. make swap out optional. A compile option can completely close
> > > swap
> > > > out. There should be no performance impact if swapout isn't enabled.
> > > >
> > > > There are still a lot of unresolved issues:
> > > > 1. just work for one vm. waiting for kvm hook into scheduler
> > > > 2. Not work for mp guest and can't swap out pages which shadow page
> > > > table point to, this require kvm .flush_tlb can send ipi to vcpu.
> > > > 3. swapoff isn't supported. We need a hook to sys_swapoff for kvm.
> > > >
> > > > Please review, suggests and comments are welcome.
> > > >
> > > >  
> > > 
> > > The patch rmaps read-only guest pages.  While this is of course
> > > required, it will make unshadowing ptes pointing to shared library
> > > pages
> > > very expensive.  If we have a guest with 200 processes, all mapping
> > > glibc, then an exit() will have to walk a linear list of size 200 per
> > > shadowed pte.
> > I'm doing performance test currently, will update you soon.
> > > Do you know how the Linux rmap implementation handles this?
> > In Linux, you can get the address_space from the page, then get all vmas
> > of the address_space, then walking the page tables of all the vmas.It's
> > also a linear list walking, I think.
> I had some basic test results. The test is under a 1G memory host and
> runs a 256M memory guest. There are nearly no swap. the benchmark is to
> measure the time compile kernel source. With swap enabled, the
> performance degrades about 9%. rmap read-only guest page hasn't any
> performance impact. The degradation is caused by 'find_get_page', it
> takes a lot of time, looks like it didn't handle big file well, but I
> need time to do further check.
I did a small optimization to bypass find_get_page if the page is
already in page cache. With it, the swap patch set hasn't any
performance degradation in the test environment above.

Thanks,
Shaohua

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/