understand KSM

understand KSM

[caiqian]

Hi everyone, while developing some tests for KSM in LTP - http://marc.info/?l=ltp-list&m=128754077917739&w=2 , noticed that pages_shared, pages_sharing and pages_unshared have different values than the expected values in the tests after read the doc. I am not sure if I misunderstood those values or there were bugs somewhere.

There are 3 programs (A, B ,C) to allocate 128M memory each using KSM.

A has memory content equal 'c'.
B has memory content equal 'a'.
C has memory content equal 'a'.

Then (using the latest mmotm tree),
pages_shared = 2
pages_sharing = 98292
pages_unshared = 0

Later,
A has memory content = 'c'
B has memory content = 'b'
C has memory content = 'a'.

Then,
pages_shared = 4
pages_sharing = 98282
pages_unshared = 0

Finally,
A has memory content = 'd'
B has memory content = 'd'
C has memory content = 'd'

Then,
pages_shared = 0
pages_sharing = 0
pages_unshared = 0

The following was the failed LTP output,

# ./ksm01 
ksm01       0  TINFO  :  KSM merging...
ksm01       0  TINFO  :  child 0 allocates 128 MB filled with 'c'.
ksm01       0  TINFO  :  child 1 allocates 128 MB filled with 'a'.
ksm01       0  TINFO  :  child 2 allocates 128 MB filled with 'a'.
ksm01       0  TINFO  :  check!
ksm01       0  TINFO  :  run is 1.
ksm01       0  TINFO  :  pages_shared is 2.
ksm01       1  TFAIL  :  pages_shared is not 32768.
ksm01       0  TINFO  :  pages_sharing is 98292.
ksm01       2  TFAIL  :  pages_sharing is not 32768.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       3  TFAIL  :  pages_unshared is not 32768.
ksm01       0  TINFO  :  child 1 continues...
ksm01       0  TINFO  :  child 1 changes memory content to 'b'.
ksm01       0  TINFO  :  check!
ksm01       0  TINFO  :  run is 1.
ksm01       0  TINFO  :  pages_shared is 4.
ksm01       4  TFAIL  :  pages_shared is not 0.
ksm01       0  TINFO  :  pages_sharing is 98282.
ksm01       5  TFAIL  :  pages_sharing is not 0.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       6  TFAIL  :  pages_unshared is not 98304.
ksm01       0  TINFO  :  child 0 continues...
ksm01       0  TINFO  :  child 0 changes memory content to 'd'.
ksm01       0  TINFO  :  child 1 continues...
ksm01       0  TINFO  :  child 1 changes memory content to 'd'
ksm01       0  TINFO  :  child 2 continues...
ksm01       0  TINFO  :  child 2 changes memory content to 'd'
ksm01       0  TINFO  :  check!
ksm01       0  TINFO  :  run is 1.
ksm01       0  TINFO  :  pages_shared is 0.
ksm01       7  TFAIL  :  pages_shared is not 32768.
ksm01       0  TINFO  :  pages_sharing is 0.
ksm01       8  TFAIL  :  pages_sharing is not 65536.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       0  TINFO  :  KSM unmerging...
ksm01       0  TINFO  :  check!
ksm01       0  TINFO  :  run is 2.
ksm01       0  TINFO  :  pages_shared is 0.
ksm01       0  TINFO  :  pages_sharing is 0.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       0  TINFO  :  stop KSM.
ksm01       0  TINFO  :  check!
ksm01       0  TINFO  :  run is 0.
ksm01       0  TINFO  :  pages_shared is 0.
ksm01       0  TINFO  :  pages_sharing is 0.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       9  TFAIL  :  ksmtest() failed with 1.

CAI Qian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: understand KSM

[Hugh Dickins]

On Mon, 25 Oct 2010, caiqian@redhat.com wrote:

> Hi everyone, while developing some tests for KSM in LTP

Thank you!

> - http://marc.info/?l=ltp-list&m=128754077917739&w=2 , noticed that pages_shared, pages_sharing and pages_unshared have different values than the expected values in the tests after read the doc. I am not sure if I misunderstood those values or there were bugs somewhere.

You were expecting KSM to share pages between processes, but you were
not expecting it to share pages within a process, which it does also.

To check the exact numbers, it would be easier if you use page-aligned
mmap() rather than byte-aligned malloc() for your MADV_MERGEABLE buffers:
some numbers are a little "off" because of part-pages at start and end.

> 
> There are 3 programs (A, B ,C) to allocate 128M memory each using KSM.
> 
> A has memory content equal 'c'.
> B has memory content equal 'a'.
> C has memory content equal 'a'.
> 
> Then (using the latest mmotm tree),
> pages_shared = 2
> pages_sharing = 98292
> pages_unshared = 0

So, after KSM has done its best, it all reduces to 1 page full of 'a's
and another 1 page full of 'c's.

> 
> Later,
> A has memory content = 'c'
> B has memory content = 'b'
> C has memory content = 'a'.
> 
> Then,
> pages_shared = 4
> pages_sharing = 98282
> pages_unshared = 0

pages_shared 3 would be the obvious: I expect the extra 1 is an artifact
of part-pages at start and end of your buffers, a page shared there too.

> 
> Finally,
> A has memory content = 'd'
> B has memory content = 'd'
> C has memory content = 'd'
> 
> Then,
> pages_shared = 0
> pages_sharing = 0
> pages_unshared = 0

The children appear to exit(1) as soon as they have filled
their buffers with 'd's, so there's nothing left to share.

Hugh

> 
> The following was the failed LTP output,
> 
> # ./ksm01 
> ksm01       0  TINFO  :  KSM merging...
> ksm01       0  TINFO  :  child 0 allocates 128 MB filled with 'c'.
> ksm01       0  TINFO  :  child 1 allocates 128 MB filled with 'a'.
> ksm01       0  TINFO  :  child 2 allocates 128 MB filled with 'a'.
> ksm01       0  TINFO  :  check!
> ksm01       0  TINFO  :  run is 1.
> ksm01       0  TINFO  :  pages_shared is 2.
> ksm01       1  TFAIL  :  pages_shared is not 32768.
> ksm01       0  TINFO  :  pages_sharing is 98292.
> ksm01       2  TFAIL  :  pages_sharing is not 32768.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       3  TFAIL  :  pages_unshared is not 32768.
> ksm01       0  TINFO  :  child 1 continues...
> ksm01       0  TINFO  :  child 1 changes memory content to 'b'.
> ksm01       0  TINFO  :  check!
> ksm01       0  TINFO  :  run is 1.
> ksm01       0  TINFO  :  pages_shared is 4.
> ksm01       4  TFAIL  :  pages_shared is not 0.
> ksm01       0  TINFO  :  pages_sharing is 98282.
> ksm01       5  TFAIL  :  pages_sharing is not 0.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       6  TFAIL  :  pages_unshared is not 98304.
> ksm01       0  TINFO  :  child 0 continues...
> ksm01       0  TINFO  :  child 0 changes memory content to 'd'.
> ksm01       0  TINFO  :  child 1 continues...
> ksm01       0  TINFO  :  child 1 changes memory content to 'd'
> ksm01       0  TINFO  :  child 2 continues...
> ksm01       0  TINFO  :  child 2 changes memory content to 'd'
> ksm01       0  TINFO  :  check!
> ksm01       0  TINFO  :  run is 1.
> ksm01       0  TINFO  :  pages_shared is 0.
> ksm01       7  TFAIL  :  pages_shared is not 32768.
> ksm01       0  TINFO  :  pages_sharing is 0.
> ksm01       8  TFAIL  :  pages_sharing is not 65536.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       0  TINFO  :  KSM unmerging...
> ksm01       0  TINFO  :  check!
> ksm01       0  TINFO  :  run is 2.
> ksm01       0  TINFO  :  pages_shared is 0.
> ksm01       0  TINFO  :  pages_sharing is 0.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       0  TINFO  :  stop KSM.
> ksm01       0  TINFO  :  check!
> ksm01       0  TINFO  :  run is 0.
> ksm01       0  TINFO  :  pages_shared is 0.
> ksm01       0  TINFO  :  pages_sharing is 0.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       9  TFAIL  :  ksmtest() failed with 1.
> 
> CAI Qian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: understand KSM

[CAI Qian]

> > There are 3 programs (A, B ,C) to allocate 128M memory each using
> KSM.
> > 
> > A has memory content equal 'c'.
> > B has memory content equal 'a'.
> > C has memory content equal 'a'.
> > 
> > Then (using the latest mmotm tree),
> > pages_shared = 2
> > pages_sharing = 98292
> > pages_unshared = 0
> 
> So, after KSM has done its best, it all reduces to 1 page full of
> 'a's and another 1 page full of 'c's.
I would expect pages_sharing to be 98302 (128 * 256 - 2), but this one looks unstable. Increased pages_to_scan to 2 * 98304 did not help either.

Thanks for the other suggestions! After modified the test accordingly, it looks like work as expected.

CAI Qian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: understand KSM

[Hugh Dickins]

On Tue, 26 Oct 2010, CAI Qian wrote:
> 
> > > There are 3 programs (A, B ,C) to allocate 128M memory each using
> > KSM.
> > > 
> > > A has memory content equal 'c'.
> > > B has memory content equal 'a'.
> > > C has memory content equal 'a'.
> > > 
> > > Then (using the latest mmotm tree),
> > > pages_shared = 2
> > > pages_sharing = 98292
> > > pages_unshared = 0
> > 
> > So, after KSM has done its best, it all reduces to 1 page full of
> > 'a's and another 1 page full of 'c's.
> I would expect pages_sharing to be 98302 (128 * 256 - 2), but this one looks unstable. Increased pages_to_scan to 2 * 98304 did not help either.

Since your 1MB malloc'ed buffers may not fall on page boundaries,
and there might occasionally be other malloc'ed areas interspersed
amongst them, I'm not surprised that pages_sharing falls a little
short of 98302.  But I am surprised that pages_unshared does not
make up the difference; probably pages_volatile does, but I don't
see why some should remain volatile indefinitely.

> 
> Thanks for the other suggestions! After modified the test accordingly, it looks like work as expected.

Oh good, that's a relief.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: understand KSM

[caiqian]

> Since your 1MB malloc'ed buffers may not fall on page boundaries,
> and there might occasionally be other malloc'ed areas interspersed
> amongst them, I'm not surprised that pages_sharing falls a little
> short of 98302.  But I am surprised that pages_unshared does not
> make up the difference; probably pages_volatile does, but I don't
> see why some should remain volatile indefinitely.
The test program (http://people.redhat.com/qcai/ksm01.c) was changed to use mmap instead of malloc, and pages_sharing was short of the expected value and pages_volatile was indeed non-zero. Those makes it is difficult to predict pages_sharing and pages_volatile although it might be fine to check pages_sharing + pages_volatile with an expected value. Any suggestion to alter the test code to check the stable numbers? Thanks.

ksm01       0  TINFO  :  child 0 allocates 128 MB filled with 'c'.
ksm01       0  TINFO  :  child 1 allocates 128 MB filled with 'a'.
ksm01       0  TINFO  :  child 2 allocates 128 MB filled with 'a'.
ksm01       0  TINFO  :  pages_shared is 2.
ksm01       0  TINFO  :  pages_sharing is 98300.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       0  TINFO  :  pages_volatile is 2.

ksm01       0  TINFO  :  child 1 changes memory content to 'b'.
ksm01       0  TINFO  :  pages_shared is 3.
ksm01       0  TINFO  :  pages_sharing is 98291.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       0  TINFO  :  pages_volatile is 10.

ksm01       0  TINFO  :  child 0 changes memory content to 'd'.
ksm01       0  TINFO  :  child 1 changes memory content to 'd'
ksm01       0  TINFO  :  child 2 changes memory content to 'd'
ksm01       0  TINFO  :  pages_shared is 1.
ksm01       0  TINFO  :  pages_sharing is 98299.
ksm01       0  TINFO  :  pages_unshared is 0.
ksm01       0  TINFO  :  pages_volatile is 4.

ksm01       0  TINFO  :  child 1 changes one page to 'e'.
ksm01       0  TINFO  :  pages_shared is 1.
ksm01       0  TINFO  :  pages_sharing is 98299.
ksm01       0  TINFO  :  pages_unshared is 1.
ksm01       0  TINFO  :  pages_volatile is 3.


CAI Qian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: understand KSM

[Hugh Dickins]

On Wed, 27 Oct 2010, caiqian@redhat.com wrote:
> > Since your 1MB malloc'ed buffers may not fall on page boundaries,
> > and there might occasionally be other malloc'ed areas interspersed
> > amongst them, I'm not surprised that pages_sharing falls a little
> > short of 98302.  But I am surprised that pages_unshared does not
> > make up the difference; probably pages_volatile does, but I don't
> > see why some should remain volatile indefinitely.
> The test program (http://people.redhat.com/qcai/ksm01.c) was changed to use mmap instead of malloc, and pages_sharing was short of the expected value and pages_volatile was indeed non-zero. Those makes it is difficult to predict pages_sharing and pages_volatile although it might be fine to check pages_sharing + pages_volatile with an expected value. Any suggestion to alter the test code to check the stable numbers? Thanks.
> 
> ksm01       0  TINFO  :  child 0 allocates 128 MB filled with 'c'.
> ksm01       0  TINFO  :  child 1 allocates 128 MB filled with 'a'.
> ksm01       0  TINFO  :  child 2 allocates 128 MB filled with 'a'.
> ksm01       0  TINFO  :  pages_shared is 2.
> ksm01       0  TINFO  :  pages_sharing is 98300.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       0  TINFO  :  pages_volatile is 2.
> 
> ksm01       0  TINFO  :  child 1 changes memory content to 'b'.
> ksm01       0  TINFO  :  pages_shared is 3.
> ksm01       0  TINFO  :  pages_sharing is 98291.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       0  TINFO  :  pages_volatile is 10.
> 
> ksm01       0  TINFO  :  child 0 changes memory content to 'd'.
> ksm01       0  TINFO  :  child 1 changes memory content to 'd'
> ksm01       0  TINFO  :  child 2 changes memory content to 'd'
> ksm01       0  TINFO  :  pages_shared is 1.
> ksm01       0  TINFO  :  pages_sharing is 98299.
> ksm01       0  TINFO  :  pages_unshared is 0.
> ksm01       0  TINFO  :  pages_volatile is 4.
> 
> ksm01       0  TINFO  :  child 1 changes one page to 'e'.
> ksm01       0  TINFO  :  pages_shared is 1.
> ksm01       0  TINFO  :  pages_sharing is 98299.
> ksm01       0  TINFO  :  pages_unshared is 1.
> ksm01       0  TINFO  :  pages_volatile is 3.

Thank you for persisting, I was surprised by that, but didn't find time
to try for myself until yesterday: yes, running your ksm01, pages_volatile
stayed non-0 for erratic periods of time, say 10 or 20 seconds.  I had to
insert more debugging to find out what and why was failing, but in the end
it was rather obvious.

Fix below, but I haven't yet signed off the patch - we usually prefer to
avoid lru_add_drain_all() (all those inter-cpu interrupts): I think this
is the natural place to call it, but I haven't quite decided yet whether
it's worth adding a few lines to limit how often we call it there.

Why did my own testing never see this?  Largely because I was using
system() to run a shell script to show the /sys/kernel/mm/ksm numbers
(whereas your ksm01.c opens and reads the files directly): there's
more than enough overhead in doing it my way to flush those pagevecs
(on one cpu, but I'm still surprised I didn't see it from other cpus).

Hugh

---

 mm/ksm.c |    9 +++++++++
 1 file changed, 9 insertions(+)

--- 2.6.37-rc1/mm/ksm.c	2010-10-20 13:30:22.000000000 -0700
+++ linux/mm/ksm.c	2010-11-07 23:49:26.000000000 -0800
@@ -1352,6 +1352,15 @@ static void ksm_do_scan(unsigned int sca
 	struct rmap_item *rmap_item;
 	struct page *uninitialized_var(page);
 
+	/*
+	 * A number of pages can hang around indefinitely on per-cpu pagevecs,
+	 * with raised page count preventing write_protect_page() from merging
+	 * them: though it doesn't really matter much, it is disturbing to see
+	 * them stuck in pages_volatile until other activity jostles them out,
+	 * and it prevents deterministic LTP success; so drain them here.
+	 */
+	lru_add_drain_all();
+
 	while (scan_npages--) {
 		cond_resched();
 		rmap_item = scan_get_next_rmap_item(&page);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: understand KSM

[CAI Qian]

>  mm/ksm.c |    9 +++++++++
>  1 file changed, 9 insertions(+)
This fixed the problem for me. Thanks.

CAI Qian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>