Re: Autoregulate swappiness & inactivation

Re: Autoregulate swappiness & inactivation

[deepfire]

Andrew Morton <akpm@osdl.org> wrote at Thu, 8 Jul 2004 09:44:06 -0700:
> Con Kolivas <kernel@kolivas.org> wrote:
> >
> > Here is another try at providing feedback to tune the vm_swappiness.
> 
> I spent some time yesterday trying to demonstrate performance improvements
> from those two patches.  Using
> 
> 	make -j4 vmlinux with mem=64m
> 
> and
> 
> 	qsbench -p 4 -m 96 with mem=256m
> 
> and was not able to do so, which is what I expected.
> 
> We do need more quantitative testing on this work.

i`ve done some of my favorite under-bridge-crafted tests, so to speak.

the test looked like this:

time find / -xdev | \
bzcat --compress | bzcat --decompress | \
bzcat --compress | bzcat --decompress | \
bzcat --compress | bzcat --decompress | \
cat > /dev/null

The patches applied were Con`s autoswap + autoregulate

it was performed on a p3-600, 10krpm scsi in two following scenarios:


1. mem=48M, swap=off

2.4.20-pre9:		3m20
2.6.7-con:		4m09
2.6.7-vanilla:		4m09


2. mem=32M, swap=on

2.4.20-pre9:		5m37
2.6.7-con:		6m37
2.6.7-vanilla:		7m31



which still leaves 2.4 the leader.


regards, Samium "2.4" Gromoff

2.6.7-ck5

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

Patchset update:

These are patches designed to improve system responsiveness with 
specific emphasis on the desktop, but suitable/configurable to any 
workload. Read details and FAQ on my web page.

http://kernel.kolivas.org


Added:
cfq bugfix
- cfq-bad-allocation.fix
security updates
- 1100_ip_tables.patch
- 1105_CAN-2004-0497.patch
- 1110_proc.patch


Updated:
Staircase cpu scheduler:
- from_2.6.7_to_staircase7.9
Batch (idle) scheduling:
- schedbatch2.2.diff
Isochronous (soft real time) scheduling:
- schediso2.2.diff
Graphical boot:
- bootsplash-3.1.4-sp3-2.6.7.diff


All patches:
-from_2.6.7_to_staircase7.9
-schedrange.diff
-schedbatch2.2.diff
-schediso2.2.diff
-autoswap.diff
-vm_autoregulate2.diff
-supermount-ng204.diff
-defaultcfq.diff
-config_hz.diff
-bootsplash-3.1.4-sp3-2.6.7.diff
-cfq-bad-allocation.fix
-1100_ip_tables.patch
-1105_CAN-2004-0497.patch
-1110_proc.patch
-ck5version.diff


Please feel free to send comments, queries, suggestions, patches.
Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: 2.6.7-ck5

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Very nice, Con.  I've been using ck1 and ck3 with pax applied, and
finding performance to be exceptional.  I'll merge current 2.6.7-pax
with this and test it out right away.

When do you think the staircase, batch, and isometric scheduling will
reach mainline-quality?  Do you think you'll be ready to ask Andrew to
merge it soon, or will it be a while before it's quite ready for that?
How about autoregulated swappiness, which seems to be very efficient at
its job?

Con Kolivas wrote:
| Patchset update:
|
| These are patches designed to improve system responsiveness with
| specific emphasis on the desktop, but suitable/configurable to any
| workload. Read details and FAQ on my web page.
|
| http://kernel.kolivas.org
|

|
| Please feel free to send comments, queries, suggestions, patches.
| Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7BkuhDd4aOud5P8RAitcAJ9OvOI9LlWlujyl3JzuQazQbzV9SQCfX7m/
oYrphGbkeT89fao/n0Y3eUA=
=i8eI
-----END PGP SIGNATURE-----

Re: 2.6.7-ck5

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1190 bytes --]

John Richard Moser wrote:
> Very nice, Con.  I've been using ck1 and ck3 with pax applied, and
> finding performance to be exceptional.  I'll merge current 2.6.7-pax
> with this and test it out right away.

Great! The ck4 performance was actually substantially better than ck3 
(on the desktop) so here's hoping you enjoy ck5 which basically performs 
the same.

> 
> When do you think the staircase, batch, and isometric scheduling will
> reach mainline-quality?  Do you think you'll be ready to ask Andrew to
> merge it soon, or will it be a while before it's quite ready for that?

Well I think they're all ready for prime time now, I just dont think 
prime time is ready for it. This is too large a change for mainline 2.6 
which keeps -ck in business ;)

> How about autoregulated swappiness, which seems to be very efficient at
> its job?

It's been around for quite a while, and akpm has not expressed any 
interest in it so I think this will only ever flounder in the -ck domain.

Cheers,
Con

P.S. You seem to have preempted the arrival of my -ck5 announcement to 
lkml, as will this response. lkml does that sometimes...
P.P.S. It's "isochronous scheduling" :P Means "same time".

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: 2.6.7-ck5

[Prakash K. Cheemplavam]

Con Kolivas wrote:
> John Richard Moser wrote:
>> When do you think the staircase, batch, and isometric scheduling will
>> reach mainline-quality?  Do you think you'll be ready to ask Andrew to
>> merge it soon, or will it be a while before it's quite ready for that?
> 
> 
> Well I think they're all ready for prime time now, I just dont think 
> prime time is ready for it. This is too large a change for mainline 2.6 
> which keeps -ck in business ;)

I don't know whether this was already discussed, but what about adding 
framework so that (like io-schedulers) the cpu scheduler could be chosen 
on boot time? This would make it easy to test different cpu schedulers.

Cheers,

Prakash

Re: 2.6.7-ck5

[P]

Prakash K. Cheemplavam wrote:
> I don't know whether this was already discussed, but what about adding 
> framework so that (like io-schedulers) the cpu scheduler could be chosen 
> on boot time? This would make it easy to test different cpu schedulers.

Discussed today actually :-)
http://marc.theaimsgroup.com/?l=linux-kernel&m=108875642724907&w=2

Pádraig.

Re: 2.6.7-ck5

[Redeeman]

On Wed, 2004-07-07 at 17:53 +0200, Prakash K. Cheemplavam wrote:
> Con Kolivas wrote:
> > John Richard Moser wrote:
> >> When do you think the staircase, batch, and isometric scheduling will
> >> reach mainline-quality?  Do you think you'll be ready to ask Andrew to
> >> merge it soon, or will it be a while before it's quite ready for that?
> > 
> > 
> > Well I think they're all ready for prime time now, I just dont think 
> > prime time is ready for it. This is too large a change for mainline 2.6 
> > which keeps -ck in business ;)
> 
> I don't know whether this was already discussed, but what about adding 
> framework so that (like io-schedulers) the cpu scheduler could be chosen 
> on boot time? This would make it easy to test different cpu schedulers.
might be a good idea, but i dont think doing such a change in 2.6 is
good, but for 2.7, a good idea
> 
> Cheers,
> 
> Prakash
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: 2.6.7-ck5

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm on gentoo here, and just copied the ck2 ebuild to ck5, and it missed
on a patch called ck-sources-2.6.IPTables-RDoS.  Any idea what this is,
and is it already in?

Con, I have no idea where it came from, but could mail it to you (I
won't post it to the list because it's not mine and I have issues about
doing such things) if you want.  I'm not sure if it applied to ck3; it
may have not existed at the time.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7CiWhDd4aOud5P8RAsxMAJ4+Jfx/l4m7LUxoKZa/io1FF2S+kQCghf5v
SN/TVYqZa756rnUtNsVBI+4=
=qwGv
-----END PGP SIGNATURE-----

Re: 2.6.7-ck5

[Prakash K. Cheemplavam]

John Richard Moser wrote:
> I'm on gentoo here, and just copied the ck2 ebuild to ck5, and it missed
> on a patch called ck-sources-2.6.IPTables-RDoS.  Any idea what this is,
> and is it already in?

I think it is already inside (look at Con's patch list). Just remove the 
line from the ebuild.

Prakash

Re: 2.6.7-ck5

[Prakash K. Cheemplavam]

P@draigBrady.com wrote:
> Prakash K. Cheemplavam wrote:
> 
>> I don't know whether this was already discussed, but what about adding 
>> framework so that (like io-schedulers) the cpu scheduler could be 
>> chosen on boot time? This would make it easy to test different cpu 
>> schedulers.
> 
> 
> Discussed today actually :-)
> http://marc.theaimsgroup.com/?l=linux-kernel&m=108875642724907&w=2

Oh Ok. :-) Great!

Prakash

Re: 2.6.7-ck5

[Wes Janzen]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

Hi Con,

I'm running ck4 and I'm getting pauses during init (alsa, hdparm, 
hotplug).  By repeatedly pressing Alt-SysRq-p, I can get things going 
again within 15 seconds, otherwise I suspect it would take that 3 1/2 
hours to finish init like it did with ck3.  I think I had the same thing 
just happen in X.  The mouse got jerky and then stopped responding, even 
numlock wouldn't respond for about 15-30 seconds.  I'm logging a vmstat 
now to see if I can reproduce it.

It seems that disabling kernel preemption solved the problem during 
init, but the system feels slower (jerky mouse under X during compile).

Would anything that you've updated since ck4 take care of this?  If not, 
is there anything you can suggest I do to troubleshoot this issue?

Thanks,

Wes

Con Kolivas wrote:

> Patchset update:
>
> ...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.7-ck5

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

Wes Janzen writes:

> Hi Con,
> 
> I'm running ck4 and I'm getting pauses during init (alsa, hdparm, 
> hotplug).  By repeatedly pressing Alt-SysRq-p, I can get things going 
> again within 15 seconds, otherwise I suspect it would take that 3 1/2 
> hours to finish init like it did with ck3.  I think I had the same thing 
> just happen in X.  The mouse got jerky and then stopped responding, even 
> numlock wouldn't respond for about 15-30 seconds.  I'm logging a vmstat 
> now to see if I can reproduce it.

The suspect patch would be bootsplash.

> It seems that disabling kernel preemption solved the problem during 
> init, but the system feels slower (jerky mouse under X during compile).

It's extremely unlikely that the system would actually feel faster with 
preemption on in -ck so I suspect a touch of placebo effect.

> Would anything that you've updated since ck4 take care of this?  If not, 
> is there anything you can suggest I do to troubleshoot this issue?

Possibly the updated bootsplash patch might help. Another user had 
instability which was tracked down to bootsplash. The staircase scheduler is 
virtually unchanged so if it is responsible (which I'd like to think isn't 
the case) then the problem will not go away.

Suggestions (in order of likelihood):
-Update to -ck5
-Disable bootsplash
-Reverse patch bootsplash
-Don't enable any funky config options like regparm, 4k stacks or different 
Hz
-Start backing out patches one by one from the last to the first till it 
goes away.

Keep us informed if you find the culprit please.

> Thanks,
> 
> Wes

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.7-ck5

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
> > How about autoregulated swappiness, which seems to be very efficient at
>  > its job?
> 
>  It's been around for quite a while, and akpm has not expressed any 
>  interest in it so I think this will only ever flounder in the -ck domain.

Nobody sent me the patch.  And the
justification/explanation/sales-brochure.  And the benchmarks...

[PATCH] Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1.1: Type: text/plain, Size: 2469 bytes --]

Andrew Morton writes:

> Con Kolivas <kernel@kolivas.org> wrote:
>>
>> > How about autoregulated swappiness, which seems to be very efficient at
>>  > its job?
>> 
>>  It's been around for quite a while, and akpm has not expressed any 
>>  interest in it so I think this will only ever flounder in the -ck domain.
> 
> Nobody sent me the patch.  And the
> justification/explanation/sales-brochure.  And the benchmarks...

Ah what the heck. They can only be knocked back to where they already are.

Attached are two patches designed to address the need to change the 
swap behaviour under different loads in 2.6.  They work on the premise that 
it is the percentage of application pages in physical ram that determines 
the need to be hitting swap. 

The first patch varies the global "swappiness" value by making it depend on 
the application pages% biased downwards in a pseudo-logarithmic fashion. It 
also looks at the percentage of swap space used and will decrease the 
swappiness value once the percentage of this free is less than 100 - 
application pages%.  It also introduces the sysctl of autoswappiness to 
disable this mechanism entirely if a manual swappiness is still 
desired.
It has the effect of running the machine with a fairly low swappiness during 
low periods of memory stress making it very unlikely to hit swap during 
large file transfers and the like, but allowing a more generous swappiness 
once physical ram is heavily consumed by applications. It also improves 
fairly dramatically the duration of swap thrash:

Make -j32 on mem=128M on P4:
8:25.92

with autoswappiness:
4:40.9

The second patch extends the autoswappiness to also start inactivating pages 
more aggressively as the application pages% increases, also with the same 
aims as the first patch. The sysctl introduced with autoswappiness is 
renamed to autoregulation to reflect the larger scope of the changes, and 
once again may be disabled to allow aiming for the fixed 2/3 active/inactive 
ratio and the manual swappiness.

with autoinactivation and no autoswappiness:
4:16.79

with autoswappiness and autoinactivation:
3:06.64

As well as the swap thrash scenario, on a desktop this has markedly reduced 
times for applications to come back to life after periods of non application 
memory stress such as copying iso images and the standard test case of the 
overnight run of updatedb.

Patches against 2.6.7-mm6 attached

Signed-off-by: Con Kolivas <kernel@kolivas.org>


[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2.1: autoswappiness --]
[-- Type: text/plain, Size: 3173 bytes --]

Index: linux-2.6.7-mm6/include/linux/swap.h
===================================================================
--- linux-2.6.7-mm6.orig/include/linux/swap.h	2004-07-05 19:41:48.000000000 +1000
+++ linux-2.6.7-mm6/include/linux/swap.h	2004-07-05 23:18:01.980100050 +1000
@@ -175,6 +175,7 @@
 extern int try_to_free_pages(struct zone **, unsigned int, unsigned int);
 extern int shrink_all_memory(int);
 extern int vm_swappiness;
+extern int auto_swappiness;
 
 #ifdef CONFIG_MMU
 /* linux/mm/shmem.c */
Index: linux-2.6.7-mm6/include/linux/sysctl.h
===================================================================
--- linux-2.6.7-mm6.orig/include/linux/sysctl.h	2004-07-05 19:44:05.000000000 +1000
+++ linux-2.6.7-mm6/include/linux/sysctl.h	2004-07-05 23:18:38.651379506 +1000
@@ -167,6 +167,7 @@
 	VM_BLOCK_DUMP=24,	/* block dump mode */
 	VM_HUGETLB_GROUP=25,	/* permitted hugetlb group */
 	VM_VFS_CACHE_PRESSURE=26, /* dcache/icache reclaim pressure */
+	VM_AUTO_SWAPPINESS=27,	/* Make vm_swappiness autoregulated */
 };
 
 
Index: linux-2.6.7-mm6/kernel/sysctl.c
===================================================================
--- linux-2.6.7-mm6.orig/kernel/sysctl.c	2004-07-05 19:44:05.000000000 +1000
+++ linux-2.6.7-mm6/kernel/sysctl.c	2004-07-05 23:18:01.983099583 +1000
@@ -727,6 +727,14 @@
 		.extra1		= &zero,
 		.extra2		= &one_hundred,
 	},
+	{
+		.ctl_name	= VM_AUTO_SWAPPINESS,
+		.procname	= "autoswappiness",
+		.data		= &auto_swappiness,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
 #ifdef CONFIG_HUGETLB_PAGE
 	 {
 		.ctl_name	= VM_HUGETLB_PAGES,
Index: linux-2.6.7-mm6/mm/vmscan.c
===================================================================
--- linux-2.6.7-mm6.orig/mm/vmscan.c	2004-07-05 19:41:49.000000000 +1000
+++ linux-2.6.7-mm6/mm/vmscan.c	2004-07-05 23:18:01.984099427 +1000
@@ -119,6 +119,7 @@
  * From 0 .. 100.  Higher means more swappy.
  */
 int vm_swappiness = 60;
+int auto_swappiness = 1;
 static long total_memory;
 
 static LIST_HEAD(shrinker_list);
@@ -691,6 +692,41 @@
 	 */
 	mapped_ratio = (sc->nr_mapped * 100) / total_memory;
 
+#ifdef CONFIG_SWAP
+	if (auto_swappiness) {
+		int app_percent;
+		struct sysinfo i;
+		
+		si_swapinfo(&i);
+			
+		if (likely(i.totalswap >= 100)) {
+			int swap_centile;
+	
+			/*
+			 * app_percent is the percentage of physical ram used
+			 * by application pages.
+			 */
+			si_meminfo(&i);
+			app_percent = 100 - ((i.freeram + get_page_cache_size() -
+				swapper_space.nrpages) / (i.totalram / 100));
+	
+			/*
+			 * swap_centile is the percentage of the last (sizeof physical
+			 * ram) of swap free.
+			 */
+			swap_centile = i.freeswap / 
+				(min(i.totalswap, i.totalram) / 100);
+			/*
+			 * Autoregulate vm_swappiness to be equal to the lowest of
+			 * app_percent and swap_centile.  Bias it downwards -ck
+			 */
+			vm_swappiness = min(app_percent, swap_centile);
+			vm_swappiness = vm_swappiness * vm_swappiness / 100;
+		} else 
+			vm_swappiness = 0;
+	}
+#endif
+	
 	/*
 	 * Now decide how much we really want to unmap some pages.  The mapped
 	 * ratio is downgraded - just because there's a lot of mapped memory

[-- Attachment #2.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #3.1: vm_autoregulate --]
[-- Type: text/plain, Size: 3380 bytes --]

Index: linux-2.6.7-mm6/include/linux/swap.h
===================================================================
--- linux-2.6.7-mm6.orig/include/linux/swap.h	2004-07-05 23:18:01.980100050 +1000
+++ linux-2.6.7-mm6/include/linux/swap.h	2004-07-05 23:19:20.614833487 +1000
@@ -175,7 +175,7 @@
 extern int try_to_free_pages(struct zone **, unsigned int, unsigned int);
 extern int shrink_all_memory(int);
 extern int vm_swappiness;
-extern int auto_swappiness;
+extern int vm_autoregulate;
 
 #ifdef CONFIG_MMU
 /* linux/mm/shmem.c */
Index: linux-2.6.7-mm6/include/linux/sysctl.h
===================================================================
--- linux-2.6.7-mm6.orig/include/linux/sysctl.h	2004-07-05 23:18:38.651379506 +1000
+++ linux-2.6.7-mm6/include/linux/sysctl.h	2004-07-05 23:19:42.367440258 +1000
@@ -167,7 +167,7 @@
 	VM_BLOCK_DUMP=24,	/* block dump mode */
 	VM_HUGETLB_GROUP=25,	/* permitted hugetlb group */
 	VM_VFS_CACHE_PRESSURE=26, /* dcache/icache reclaim pressure */
-	VM_AUTO_SWAPPINESS=27,	/* Make vm_swappiness autoregulated */
+	VM_AUTOREGULATE=27,     /* swappiness and inactivation autoregulated */
 };
 
 
Index: linux-2.6.7-mm6/kernel/sysctl.c
===================================================================
--- linux-2.6.7-mm6.orig/kernel/sysctl.c	2004-07-05 23:18:01.983099583 +1000
+++ linux-2.6.7-mm6/kernel/sysctl.c	2004-07-05 23:19:20.618832863 +1000
@@ -728,9 +728,9 @@
 		.extra2		= &one_hundred,
 	},
 	{
-		.ctl_name	= VM_AUTO_SWAPPINESS,
-		.procname	= "autoswappiness",
-		.data		= &auto_swappiness,
+		.ctl_name	= VM_AUTOREGULATE,
+		.procname	= "autoregulate",
+		.data		= &vm_autoregulate,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
Index: linux-2.6.7-mm6/mm/vmscan.c
===================================================================
--- linux-2.6.7-mm6.orig/mm/vmscan.c	2004-07-05 23:18:01.984099427 +1000
+++ linux-2.6.7-mm6/mm/vmscan.c	2004-07-05 23:19:20.619832707 +1000
@@ -119,7 +119,8 @@
  * From 0 .. 100.  Higher means more swappy.
  */
 int vm_swappiness = 60;
-int auto_swappiness = 1;
+int vm_autoregulate = 1;
+static int app_percent = 1;
 static long total_memory;
 
 static LIST_HEAD(shrinker_list);
@@ -650,7 +651,9 @@
 	long mapped_ratio;
 	long distress;
 	long swap_tendency;
+	struct sysinfo i;
 
+	si_meminfo(&i);
 	lru_add_drain();
 	pgmoved = 0;
 	spin_lock_irq(&zone->lru_lock);
@@ -692,23 +695,21 @@
 	 */
 	mapped_ratio = (sc->nr_mapped * 100) / total_memory;
 
+	/*
+	 * app_percent is the percentage of physical ram used
+	 * by application pages.
+	 */
+	si_meminfo(&i);
 #ifdef CONFIG_SWAP
-	if (auto_swappiness) {
-		int app_percent;
-		struct sysinfo i;
-		
+	app_percent = 100 - ((i.freeram + get_page_cache_size() -
+		swapper_space.nrpages) / (i.totalram / 100));
+
+	if (vm_autoregulate) {
 		si_swapinfo(&i);
 			
 		if (likely(i.totalswap >= 100)) {
 			int swap_centile;
 	
-			/*
-			 * app_percent is the percentage of physical ram used
-			 * by application pages.
-			 */
-			si_meminfo(&i);
-			app_percent = 100 - ((i.freeram + get_page_cache_size() -
-				swapper_space.nrpages) / (i.totalram / 100));
 	
 			/*
 			 * swap_centile is the percentage of the last (sizeof physical
@@ -725,6 +726,9 @@
 		} else 
 			vm_swappiness = 0;
 	}
+#else
+	app_percent = 100 - ((i.freeram + get_page_cache_size()) / 
+		(i.totalram / 100));
 #endif
 	
 	/*

[-- Attachment #3.2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1.1: Type: text/plain, Size: 753 bytes --]

Con Kolivas writes:

> Andrew Morton writes:
> 
>> Con Kolivas <kernel@kolivas.org> wrote:
>>>
>>> > How about autoregulated swappiness, which seems to be very efficient at
>>>  > its job?
>>> 
>>>  It's been around for quite a while, and akpm has not expressed any 
>>>  interest in it so I think this will only ever flounder in the -ck domain.
>> 
>> Nobody sent me the patch.  And the
>> justification/explanation/sales-brochure.  And the benchmarks...
> 
> Ah what the heck. They can only be knocked back to where they already are.

Hmm the second patch doesn't appear complete against mm6. I'll have to 
re-create it once I can look at the code at home. Here is the version 
against vanilla so you can see what it's supposed to do. Apologies.

Con

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2.1: vm_autoregulate --]
[-- Type: text/plain, Size: 4654 bytes --]

Index: linux-2.6.7-ck/include/linux/swap.h
===================================================================
--- linux-2.6.7-ck.orig/include/linux/swap.h	2004-07-01 19:41:13.375151780 +1000
+++ linux-2.6.7-ck/include/linux/swap.h	2004-07-01 19:41:30.134517540 +1000
@@ -175,7 +175,7 @@
 extern int try_to_free_pages(struct zone **, unsigned int, unsigned int);
 extern int shrink_all_memory(int);
 extern int vm_swappiness;
-extern int auto_swappiness;
+extern int vm_autoregulate;
 
 #ifdef CONFIG_MMU
 /* linux/mm/shmem.c */
Index: linux-2.6.7-ck/include/linux/sysctl.h
===================================================================
--- linux-2.6.7-ck.orig/include/linux/sysctl.h	2004-07-01 19:41:13.376151623 +1000
+++ linux-2.6.7-ck/include/linux/sysctl.h	2004-07-01 19:41:30.136517226 +1000
@@ -166,7 +166,7 @@
 	VM_LAPTOP_MODE=23,	/* vm laptop mode */
 	VM_BLOCK_DUMP=24,	/* block dump mode */
 	VM_HUGETLB_GROUP=25,	/* permitted hugetlb group */
-	VM_AUTO_SWAPPINESS=26,	/* Make vm_swappiness autoregulated */
+	VM_AUTOREGULATE=26,     /* swappiness and inactivation autoregulated */
 };
 
 
Index: linux-2.6.7-ck/kernel/sysctl.c
===================================================================
--- linux-2.6.7-ck.orig/kernel/sysctl.c	2004-07-01 19:41:13.377151466 +1000
+++ linux-2.6.7-ck/kernel/sysctl.c	2004-07-01 19:41:30.137517069 +1000
@@ -744,9 +744,9 @@
 		.extra2		= &one_hundred,
 	},
 	{
-		.ctl_name	= VM_AUTO_SWAPPINESS,
-		.procname	= "autoswappiness",
-		.data		= &auto_swappiness,
+		.ctl_name	= VM_AUTOREGULATE,
+		.procname	= "autoregulate",
+		.data		= &vm_autoregulate,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
Index: linux-2.6.7-ck/mm/vmscan.c
===================================================================
--- linux-2.6.7-ck.orig/mm/vmscan.c	2004-07-01 19:41:13.378151309 +1000
+++ linux-2.6.7-ck/mm/vmscan.c	2004-07-01 19:45:01.086359211 +1000
@@ -43,7 +43,8 @@
  * From 0 .. 100.  Higher means more swappy.
  */
 int vm_swappiness = 60;
-int auto_swappiness = 1;
+int vm_autoregulate = 1;
+static int app_percent = 1;
 static long total_memory;
 
 #define lru_to_page(_head) (list_entry((_head)->prev, struct page, lru))
@@ -649,7 +650,9 @@
 	long mapped_ratio;
 	long distress;
 	long swap_tendency;
+	struct sysinfo i;
 
+	si_meminfo(&i);
 	lru_add_drain();
 	pgmoved = 0;
 	spin_lock_irq(&zone->lru_lock);
@@ -691,23 +694,21 @@
 	 */
 	mapped_ratio = (sc->nr_mapped * 100) / total_memory;
 
+	/*
+	 * app_percent is the percentage of physical ram used
+	 * by application pages.
+	 */
+	si_meminfo(&i);
 #ifdef CONFIG_SWAP
-	if (auto_swappiness) {
-		int app_percent;
-		struct sysinfo i;
-		
+	app_percent = 100 - ((i.freeram + get_page_cache_size() -
+		swapper_space.nrpages) / (i.totalram / 100));
+
+	if (vm_autoregulate) {
 		si_swapinfo(&i);
 			
 		if (likely(i.totalswap >= 100)) {
 			int swap_centile;
 	
-			/*
-			 * app_percent is the percentage of physical ram used
-			 * by application pages.
-			 */
-			si_meminfo(&i);
-			app_percent = 100 - ((i.freeram + get_page_cache_size() -
-				swapper_space.nrpages) / (i.totalram / 100));
 	
 			/*
 			 * swap_centile is the percentage of the last (sizeof physical
@@ -724,6 +725,9 @@
 		} else 
 			vm_swappiness = 0;
 	}
+#else
+	app_percent = 100 - ((i.freeram + get_page_cache_size()) / 
+		(i.totalram / 100));
 #endif
 	
 	/*
@@ -834,11 +838,16 @@
 static void
 shrink_zone(struct zone *zone, struct scan_control *sc)
 {
-	unsigned long scan_active, scan_inactive;
-	int count;
+	unsigned long scan_active, scan_inactive, biased_active;
+	int count, biased_ap;
 
 	scan_inactive = (zone->nr_active + zone->nr_inactive) >> sc->priority;
 
+	if (vm_autoregulate) {
+		biased_ap = app_percent * app_percent / 100;
+		biased_active = zone->nr_active / (101 - biased_ap) * 100;
+	} else
+		biased_active = zone->nr_active;
 	/*
 	 * Try to keep the active list 2/3 of the size of the cache.  And
 	 * make sure that refill_inactive is given a decent number of pages.
@@ -849,7 +858,7 @@
 	 * `scan_active' just to make sure that the kernel will slowly sift
 	 * through the active list.
 	 */
-	if (zone->nr_active >= 4*(zone->nr_inactive*2 + 1)) {
+	if (biased_active >= 4*(zone->nr_inactive*2 + 1)) {
 		/* Don't scan more than 4 times the inactive list scan size */
 		scan_active = 4*scan_inactive;
 	} else {
@@ -857,7 +866,7 @@
 
 		/* Cast to long long so the multiply doesn't overflow */
 
-		tmp = (unsigned long long)scan_inactive * zone->nr_active;
+		tmp = (unsigned long long)scan_inactive * biased_active;
 		do_div(tmp, zone->nr_inactive*2 + 1);
 		scan_active = (unsigned long)tmp;
 	}

[-- Attachment #2.2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Autoregulate swappiness & inactivation

[Nick Piggin]

Con Kolivas wrote:
> Andrew Morton writes:
> 
>> Con Kolivas <kernel@kolivas.org> wrote:
>>
>>>
>>> > How about autoregulated swappiness, which seems to be very 
>>> efficient at
>>>  > its job?
>>>
>>>  It's been around for quite a while, and akpm has not expressed any 
>>>  interest in it so I think this will only ever flounder in the -ck 
>>> domain.
>>
>>
>> Nobody sent me the patch.  And the
>> justification/explanation/sales-brochure.  And the benchmarks...
> 
> 
> Ah what the heck. They can only be knocked back to where they already are.
> 

A few comments. I think making swappiness depend on the amount of
swap you have used is not a good idea. I might be wrong though, but
generally you should only make something *more* complex if you have
a good rationale and good numbers (you have the later, Andrew might
consider this enough). I especially don't like this sort of temporal
dependancy either, because it makes things much harder to reproduce
and think through.

Secondly, can you please not mess with the exported sysctl. If you
think your "autoswappiness" calculation is better than the current
swappiness one, just completely replace it. Bonus points if you can
retain the swappiness knob in some capacity.

Numbers look good though. I'll get around to doing some tests soon.

Re: [PATCH] Autoregulate swappiness & inactivation

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
>  Ah what the heck. They can only be knocked back to where they already are.

hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
nice numbers though.

How come vm_swappiness gets squared?  That's the mysterious "bias
downwards", yes?  What's the theory there?

Please define this new term "application pages"?

Those si_swapinfo() and si_meminfo() calls need to come out of there.

A diff against Documentation/filesystems/proc.txt will be needed sometime,
please.

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

Nick Piggin writes:

> Con Kolivas wrote:
>> Andrew Morton writes:
>> 
>>> Con Kolivas <kernel@kolivas.org> wrote:
>>>
>>>>
>>>> > How about autoregulated swappiness, which seems to be very 
>>>> efficient at
>>>>  > its job?
>>>>
>>>>  It's been around for quite a while, and akpm has not expressed any 
>>>>  interest in it so I think this will only ever flounder in the -ck 
>>>> domain.
>>>
>>>
>>> Nobody sent me the patch.  And the
>>> justification/explanation/sales-brochure.  And the benchmarks...
>> 
>> 
>> Ah what the heck. They can only be knocked back to where they already are.
>> 
> 
> A few comments. I think making swappiness depend on the amount of
> swap you have used is not a good idea. I might be wrong though, but
> generally you should only make something *more* complex if you have
> a good rationale and good numbers (you have the later, Andrew might
> consider this enough). I especially don't like this sort of temporal
> dependancy either, because it makes things much harder to reproduce
> and think through.

Noted. The amount of swap hardly has any effect on the swappiness except 
when you're close to OOMing and it is harder to OOM with this in place.

> Secondly, can you please not mess with the exported sysctl. If you
> think your "autoswappiness" calculation is better than the current
> swappiness one, just completely replace it. Bonus points if you can
> retain the swappiness knob in some capacity.

I agree and would like them all removed, but people just love to leave the 
knobs in place. While I dont think the knobs should still be there either, 
I'm not reluctant to leave something that innocuous if the users want them.

> Numbers look good though. I'll get around to doing some tests soon.

Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

Con Kolivas wrote:
> Nick Piggin writes:

>> A few comments. I think making swappiness depend on the amount of
>> swap you have used is not a good idea. I might be wrong though, but
>> generally you should only make something *more* complex if you have
>> a good rationale and good numbers (you have the later, Andrew might
>> consider this enough). I especially don't like this sort of temporal
>> dependancy either, because it makes things much harder to reproduce
>> and think through.
> 
> 
> Noted. The amount of swap hardly has any effect on the swappiness except 
> when you're close to OOMing and it is harder to OOM with this in place.
> 

OK that's easy then. The OOM algorithm can be changed if it is
OOMing too easily.

>> Secondly, can you please not mess with the exported sysctl. If you
>> think your "autoswappiness" calculation is better than the current
>> swappiness one, just completely replace it. Bonus points if you can
>> retain the swappiness knob in some capacity.
> 
> 
> I agree and would like them all removed, but people just love to leave 
> the knobs in place. While I dont think the knobs should still be there 
> either, I'm not reluctant to leave something that innocuous if the users 
> want them.
> 

Well, get rid of the auto-tuning thing to start with, and merge
it into the swappiness calculation..

Regarding all these knobs, the main thing you want to avoid is
having loads of them because you can't find acceptable defaults.
I think "swappiness" is in the category of a good sysctl: it is
simple, meaningful to the admin, works, etc.

It has proven somewhat useful in testing ("set it to blah and see
if it still happens"). Or for people who know what they are doing.

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Andrew Morton writes:

> Con Kolivas <kernel@kolivas.org> wrote:
>>
>>  Ah what the heck. They can only be knocked back to where they already are.
> 
> hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
> nice numbers though.
> 
> How come vm_swappiness gets squared?  That's the mysterious "bias
> downwards", yes?  What's the theory there?

No real world feedback mechanism is linear. As the pressure grows the 
positive/negative feedback grows exponentially.

> Please define this new term "application pages"?

errm it's fuzzy to say the least. It's the closest I can come to 
representing what end users understand as "non-cached" pages.

> Those si_swapinfo() and si_meminfo() calls need to come out of there.

I'm game. I had the idea but not the skill. Anyone wanna help me with that?

> A diff against Documentation/filesystems/proc.txt will be needed sometime,
> please.

Ok. I'll try and put together one patch that does the lot.

Con


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

Nick Piggin writes:

> Con Kolivas wrote:
>> Nick Piggin writes:
> 
>>> A few comments. I think making swappiness depend on the amount of
>>> swap you have used is not a good idea. I might be wrong though, but
>>> generally you should only make something *more* complex if you have
>>> a good rationale and good numbers (you have the later, Andrew might
>>> consider this enough). I especially don't like this sort of temporal
>>> dependancy either, because it makes things much harder to reproduce
>>> and think through.
>> 
>> 
>> Noted. The amount of swap hardly has any effect on the swappiness except 
>> when you're close to OOMing and it is harder to OOM with this in place.
>> 
> 
> OK that's easy then. The OOM algorithm can be changed if it is
> OOMing too easily.

I didn't say it was easy, just harder with; but whatever - I can get rid of 
it.

>>> Secondly, can you please not mess with the exported sysctl. If you
>>> think your "autoswappiness" calculation is better than the current
>>> swappiness one, just completely replace it. Bonus points if you can
>>> retain the swappiness knob in some capacity.
>> 
>> 
>> I agree and would like them all removed, but people just love to leave 
>> the knobs in place. While I dont think the knobs should still be there 
>> either, I'm not reluctant to leave something that innocuous if the users 
>> want them.
>> 
> 
> Well, get rid of the auto-tuning thing to start with, and merge
> it into the swappiness calculation..
> 
> Regarding all these knobs, the main thing you want to avoid is
> having loads of them because you can't find acceptable defaults.
> I think "swappiness" is in the category of a good sysctl: it is
> simple, meaningful to the admin, works, etc.
> 
> It has proven somewhat useful in testing ("set it to blah and see
> if it still happens"). Or for people who know what they are doing.

Umm I think we're agreeing, no? I'm trying to leave the swappiness knob in 
for those who (think?) they know what they're doing. Somehow it needs to be 
turned to "manual" again.

Con


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
> Andrew Morton writes:
> 
> > Con Kolivas <kernel@kolivas.org> wrote:
> >>
> >>  Ah what the heck. They can only be knocked back to where they already are.
> > 
> > hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
> > nice numbers though.
> > 
> > How come vm_swappiness gets squared?  That's the mysterious "bias
> > downwards", yes?  What's the theory there?
> 
> No real world feedback mechanism is linear. As the pressure grows the 
> positive/negative feedback grows exponentially.

That takes me back.  The classic control system is PID:
Proportional/Integral/Derivative - they refer to the way in which the error
term (output-desired output) is fed back to the input:

Proportional: the bigger the error, the more input drive

Integral: feeding back a bit of the integral of the error prevents
permanent output skew due to non-infinite forward gain.

Derivative: feeding back -(rate of change) provides damping.

You can live without I and D - the main thing is to feed back the -error.

IOW: linear works just fine :)

Your answer didn't help me understand the design though.

> > Please define this new term "application pages"?
> 
> errm it's fuzzy to say the least. It's the closest I can come to 
> representing what end users understand as "non-cached" pages.

Isn't that mapped pages?

> > Those si_swapinfo() and si_meminfo() calls need to come out of there.
> 
> I'm game. I had the idea but not the skill. Anyone wanna help me with that?

Need to work out what cen be removed first.  The freeswap/totalswap can go.
 That leaves us needing what?  totalram and freeram.  If the algorithm can
be flipped over to use nr_mapped, we'd be looking good.

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

Con Kolivas wrote:
> Nick Piggin writes:
> 

>> OK that's easy then. The OOM algorithm can be changed if it is
>> OOMing too easily.
> 
> 
> I didn't say it was easy, just harder with; but whatever - I can get rid 
> of it.
> 

Please.

>>>> Secondly, can you please not mess with the exported sysctl. If you
>>>> think your "autoswappiness" calculation is better than the current
>>>> swappiness one, just completely replace it. Bonus points if you can
>>>> retain the swappiness knob in some capacity.
>>>
>>>
>>>
>>> I agree and would like them all removed, but people just love to 
>>> leave the knobs in place. While I dont think the knobs should still 
>>> be there either, I'm not reluctant to leave something that innocuous 
>>> if the users want them.
>>>
>>
>> Well, get rid of the auto-tuning thing to start with, and merge
>> it into the swappiness calculation..
>>
>> Regarding all these knobs, the main thing you want to avoid is
>> having loads of them because you can't find acceptable defaults.
>> I think "swappiness" is in the category of a good sysctl: it is
>> simple, meaningful to the admin, works, etc.
>>
>> It has proven somewhat useful in testing ("set it to blah and see
>> if it still happens"). Or for people who know what they are doing.
> 
> 
> Umm I think we're agreeing, no? I'm trying to leave the swappiness knob 
> in for those who (think?) they know what they're doing. Somehow it needs 
> to be turned to "manual" again.
> 

No. Fold your all "autoswappiness" stuff directly into the
reclaim_mapped calculation that was previously keyed off swappiness.
Don't have it modify vm_swappiness at all: work directly on
reclaim_mapped.

Then, you should be able to retain the user's vm_swappiness input
into the system as well. If you can't figure out a good place to
put this in, don't worry about it to start with.

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

Andrew Morton writes:

> Con Kolivas <kernel@kolivas.org> wrote:
>>
>> Andrew Morton writes:
>> 
>> > Con Kolivas <kernel@kolivas.org> wrote:
>> >>
>> >>  Ah what the heck. They can only be knocked back to where they already are.
>> > 
>> > hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
>> > nice numbers though.
>> > 
>> > How come vm_swappiness gets squared?  That's the mysterious "bias
>> > downwards", yes?  What's the theory there?
>> 
>> No real world feedback mechanism is linear. As the pressure grows the 
>> positive/negative feedback grows exponentially.
> 
> That takes me back.  The classic control system is PID:
> Proportional/Integral/Derivative - they refer to the way in which the error
> term (output-desired output) is fed back to the input:
> 
> Proportional: the bigger the error, the more input drive
> 
> Integral: feeding back a bit of the integral of the error prevents
> permanent output skew due to non-infinite forward gain.
> 
> Derivative: feeding back -(rate of change) provides damping.
> 
> You can live without I and D - the main thing is to feed back the -error.
> 
> IOW: linear works just fine :)

/me hides

Umm sorry the control systems I look at are physiological and tend to be 
exponential, so ignore me.

> Your answer didn't help me understand the design though.

Ok I'll just describe it. I should have just said that when mapped pages are 
low the best seems to be a very low swappiness, but not zero as zero seems 
to get bogged down easily (kswapd gets busy and basic things take longer) as 
occasionally slipping some pages onto swap helps. Generally it's when what I 
called the application pages (blush) get to around 75% that allowing things 
to swap at the rate equivalent to swappiness==60 is helpful. Once the mapped 
pages went greater than 75% the machine would start bogging down again if 
the swappiness remained at 60. I guess I made up the maths to fill the way 
the design worked best. Linear brought up the swappiness too easily and 
under swap thrash made things worse. It looked nicer but didn't really 
behave well.

>> > Please define this new term "application pages"?
>> 
>> errm it's fuzzy to say the least. It's the closest I can come to 
>> representing what end users understand as "non-cached" pages.
> 
> Isn't that mapped pages?

I'm all ears.

>> > Those si_swapinfo() and si_meminfo() calls need to come out of there.
>> 
>> I'm game. I had the idea but not the skill. Anyone wanna help me with that?
> 
> Need to work out what cen be removed first.  The freeswap/totalswap can go.
>  That leaves us needing what?  totalram and freeram.  If the algorithm can
> be flipped over to use nr_mapped, we'd be looking good.

Ok. I need some time to clean up this mess and try and figure out what to do 
then.

Con

Re: Autoregulate swappiness & inactivation

[FabF]

On Thu, 2004-07-08 at 10:27, Con Kolivas wrote:
> Andrew Morton writes:
> 
> > Con Kolivas <kernel@kolivas.org> wrote:
> >>
> >> Andrew Morton writes:
> >> 
> >> > Con Kolivas <kernel@kolivas.org> wrote:
> >> >>
> >> >>  Ah what the heck. They can only be knocked back to where they already are.
> >> > 
> >> > hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
> >> > nice numbers though.
> >> > 
> >> > How come vm_swappiness gets squared?  That's the mysterious "bias
> >> > downwards", yes?  What's the theory there?
> >> 
> >> No real world feedback mechanism is linear. As the pressure grows the 
> >> positive/negative feedback grows exponentially.
> > 
> > That takes me back.  The classic control system is PID:
> > Proportional/Integral/Derivative - they refer to the way in which the error
> > term (output-desired output) is fed back to the input:
> > 
> > Proportional: the bigger the error, the more input drive
> > 
> > Integral: feeding back a bit of the integral of the error prevents
> > permanent output skew due to non-infinite forward gain.
> > 
> > Derivative: feeding back -(rate of change) provides damping.
> > 
> > You can live without I and D - the main thing is to feed back the -error.
> > 
> > IOW: linear works just fine :)
> 
> /me hides
> 
> Umm sorry the control systems I look at are physiological and tend to be 
> exponential, so ignore me.
> 
> > Your answer didn't help me understand the design though.
> 
> Ok I'll just describe it. I should have just said that when mapped pages are 
> low the best seems to be a very low swappiness, but not zero as zero seems 
> to get bogged down easily (kswapd gets busy and basic things take longer) as 
> occasionally slipping some pages onto swap helps. Generally it's when what I 
> called the application pages (blush) get to around 75% that allowing things 
> to swap at the rate equivalent to swappiness==60 is helpful. Once the mapped 
> pages went greater than 75% the machine would start bogging down again if 
> the swappiness remained at 60. I guess I made up the maths to fill the way 
> the design worked best. Linear brought up the swappiness too easily and 
> under swap thrash made things worse. It looked nicer but didn't really 
> behave well.
> 
> >> > Please define this new term "application pages"?
> >> 
> >> errm it's fuzzy to say the least. It's the closest I can come to 
> >> representing what end users understand as "non-cached" pages.
> > 
> > Isn't that mapped pages?
> 
> I'm all ears.
> 
> >> > Those si_swapinfo() and si_meminfo() calls need to come out of there.
> >> 
> >> I'm game. I had the idea but not the skill. Anyone wanna help me with that?
> > 
> > Need to work out what cen be removed first.  The freeswap/totalswap can go.
> >  That leaves us needing what?  totalram and freeram.  If the algorithm can
> > be flipped over to use nr_mapped, we'd be looking good.
> 
> Ok. I need some time to clean up this mess and try and figure out what to do 
> then.
Con,
	What's interesting is try_to_free_pages comment :

" the zone may be full of dirty or under-writeback pages, which this
 * caller can't do much about.  We kick pdflush and take explicit naps
in the
 * hope that some of these pages can be written.  But if the allocating
task..."

	I mean do we have high activity profile of that side of the kernel when
bringing up some big application to life ?
	Does work consist here in 50% out, 50% in (time) ? Your anticipation
algorithm can help the "in" side but maybe we can optimize yet the "out"
side.btw, I'm surprised to see autoswappiness so far in fx tree:

page_reclaim
	try_to_free_pages
		shrink_caches
			shrink_zone
				refill_inactive_zone
					auto_swap calculation


IOW, does such parameter could not involve more decisions ?

Regards,
FabF

> 
> Con
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [ck] Re: 2.6.7-ck5

[GSehp]

Andrew Morton wrote:

>Con Kolivas <kernel@kolivas.org> wrote:
>  
>
>>>How about autoregulated swappiness, which seems to be very efficient at
>>>      
>>>
>> > its job?
>>
>> It's been around for quite a while, and akpm has not expressed any 
>> interest in it so I think this will only ever flounder in the -ck domain.
>>    
>>
>
>Nobody sent me the patch.  And the
>justification/explanation/sales-brochure.  And the benchmarks...
>_______________________________________________
>ck@vds.kolivas.org
>ck mailing list - unmoderated. Please reply-to-all when posting.
>http://bhhdoa.org.au/mailman/listinfo/ck
>
>  
>
      Hi All,
   I just joined the list. I have "played" with linux for about 3 years 
now. I am finally tired of windows to the point I have joined several 
Linux oriented Lists. I have to admit this list is one of the better 
lists I have joined. It is informative and active in a friendly and 
dynamic way. Thank you all for a great list!

Gary Sheppard

[PATCH] Autotune swappiness

[Con Kolivas]

[-- Attachment #1.1: Type: text/plain, Size: 1657 bytes --]

Here is another try at providing feedback to tune the vm_swappiness.

This patch tries to tune it dynamically according to the mapped_ratio. 
After some consideration I thought it would be easiest to simply remove 
the vm_swappiness tunable entirely, since that is the point of this patch.

Some in the field modelling and testing showed a biased downwards 
feedback based on the mapped ratio provided good performance under 
different workloads especially improving the desktop experience. The 
practical upshot of this is the machine will only ever swap very lightly 
while the percentage of mapped pages is low and allow more generous 
swapping as the percentage rises within the higher ranges.

The earlier design of this patch was much more complicated and the 
practical upshot of it was that it would make

vm_swappiness = mapped_ratio * mapped_ratio / 100


This had it's effect on the swappiness value in the setting of 
swap_tendency which was:

swap_tendency = mapped_ratio / 2 + distress + vm_swappiness


A close approximation was to make vm_swappiness a range of 0-150 instead 
of 100 and simplify swap tendency to be equal to this:

swap_tendency = distress + vm_swappiness


A follow up patch will use the vm_swappiness value to alter the rate of 
page inactivation as well, but for the moment I'll offer this one change 
for comments.

Patch applies to 2.6.7-mm6
Signed-off-by: Con Kolivas <kernel@kolivas.org>

  include/linux/swap.h   |    1 -
  include/linux/sysctl.h |   15 +++++++--------
  kernel/sysctl.c        |   11 -----------
  mm/vmscan.c            |   15 ++++++++-------
  4 files changed, 15 insertions(+), 27 deletions(-)


[-- Attachment #1.2: autotune_swappiness.diff --]
[-- Type: text/x-patch, Size: 3962 bytes --]

Index: linux-2.6.7-mm6/include/linux/swap.h
===================================================================
--- linux-2.6.7-mm6.orig/include/linux/swap.h	2004-07-09 02:15:07.509902884 +1000
+++ linux-2.6.7-mm6/include/linux/swap.h	2004-07-09 02:15:10.949349575 +1000
@@ -174,7 +174,6 @@
 /* linux/mm/vmscan.c */
 extern int try_to_free_pages(struct zone **, unsigned int, unsigned int);
 extern int shrink_all_memory(int);
-extern int vm_swappiness;
 
 #ifdef CONFIG_MMU
 /* linux/mm/shmem.c */
Index: linux-2.6.7-mm6/include/linux/sysctl.h
===================================================================
--- linux-2.6.7-mm6.orig/include/linux/sysctl.h	2004-07-09 02:15:07.509902884 +1000
+++ linux-2.6.7-mm6/include/linux/sysctl.h	2004-07-09 02:15:10.950349415 +1000
@@ -157,14 +157,13 @@
 	VM_OVERCOMMIT_RATIO=16, /* percent of RAM to allow overcommit in */
 	VM_PAGEBUF=17,		/* struct: Control pagebuf parameters */
 	VM_HUGETLB_PAGES=18,	/* int: Number of available Huge Pages */
-	VM_SWAPPINESS=19,	/* Tendency to steal mapped memory */
-	VM_LOWER_ZONE_PROTECTION=20,/* Amount of protection of lower zones */
-	VM_MIN_FREE_KBYTES=21,	/* Minimum free kilobytes to maintain */
-	VM_MAX_MAP_COUNT=22,	/* int: Maximum number of mmaps/address-space */
-	VM_LAPTOP_MODE=23,	/* vm laptop mode */
-	VM_BLOCK_DUMP=24,	/* block dump mode */
-	VM_HUGETLB_GROUP=25,	/* permitted hugetlb group */
-	VM_VFS_CACHE_PRESSURE=26, /* dcache/icache reclaim pressure */
+	VM_LOWER_ZONE_PROTECTION=19,/* Amount of protection of lower zones */
+	VM_MIN_FREE_KBYTES=20,	/* Minimum free kilobytes to maintain */
+	VM_MAX_MAP_COUNT=21,	/* int: Maximum number of mmaps/address-space */
+	VM_LAPTOP_MODE=22,	/* vm laptop mode */
+	VM_BLOCK_DUMP=23,	/* block dump mode */
+	VM_HUGETLB_GROUP=24,	/* permitted hugetlb group */
+	VM_VFS_CACHE_PRESSURE=25, /* dcache/icache reclaim pressure */
 };
 
 
Index: linux-2.6.7-mm6/kernel/sysctl.c
===================================================================
--- linux-2.6.7-mm6.orig/kernel/sysctl.c	2004-07-09 02:15:07.496904975 +1000
+++ linux-2.6.7-mm6/kernel/sysctl.c	2004-07-09 02:15:10.951349254 +1000
@@ -700,17 +700,6 @@
 		.mode		= 0444 /* read-only*/,
 		.proc_handler	= &proc_dointvec,
 	},
-	{
-		.ctl_name	= VM_SWAPPINESS,
-		.procname	= "swappiness",
-		.data		= &vm_swappiness,
-		.maxlen		= sizeof(vm_swappiness),
-		.mode		= 0644,
-		.proc_handler	= &proc_dointvec_minmax,
-		.strategy	= &sysctl_intvec,
-		.extra1		= &zero,
-		.extra2		= &one_hundred,
-	},
 #ifdef CONFIG_HUGETLB_PAGE
 	 {
 		.ctl_name	= VM_HUGETLB_PAGES,
Index: linux-2.6.7-mm6/mm/vmscan.c
===================================================================
--- linux-2.6.7-mm6.orig/mm/vmscan.c	2004-07-09 02:15:07.495905136 +1000
+++ linux-2.6.7-mm6/mm/vmscan.c	2004-07-09 02:15:21.366673884 +1000
@@ -116,9 +116,9 @@
 #endif
 
 /*
- * From 0 .. 100.  Higher means more swappy.
+ * From 0 .. 150.  Higher means more swappy.
  */
-int vm_swappiness = 60;
+static int vm_swappiness = 0;
 static long total_memory;
 
 static LIST_HEAD(shrinker_list);
@@ -690,17 +690,18 @@
 	 * is mapped.
 	 */
 	mapped_ratio = (sc->nr_mapped * 100) / total_memory;
-
+	
 	/*
 	 * Now decide how much we really want to unmap some pages.  The mapped
-	 * ratio is downgraded - just because there's a lot of mapped memory
+	 * ratio effect is downgraded by biasing downwards the value of
+	 * vm_swappiness - just because there's a lot of mapped memory
 	 * doesn't necessarily mean that page reclaim isn't succeeding.
 	 *
 	 * The distress ratio is important - we don't want to start going oom.
-	 *
-	 * A 100% value of vm_swappiness overrides this algorithm altogether.
 	 */
-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	vm_swappiness = mapped_ratio * 150 / 100;
+	vm_swappiness = vm_swappiness * vm_swappiness / 150;
+	swap_tendency = distress + vm_swappiness;
 
 	/*
 	 * Now use this metric to decide whether to start moving mapped memory

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Autoregulate swappiness & inactivation

[Timothy Miller]

Con Kolivas wrote:

> /me hides
> 
> Umm sorry the control systems I look at are physiological and tend to be 
> exponential, so ignore me.

No.  I see no reason to disregard your understanding of biological 
control systems.  Millions of years of evolution have fine-tuned some 
very complex and robust control feedback systems.  While I wouldn't 
suggest that they're the only way to do the job, they're something that 
we should definately pay attention to.

Frankly, I think the cross-pollination that you bring from your 
background in medicine can do nothing but help us.

Re: [PATCH] Autotune swappiness

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
> Here is another try at providing feedback to tune the vm_swappiness.

I spent some time yesterday trying to demonstrate performance improvements
from those two patches.  Using

	make -j4 vmlinux with mem=64m

and

	qsbench -p 4 -m 96 with mem=256m

and was not able to do so, which is what I expected.

We do need more quantitative testing on this work.

Re: Autoregulate swappiness & inactivation

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Nick Piggin wrote:
| Con Kolivas wrote:
|
|> Nick Piggin writes:
|> Umm I think we're agreeing, no? I'm trying to leave the swappiness
|> knob in for those who (think?) they know what they're doing. Somehow
|> it needs to be turned to "manual" again.
|>
|
| No. Fold your all "autoswappiness" stuff directly into the
| reclaim_mapped calculation that was previously keyed off swappiness.
| Don't have it modify vm_swappiness at all: work directly on
| reclaim_mapped.
|
| Then, you should be able to retain the user's vm_swappiness input
| into the system as well. If you can't figure out a good place to
| put this in, don't worry about it to start with.
|

Wasn't the point of this patch to allow the machine to tweak the
swappiness knob itself according to what it thinks is best, unless the
user tells it not to?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7X8RhDd4aOud5P8RAkNBAJ99+wIoTY1sHTTwOdO5fH8lggBpPgCfVFuv
Db7yGOwZjB+nTd6GxnM8KdM=
=/eIv
-----END PGP SIGNATURE-----

Re: [ck] Re: [PATCH] Autoregulate swappiness & inactivation

[Mikhail Ramendik]

Nick Piggin wrote:

> Secondly, can you please not mess with the exported sysctl. If you
> think your "autoswappiness" calculation is better than the current
> swappiness one, just completely replace it. Bonus points if you can
> retain the swappiness knob in some capacity.

I as a user of -ck *strongly* disagree with this proposal. I want to be
able to try both manual and automatic setting, without recompiling the
kernel.

If you really must avoid another named exported sysctl, I suggest making
a "reserved" swappiness value, like 255, that would mean
"auto-regulate".

Yours, Mikhail Ramendik

Re: Autoregulate swappiness & inactivation

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Timothy Miller wrote:
|
|
| Con Kolivas wrote:
|
|> /me hides
|>
|> Umm sorry the control systems I look at are physiological and tend to
|> be exponential, so ignore me.
|
|
| No.  I see no reason to disregard your understanding of biological
| control systems.  Millions of years of evolution have fine-tuned some
| very complex and robust control feedback systems.  While I wouldn't
| suggest that they're the only way to do the job, they're something that
| we should definately pay attention to.
|
| Frankly, I think the cross-pollination that you bring from your
| background in medicine can do nothing but help us.
|

Con is a medic?  . . . . shit.  I'm 5 years overdue for my
measles/mumps/rhubella shot

/me hides *cough*

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7YB6hDd4aOud5P8RApYOAJ46eBZIWctU6vZ1fJWVGOlX+HvhagCdEp8L
Ved/y9Wwb+8wJodTkZY4spY=
=wNu5
-----END PGP SIGNATURE-----

Re: [ck] Re: Autoregulate swappiness & inactivation

[Mikhail Ramendik]

Hello,

Nick Piggin wrote:

> > Umm I think we're agreeing, no? I'm trying to leave the swappiness knob 
> > in for those who (think?) they know what they're doing. Somehow it needs 
> > to be turned to "manual" again.
> No. Fold your all "autoswappiness" stuff directly into the
> reclaim_mapped calculation that was previously keyed off swappiness.
> Don't have it modify vm_swappiness at all: work directly on
> reclaim_mapped.
> 
> Then, you should be able to retain the user's vm_swappiness input
> into the system as well. If you can't figure out a good place to
> put this in, don't worry about it to start with.

I, as a user, would be far less happy without the ability to set it to
"the old way". Of course "the old" vs "the new" may become a kernel
config option, but why is a recompile better than a sysctl? Out of 
principle?

Yours, Mikhail Ramendik

Re: Autoregulate swappiness & inactivation

[Timothy Miller]

John Richard Moser wrote:

> 
> Con is a medic?  . . . . shit.  I'm 5 years overdue for my
> measles/mumps/rhubella shot
> 
> /me hides *cough*


I'm not sure the term 'medic' quite does him justice.

Re: Autoregulate swappiness & inactivation

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Timothy Miller wrote:
|
| I'm not sure the term 'medic' quite does him justice.
|

Heh.  I didn't mean any offense.  :P

On a side note, Thunderbird keeps changing > to | for inline forwarding.
~ Any idea how to disable this?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7b85hDd4aOud5P8RAk/4AJ9urtDoFyxaLyUk8WD9X4JryHl0IQCeNdyD
i0cUhDcTwjqd6uJiN6MIO/o=
=+zqR
-----END PGP SIGNATURE-----

Re: [PATCH] Autotune swappiness

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Andrew Morton wrote:
> Con Kolivas <kernel@kolivas.org> wrote:
> 
>>Here is another try at providing feedback to tune the vm_swappiness.
> 
> 
> I spent some time yesterday trying to demonstrate performance improvements
> from those two patches.  Using
> 
> 	make -j4 vmlinux with mem=64m
> 
> and
> 
> 	qsbench -p 4 -m 96 with mem=256m
> 
> and was not able to do so, which is what I expected.
> 
> We do need more quantitative testing on this work.

Sure thing.

I need to point out a few things:
The point of this patch was to improve the swap behaviour on desktop 
like loads.
The fact that it improved the "when swap is thrashing" scenario (in my 
testing) was an unintentional bonus.
I dont think your load of j4 will induce quite the same swap thrash as 
what I was testing. I actually suspect the faster cpu & more jobs over 
fixed memory shows it more.
I need someone with more varied hardware to test it for me. I can 
recreate equivalent results on my current machine which has similar 
hardware, but I think results showing improvement on different machines 
  and different loads is what you're looking for... and since I'm 
currently quite low on hardware I can only offer results from this one 
(and my wife is hating it being offline o_0)

Anyone willing to offer to do some tests?

Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [ck] Re: [PATCH] Autoregulate swappiness & inactivation

[Nick Piggin]

Mikhail Ramendik wrote:
> Nick Piggin wrote:
> 
> 
>>Secondly, can you please not mess with the exported sysctl. If you
>>think your "autoswappiness" calculation is better than the current
>>swappiness one, just completely replace it. Bonus points if you can
>>retain the swappiness knob in some capacity.
> 
> 
> I as a user of -ck *strongly* disagree with this proposal. I want to be
> able to try both manual and automatic setting, without recompiling the
> kernel.
> 
> If you really must avoid another named exported sysctl, I suggest making
> a "reserved" swappiness value, like 255, that would mean
> "auto-regulate".

The point is, there really isn't anything fancy about this
"auto tuning". It just alters the reclaim_mapped formula.

If we decide that the new formula gives better results, then
we should go with that. Exposing an intermediate calculation
in the swappiness sysctl is meaningless.

You can then work out somewhere to input a manual "swappiness"
bias into the new calculation.

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

FabF wrote:
> Con,
> 	What's interesting is try_to_free_pages comment :
> 
> " the zone may be full of dirty or under-writeback pages, which this
>  * caller can't do much about.  We kick pdflush and take explicit naps
> in the
>  * hope that some of these pages can be written.  But if the allocating
> task..."
> 
> 	I mean do we have high activity profile of that side of the kernel when
> bringing up some big application to life ?
> 	Does work consist here in 50% out, 50% in (time) ? Your anticipation
> algorithm can help the "in" side but maybe we can optimize yet the "out"
> side.btw, I'm surprised to see autoswappiness so far in fx tree:
> 
> page_reclaim
> 	try_to_free_pages
> 		shrink_caches
> 			shrink_zone
> 				refill_inactive_zone
> 					auto_swap calculation
> 
> 
> IOW, does such parameter could not involve more decisions ?

If you put it that way, yes - it would classify as duct tape. However 
the code already acted based upon mapped_ratio which is pretty much all 
this patch does. Folded in in that sample patch I sent out earlier you 
can see that all it does is acted on mapped_ratio in a different manner 
so it's not really an extra layer at all.

-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	vm_swappiness = mapped_ratio * 150 / 100;
+	vm_swappiness = vm_swappiness * vm_swappiness / 150;
+	swap_tendency = distress + vm_swappiness;

Con

> Regards,
> FabF

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [ck] Re: [PATCH] Autotune swappiness

[Kerin Millar]

<snip>

> I need someone with more varied hardware to test it for me. I can
> recreate equivalent results on my current machine which has similar
> hardware, but I think results showing improvement on different machines
>   and different loads is what you're looking for... and since I'm
> currently quite low on hardware I can only offer results from this one
> (and my wife is hating it being offline o_0)
>
> Anyone willing to offer to do some tests?

Yes - I would. If you can define a digestible framework for testing then I
would be more than happy to work through it - digestible for mere mortals,
that is ;). I have three machines on which I currently have the
opportunity to conduct such tests:

* PIII (733MHz), 384Mb RAM, VIA Apollo MVP3 chipset
* Compaq Presario, P4 2GHz, 256Mb RAM, Brookdale-G chipset
* Compaq Proliant ML370 G3 server, P4 Xeon 3.06GHz (HT), 1Gb RAM

All machines have similar software characteristics - identical toolchain,
same versions of core GNU tools and so forth. By all means, let me know if
there is anything I can do to help.

Regards,

--Kerin Francis Millar

Re: Autoregulate swappiness & inactivation

[Felipe Alfaro Solana]

On Thu, 2004-07-08 at 13:12 -0400, John Richard Moser wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> 
> Timothy Miller wrote:
> |
> |
> | Con Kolivas wrote:
> |
> |> /me hides
> |>
> |> Umm sorry the control systems I look at are physiological and tend to
> |> be exponential, so ignore me.
> |
> |
> | No.  I see no reason to disregard your understanding of biological
> | control systems.  Millions of years of evolution have fine-tuned some
> | very complex and robust control feedback systems.  While I wouldn't
> | suggest that they're the only way to do the job, they're something that
> | we should definately pay attention to.
> |
> | Frankly, I think the cross-pollination that you bring from your
> | background in medicine can do nothing but help us.
> |
> 
> Con is a medic?  . . . . shit.  I'm 5 years overdue for my
> measles/mumps/rhubella shot

He's a Doctor, IIRC.

Re: Autoregulate swappiness & inactivation

[FabF]

On Fri, 2004-07-09 at 03:05, Con Kolivas wrote:
> FabF wrote:
> > Con,
> > 	What's interesting is try_to_free_pages comment :
> > 
> > " the zone may be full of dirty or under-writeback pages, which this
> >  * caller can't do much about.  We kick pdflush and take explicit naps
> > in the
> >  * hope that some of these pages can be written.  But if the allocating
> > task..."
> > 
> > 	I mean do we have high activity profile of that side of the kernel when
> > bringing up some big application to life ?
> > 	Does work consist here in 50% out, 50% in (time) ? Your anticipation
> > algorithm can help the "in" side but maybe we can optimize yet the "out"
> > side.btw, I'm surprised to see autoswappiness so far in fx tree:
> > 
> > page_reclaim
> > 	try_to_free_pages
> > 		shrink_caches
> > 			shrink_zone
> > 				refill_inactive_zone
> > 					auto_swap calculation
> > 
> > 
> > IOW, does such parameter could not involve more decisions ?
> 
> If you put it that way, yes - it would classify as duct tape. However 
> the code already acted based upon mapped_ratio which is pretty much all 
> this patch does. Folded in in that sample patch I sent out earlier you 
> can see that all it does is acted on mapped_ratio in a different manner 
> so it's not really an extra layer at all.
> 
> -	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
> +	vm_swappiness = mapped_ratio * 150 / 100;
> +	vm_swappiness = vm_swappiness * vm_swappiness / 150;
> +	swap_tendency = distress + vm_swappiness;

Here's an easy benchmark to demonstrate problem :
1.Run Mozilla
2.Minimize
3=>Mozilla Resident Size (mrs) : 24Mb
4.Run updatedb
5.=>mrs : 15Mb
6.updatedb ends up
7.mrs doesn't move at all (yes, it goes down as I'm typing this msg :)).

So my question is :
Don't we have a way to say "whose pages were reclaimed from and
 reattribute its" ? (having in mind memory status per se).
IOW flushing (I guess it's pdflush relevant ? ) do work for dead
processes but doesn't care about applications alive...

Regards,
FabF

> 
> Con
> 
> > Regards,
> > FabF

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

FabF wrote:

> 
> Here's an easy benchmark to demonstrate problem :
> 1.Run Mozilla
> 2.Minimize
> 3=>Mozilla Resident Size (mrs) : 24Mb
> 4.Run updatedb
> 5.=>mrs : 15Mb
> 6.updatedb ends up
> 7.mrs doesn't move at all (yes, it goes down as I'm typing this msg :)).
> 

How much RAM do you have? Does this happen with and without Con's
patch?

I don't have a problem here with your problem, however I'm running
my -np patchset, which has different use-once heuristics.

> So my question is :
> Don't we have a way to say "whose pages were reclaimed from and
>  reattribute its" ? (having in mind memory status per se).
> IOW flushing (I guess it's pdflush relevant ? ) do work for dead
> processes but doesn't care about applications alive...
> 

Page reclaim doesn't really know or care about processes, it
basically works on a global page pool.

pdflush is used to perform writeout of dirty data, so it has
no part in reducing Mozilla's RSS.

I don't really understand what you are asking though. Your basic
problem is that mozilla's resident memory gets evicted too easily,
is that right?

Re: Autoregulate swappiness & inactivation

[FabF]

On Fri, 2004-07-09 at 12:43, Nick Piggin wrote:
> FabF wrote:
> 
> > 
> > Here's an easy benchmark to demonstrate problem :
> > 1.Run Mozilla
> > 2.Minimize
> > 3=>Mozilla Resident Size (mrs) : 24Mb
> > 4.Run updatedb
> > 5.=>mrs : 15Mb
> > 6.updatedb ends up
> > 7.mrs doesn't move at all (yes, it goes down as I'm typing this msg :)).
> > 
> 
> How much RAM do you have? Does this happen with and without Con's
> patch?
Hi Nick,

256Mb with Con's patch 1 (autoswappiness activated) mm6
but it's general behaviour on my box :(

> 
> I don't have a problem here with your problem, however I'm running
> my -np patchset, which has different use-once heuristics.
> 
> > So my question is :
> > Don't we have a way to say "whose pages were reclaimed from and
> >  reattribute its" ? (having in mind memory status per se).
> > IOW flushing (I guess it's pdflush relevant ? ) do work for dead
> > processes but doesn't care about applications alive...
> > 
> 
> Page reclaim doesn't really know or care about processes, it
> basically works on a global page pool.
That's exactly the nerve center of the problem I guess.
When we swap we don't care about different processes but when some of
its is going in, we _quickly_ need to refresh memory but isn't it too
late ? I mean what do we do here ? We recover pages and "get application
to life".Desktop side of the story reminds me about some oses giving
_impression_ all was alright.I mean there must be a way to anticipate
 such trouble without renice -xx all GUI relevant processes
 in order to have both server/client cfg synergy.
 
> 
> pdflush is used to perform writeout of dirty data, so it has
> no part in reducing Mozilla's RSS.
Oops ... kswapd then ?

> 
> I don't really understand what you are asking though. Your basic
> problem is that mozilla's resident memory gets evicted too easily,
> is that right?
> 
Not at all.My problem is mozilla has some MB to recover when
reactivating; meanwhile, I consider there was sufficient resource to
share with it _before_ reactivation as I'm waiting some minutes after an
heavy process (e.g updatedb) to be done and over.

AFAICS, Con's patches are about auto-regulation, not about anticipation
(?)

Regards,
FabF

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

FabF wrote:
> On Fri, 2004-07-09 at 12:43, Nick Piggin wrote:

>>pdflush is used to perform writeout of dirty data, so it has
>>no part in reducing Mozilla's RSS.
> 
> Oops ... kswapd then ?
> 

Yep.

> 
>>I don't really understand what you are asking though. Your basic
>>problem is that mozilla's resident memory gets evicted too easily,
>>is that right?
>>
> 
> Not at all.My problem is mozilla has some MB to recover when
> reactivating; meanwhile, I consider there was sufficient resource to
> share with it _before_ reactivation as I'm waiting some minutes after an
> heavy process (e.g updatedb) to be done and over.
> 

You could try my -np7 patch, which would hopefully fix the problem
for you.

It may take some time and work before (if ever) the memory management
patches in it are merged though.

> AFAICS, Con's patches are about auto-regulation, not about anticipation
> (?)
> 

Yep. Con's patch changes the conditions that are required to start
reclaiming mapped pages. Basically: when to start swapping.

Re: [PATCH] Autotune swappiness

[Martin J. Bligh]

>>> Here is another try at providing feedback to tune the vm_swappiness.
>> 
>> 
>> I spent some time yesterday trying to demonstrate performance improvements
>> from those two patches.  Using
>> 
>> 	make -j4 vmlinux with mem=64m
>> 
>> and
>> 
>> 	qsbench -p 4 -m 96 with mem=256m
>> 
>> and was not able to do so, which is what I expected.
>> 
>> We do need more quantitative testing on this work.
> 
> Sure thing.
> 
> I need to point out a few things:
> The point of this patch was to improve the swap behaviour on desktop like loads.
> The fact that it improved the "when swap is thrashing" scenario (in my testing) was an unintentional bonus.
> I dont think your load of j4 will induce quite the same swap thrash as what I was testing. I actually suspect the faster cpu & more jobs over fixed memory shows it more.
> I need someone with more varied hardware to test it for me. I can recreate equivalent results on my current machine which has similar hardware, but I think results showing improvement on different machines   and different loads is what you're looking for... and since I'm currently quite low on hardware I can only offer results from this one (and my wife is hating it being offline o_0)
> 
> Anyone willing to offer to do some tests?

What kind of mem pressure are you looking for? kernel compile is easy to run, 
and straight "-j" should kill a small system fairly well ... you want it just
into swap, or thrashing the crap out of it?

M.

Re: [PATCH] Autotune swappiness

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1678 bytes --]

Martin J. Bligh wrote:
>>>>Here is another try at providing feedback to tune the vm_swappiness.
>>>
>>>
>>>I spent some time yesterday trying to demonstrate performance improvements
>>>from those two patches.  Using
>>>
>>>	make -j4 vmlinux with mem=64m
>>>
>>>and
>>>
>>>	qsbench -p 4 -m 96 with mem=256m
>>>
>>>and was not able to do so, which is what I expected.
>>>
>>>We do need more quantitative testing on this work.
>>
>>Sure thing.
>>
>>I need to point out a few things:
>>The point of this patch was to improve the swap behaviour on desktop like loads.
>>The fact that it improved the "when swap is thrashing" scenario (in my testing) was an unintentional bonus.
>>I dont think your load of j4 will induce quite the same swap thrash as what I was testing. I actually suspect the faster cpu & more jobs over fixed memory shows it more.
>>I need someone with more varied hardware to test it for me. I can recreate equivalent results on my current machine which has similar hardware, but I think results showing improvement on different machines   and different loads is what you're looking for... and since I'm currently quite low on hardware I can only offer results from this one (and my wife is hating it being offline o_0)
>>
>>Anyone willing to offer to do some tests?
> 
> 
> What kind of mem pressure are you looking for? kernel compile is easy to run, 
> and straight "-j" should kill a small system fairly well ... you want it just
> into swap, or thrashing the crap out of it?

It was coincidental that it helped the swap thash scenario. But yes, 
thrashing the crap out of it. Maybe taking 5-10 times longer than when 
there's enough memory for the job.

Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Autoregulate swappiness & inactivation

[FabF]

On Fri, 2004-07-09 at 13:24, Nick Piggin wrote:
> FabF wrote:
> > On Fri, 2004-07-09 at 12:43, Nick Piggin wrote:
> 
> >>pdflush is used to perform writeout of dirty data, so it has
> >>no part in reducing Mozilla's RSS.
> > 
> > Oops ... kswapd then ?
> > 
> 
> Yep.
> 
> > 
> >>I don't really understand what you are asking though. Your basic
> >>problem is that mozilla's resident memory gets evicted too easily,
> >>is that right?
> >>
> > 
> > Not at all.My problem is mozilla has some MB to recover when
> > reactivating; meanwhile, I consider there was sufficient resource to
> > share with it _before_ reactivation as I'm waiting some minutes after an
> > heavy process (e.g updatedb) to be done and over.
> > 
> 
> You could try my -np7 patch, which would hopefully fix the problem
> for you.

Hi Nick:)
	 I've been busy benchmarking Con's autoregulation which _is_ proved
interesting in middle pressure.
AFAICS mm7np works that way : it cleans up memory brightly so we do see
attractive free memory available _but_ relevant rss doesn't move 1 byte
:( So we'll have foregrounding elevation for sure (no cleaning required)
but application still have to 'emerge' and that's the heavier side of
the story I'm afraid.

Regards,
FabF

rss recovery

[FabF]

Nick,
	Putting some more pressure I finally saw the awaited behaviour from np
: rss gaining 1MB (or at least 1 byte :) : top reports 10M -> 11M )
directly after make was done with 10 threads.

But I guess it can do much better than that (IOW recover original rss).
Where does re-attribution takes place in np ?

Regards,
FabF

Re: rss recovery

[Nick Piggin]

FabF wrote:
> Nick,
> 	Putting some more pressure I finally saw the awaited behaviour from np
> : rss gaining 1MB (or at least 1 byte :) : top reports 10M -> 11M )
> directly after make was done with 10 threads.
> 
> But I guess it can do much better than that (IOW recover original rss).
> Where does re-attribution takes place in np ?
> 

I don't do any sort of preemptive RSS recovery. The pagein mechanisms
are unchanged with my patch. The point was that mozilla no longer got
swapped out by updatedb, isn't that what you wanted?

Re: rss recovery

[FabF]

On Sat, 2004-07-10 at 14:03, Nick Piggin wrote:
> FabF wrote:
> > Nick,
> > 	Putting some more pressure I finally saw the awaited behaviour from np
> > : rss gaining 1MB (or at least 1 byte :) : top reports 10M -> 11M )
> > directly after make was done with 10 threads.
> > 
> > But I guess it can do much better than that (IOW recover original rss).
> > Where does re-attribution takes place in np ?
> > 
> 
> I don't do any sort of preemptive RSS recovery. The pagein mechanisms
> are unchanged with my patch. The point was that mozilla no longer got
> swapped out by updatedb, isn't that what you wanted?
> 
Nick,

Your patch is great as system delves for pages without eating too much
RSS around.

I just thought about some sort of combination :

	-On one side a swapout regulation
	-But also somekind of smooth swapin operation.

Reason for this being box freeze effect after some heavy load.

I made a slight patchset which tries to do the second path.It's being
called RGR for "RSS Gradual Recovery".It works with 2 sysctl parameters
for testing :

	-swapoff_max_swapout :  It proceeds when kswapd has not reported more
than this.
	-swapoff_smooth_range : Number of pages to swap in at once.

That process uses a try_to_unuse patched version to emerge some pages
when swapout is relaxed.That stuff is done in a swap device transparent
poll method and should result in GUI application foregrounding done
quickly even after some heavy-load storm; my problem being where this
one can be called from ? As an example, I put a swapoff_smooth in
do_anonymous_page but it's not the right location I guess :))) just
wanted some place frequently called to see effects.

Of course, your help would be appreciated ;)

patchset is available from:
http://fabian.unixtech.be/ff/linux-2.6.7-mm7-rgr1.diff

Regards,
FabF