Re: Autoregulate swappiness & inactivation

Re: Autoregulate swappiness & inactivation

[deepfire]

Andrew Morton <akpm@osdl.org> wrote at Thu, 8 Jul 2004 09:44:06 -0700:
> Con Kolivas <kernel@kolivas.org> wrote:
> >
> > Here is another try at providing feedback to tune the vm_swappiness.
> 
> I spent some time yesterday trying to demonstrate performance improvements
> from those two patches.  Using
> 
> 	make -j4 vmlinux with mem=64m
> 
> and
> 
> 	qsbench -p 4 -m 96 with mem=256m
> 
> and was not able to do so, which is what I expected.
> 
> We do need more quantitative testing on this work.

i`ve done some of my favorite under-bridge-crafted tests, so to speak.

the test looked like this:

time find / -xdev | \
bzcat --compress | bzcat --decompress | \
bzcat --compress | bzcat --decompress | \
bzcat --compress | bzcat --decompress | \
cat > /dev/null

The patches applied were Con`s autoswap + autoregulate

it was performed on a p3-600, 10krpm scsi in two following scenarios:


1. mem=48M, swap=off

2.4.20-pre9:		3m20
2.6.7-con:		4m09
2.6.7-vanilla:		4m09


2. mem=32M, swap=on

2.4.20-pre9:		5m37
2.6.7-con:		6m37
2.6.7-vanilla:		7m31



which still leaves 2.4 the leader.


regards, Samium "2.4" Gromoff

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1.1: Type: text/plain, Size: 753 bytes --]

Con Kolivas writes:

> Andrew Morton writes:
> 
>> Con Kolivas <kernel@kolivas.org> wrote:
>>>
>>> > How about autoregulated swappiness, which seems to be very efficient at
>>>  > its job?
>>> 
>>>  It's been around for quite a while, and akpm has not expressed any 
>>>  interest in it so I think this will only ever flounder in the -ck domain.
>> 
>> Nobody sent me the patch.  And the
>> justification/explanation/sales-brochure.  And the benchmarks...
> 
> Ah what the heck. They can only be knocked back to where they already are.

Hmm the second patch doesn't appear complete against mm6. I'll have to 
re-create it once I can look at the code at home. Here is the version 
against vanilla so you can see what it's supposed to do. Apologies.

Con

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2.1: vm_autoregulate --]
[-- Type: text/plain, Size: 4654 bytes --]

Index: linux-2.6.7-ck/include/linux/swap.h
===================================================================
--- linux-2.6.7-ck.orig/include/linux/swap.h	2004-07-01 19:41:13.375151780 +1000
+++ linux-2.6.7-ck/include/linux/swap.h	2004-07-01 19:41:30.134517540 +1000
@@ -175,7 +175,7 @@
 extern int try_to_free_pages(struct zone **, unsigned int, unsigned int);
 extern int shrink_all_memory(int);
 extern int vm_swappiness;
-extern int auto_swappiness;
+extern int vm_autoregulate;
 
 #ifdef CONFIG_MMU
 /* linux/mm/shmem.c */
Index: linux-2.6.7-ck/include/linux/sysctl.h
===================================================================
--- linux-2.6.7-ck.orig/include/linux/sysctl.h	2004-07-01 19:41:13.376151623 +1000
+++ linux-2.6.7-ck/include/linux/sysctl.h	2004-07-01 19:41:30.136517226 +1000
@@ -166,7 +166,7 @@
 	VM_LAPTOP_MODE=23,	/* vm laptop mode */
 	VM_BLOCK_DUMP=24,	/* block dump mode */
 	VM_HUGETLB_GROUP=25,	/* permitted hugetlb group */
-	VM_AUTO_SWAPPINESS=26,	/* Make vm_swappiness autoregulated */
+	VM_AUTOREGULATE=26,     /* swappiness and inactivation autoregulated */
 };
 
 
Index: linux-2.6.7-ck/kernel/sysctl.c
===================================================================
--- linux-2.6.7-ck.orig/kernel/sysctl.c	2004-07-01 19:41:13.377151466 +1000
+++ linux-2.6.7-ck/kernel/sysctl.c	2004-07-01 19:41:30.137517069 +1000
@@ -744,9 +744,9 @@
 		.extra2		= &one_hundred,
 	},
 	{
-		.ctl_name	= VM_AUTO_SWAPPINESS,
-		.procname	= "autoswappiness",
-		.data		= &auto_swappiness,
+		.ctl_name	= VM_AUTOREGULATE,
+		.procname	= "autoregulate",
+		.data		= &vm_autoregulate,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
Index: linux-2.6.7-ck/mm/vmscan.c
===================================================================
--- linux-2.6.7-ck.orig/mm/vmscan.c	2004-07-01 19:41:13.378151309 +1000
+++ linux-2.6.7-ck/mm/vmscan.c	2004-07-01 19:45:01.086359211 +1000
@@ -43,7 +43,8 @@
  * From 0 .. 100.  Higher means more swappy.
  */
 int vm_swappiness = 60;
-int auto_swappiness = 1;
+int vm_autoregulate = 1;
+static int app_percent = 1;
 static long total_memory;
 
 #define lru_to_page(_head) (list_entry((_head)->prev, struct page, lru))
@@ -649,7 +650,9 @@
 	long mapped_ratio;
 	long distress;
 	long swap_tendency;
+	struct sysinfo i;
 
+	si_meminfo(&i);
 	lru_add_drain();
 	pgmoved = 0;
 	spin_lock_irq(&zone->lru_lock);
@@ -691,23 +694,21 @@
 	 */
 	mapped_ratio = (sc->nr_mapped * 100) / total_memory;
 
+	/*
+	 * app_percent is the percentage of physical ram used
+	 * by application pages.
+	 */
+	si_meminfo(&i);
 #ifdef CONFIG_SWAP
-	if (auto_swappiness) {
-		int app_percent;
-		struct sysinfo i;
-		
+	app_percent = 100 - ((i.freeram + get_page_cache_size() -
+		swapper_space.nrpages) / (i.totalram / 100));
+
+	if (vm_autoregulate) {
 		si_swapinfo(&i);
 			
 		if (likely(i.totalswap >= 100)) {
 			int swap_centile;
 	
-			/*
-			 * app_percent is the percentage of physical ram used
-			 * by application pages.
-			 */
-			si_meminfo(&i);
-			app_percent = 100 - ((i.freeram + get_page_cache_size() -
-				swapper_space.nrpages) / (i.totalram / 100));
 	
 			/*
 			 * swap_centile is the percentage of the last (sizeof physical
@@ -724,6 +725,9 @@
 		} else 
 			vm_swappiness = 0;
 	}
+#else
+	app_percent = 100 - ((i.freeram + get_page_cache_size()) / 
+		(i.totalram / 100));
 #endif
 	
 	/*
@@ -834,11 +838,16 @@
 static void
 shrink_zone(struct zone *zone, struct scan_control *sc)
 {
-	unsigned long scan_active, scan_inactive;
-	int count;
+	unsigned long scan_active, scan_inactive, biased_active;
+	int count, biased_ap;
 
 	scan_inactive = (zone->nr_active + zone->nr_inactive) >> sc->priority;
 
+	if (vm_autoregulate) {
+		biased_ap = app_percent * app_percent / 100;
+		biased_active = zone->nr_active / (101 - biased_ap) * 100;
+	} else
+		biased_active = zone->nr_active;
 	/*
 	 * Try to keep the active list 2/3 of the size of the cache.  And
 	 * make sure that refill_inactive is given a decent number of pages.
@@ -849,7 +858,7 @@
 	 * `scan_active' just to make sure that the kernel will slowly sift
 	 * through the active list.
 	 */
-	if (zone->nr_active >= 4*(zone->nr_inactive*2 + 1)) {
+	if (biased_active >= 4*(zone->nr_inactive*2 + 1)) {
 		/* Don't scan more than 4 times the inactive list scan size */
 		scan_active = 4*scan_inactive;
 	} else {
@@ -857,7 +866,7 @@
 
 		/* Cast to long long so the multiply doesn't overflow */
 
-		tmp = (unsigned long long)scan_inactive * zone->nr_active;
+		tmp = (unsigned long long)scan_inactive * biased_active;
 		do_div(tmp, zone->nr_inactive*2 + 1);
 		scan_active = (unsigned long)tmp;
 	}

[-- Attachment #2.2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

Nick Piggin writes:

> Con Kolivas wrote:
>> Andrew Morton writes:
>> 
>>> Con Kolivas <kernel@kolivas.org> wrote:
>>>
>>>>
>>>> > How about autoregulated swappiness, which seems to be very 
>>>> efficient at
>>>>  > its job?
>>>>
>>>>  It's been around for quite a while, and akpm has not expressed any 
>>>>  interest in it so I think this will only ever flounder in the -ck 
>>>> domain.
>>>
>>>
>>> Nobody sent me the patch.  And the
>>> justification/explanation/sales-brochure.  And the benchmarks...
>> 
>> 
>> Ah what the heck. They can only be knocked back to where they already are.
>> 
> 
> A few comments. I think making swappiness depend on the amount of
> swap you have used is not a good idea. I might be wrong though, but
> generally you should only make something *more* complex if you have
> a good rationale and good numbers (you have the later, Andrew might
> consider this enough). I especially don't like this sort of temporal
> dependancy either, because it makes things much harder to reproduce
> and think through.

Noted. The amount of swap hardly has any effect on the swappiness except 
when you're close to OOMing and it is harder to OOM with this in place.

> Secondly, can you please not mess with the exported sysctl. If you
> think your "autoswappiness" calculation is better than the current
> swappiness one, just completely replace it. Bonus points if you can
> retain the swappiness knob in some capacity.

I agree and would like them all removed, but people just love to leave the 
knobs in place. While I dont think the knobs should still be there either, 
I'm not reluctant to leave something that innocuous if the users want them.

> Numbers look good though. I'll get around to doing some tests soon.

Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

Con Kolivas wrote:
> Nick Piggin writes:

>> A few comments. I think making swappiness depend on the amount of
>> swap you have used is not a good idea. I might be wrong though, but
>> generally you should only make something *more* complex if you have
>> a good rationale and good numbers (you have the later, Andrew might
>> consider this enough). I especially don't like this sort of temporal
>> dependancy either, because it makes things much harder to reproduce
>> and think through.
> 
> 
> Noted. The amount of swap hardly has any effect on the swappiness except 
> when you're close to OOMing and it is harder to OOM with this in place.
> 

OK that's easy then. The OOM algorithm can be changed if it is
OOMing too easily.

>> Secondly, can you please not mess with the exported sysctl. If you
>> think your "autoswappiness" calculation is better than the current
>> swappiness one, just completely replace it. Bonus points if you can
>> retain the swappiness knob in some capacity.
> 
> 
> I agree and would like them all removed, but people just love to leave 
> the knobs in place. While I dont think the knobs should still be there 
> either, I'm not reluctant to leave something that innocuous if the users 
> want them.
> 

Well, get rid of the auto-tuning thing to start with, and merge
it into the swappiness calculation..

Regarding all these knobs, the main thing you want to avoid is
having loads of them because you can't find acceptable defaults.
I think "swappiness" is in the category of a good sysctl: it is
simple, meaningful to the admin, works, etc.

It has proven somewhat useful in testing ("set it to blah and see
if it still happens"). Or for people who know what they are doing.

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Andrew Morton writes:

> Con Kolivas <kernel@kolivas.org> wrote:
>>
>>  Ah what the heck. They can only be knocked back to where they already are.
> 
> hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
> nice numbers though.
> 
> How come vm_swappiness gets squared?  That's the mysterious "bias
> downwards", yes?  What's the theory there?

No real world feedback mechanism is linear. As the pressure grows the 
positive/negative feedback grows exponentially.

> Please define this new term "application pages"?

errm it's fuzzy to say the least. It's the closest I can come to 
representing what end users understand as "non-cached" pages.

> Those si_swapinfo() and si_meminfo() calls need to come out of there.

I'm game. I had the idea but not the skill. Anyone wanna help me with that?

> A diff against Documentation/filesystems/proc.txt will be needed sometime,
> please.

Ok. I'll try and put together one patch that does the lot.

Con


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

Nick Piggin writes:

> Con Kolivas wrote:
>> Nick Piggin writes:
> 
>>> A few comments. I think making swappiness depend on the amount of
>>> swap you have used is not a good idea. I might be wrong though, but
>>> generally you should only make something *more* complex if you have
>>> a good rationale and good numbers (you have the later, Andrew might
>>> consider this enough). I especially don't like this sort of temporal
>>> dependancy either, because it makes things much harder to reproduce
>>> and think through.
>> 
>> 
>> Noted. The amount of swap hardly has any effect on the swappiness except 
>> when you're close to OOMing and it is harder to OOM with this in place.
>> 
> 
> OK that's easy then. The OOM algorithm can be changed if it is
> OOMing too easily.

I didn't say it was easy, just harder with; but whatever - I can get rid of 
it.

>>> Secondly, can you please not mess with the exported sysctl. If you
>>> think your "autoswappiness" calculation is better than the current
>>> swappiness one, just completely replace it. Bonus points if you can
>>> retain the swappiness knob in some capacity.
>> 
>> 
>> I agree and would like them all removed, but people just love to leave 
>> the knobs in place. While I dont think the knobs should still be there 
>> either, I'm not reluctant to leave something that innocuous if the users 
>> want them.
>> 
> 
> Well, get rid of the auto-tuning thing to start with, and merge
> it into the swappiness calculation..
> 
> Regarding all these knobs, the main thing you want to avoid is
> having loads of them because you can't find acceptable defaults.
> I think "swappiness" is in the category of a good sysctl: it is
> simple, meaningful to the admin, works, etc.
> 
> It has proven somewhat useful in testing ("set it to blah and see
> if it still happens"). Or for people who know what they are doing.

Umm I think we're agreeing, no? I'm trying to leave the swappiness knob in 
for those who (think?) they know what they're doing. Somehow it needs to be 
turned to "manual" again.

Con


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Autoregulate swappiness & inactivation

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
> Andrew Morton writes:
> 
> > Con Kolivas <kernel@kolivas.org> wrote:
> >>
> >>  Ah what the heck. They can only be knocked back to where they already are.
> > 
> > hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
> > nice numbers though.
> > 
> > How come vm_swappiness gets squared?  That's the mysterious "bias
> > downwards", yes?  What's the theory there?
> 
> No real world feedback mechanism is linear. As the pressure grows the 
> positive/negative feedback grows exponentially.

That takes me back.  The classic control system is PID:
Proportional/Integral/Derivative - they refer to the way in which the error
term (output-desired output) is fed back to the input:

Proportional: the bigger the error, the more input drive

Integral: feeding back a bit of the integral of the error prevents
permanent output skew due to non-infinite forward gain.

Derivative: feeding back -(rate of change) provides damping.

You can live without I and D - the main thing is to feed back the -error.

IOW: linear works just fine :)

Your answer didn't help me understand the design though.

> > Please define this new term "application pages"?
> 
> errm it's fuzzy to say the least. It's the closest I can come to 
> representing what end users understand as "non-cached" pages.

Isn't that mapped pages?

> > Those si_swapinfo() and si_meminfo() calls need to come out of there.
> 
> I'm game. I had the idea but not the skill. Anyone wanna help me with that?

Need to work out what cen be removed first.  The freeswap/totalswap can go.
 That leaves us needing what?  totalram and freeram.  If the algorithm can
be flipped over to use nr_mapped, we'd be looking good.

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

Con Kolivas wrote:
> Nick Piggin writes:
> 

>> OK that's easy then. The OOM algorithm can be changed if it is
>> OOMing too easily.
> 
> 
> I didn't say it was easy, just harder with; but whatever - I can get rid 
> of it.
> 

Please.

>>>> Secondly, can you please not mess with the exported sysctl. If you
>>>> think your "autoswappiness" calculation is better than the current
>>>> swappiness one, just completely replace it. Bonus points if you can
>>>> retain the swappiness knob in some capacity.
>>>
>>>
>>>
>>> I agree and would like them all removed, but people just love to 
>>> leave the knobs in place. While I dont think the knobs should still 
>>> be there either, I'm not reluctant to leave something that innocuous 
>>> if the users want them.
>>>
>>
>> Well, get rid of the auto-tuning thing to start with, and merge
>> it into the swappiness calculation..
>>
>> Regarding all these knobs, the main thing you want to avoid is
>> having loads of them because you can't find acceptable defaults.
>> I think "swappiness" is in the category of a good sysctl: it is
>> simple, meaningful to the admin, works, etc.
>>
>> It has proven somewhat useful in testing ("set it to blah and see
>> if it still happens"). Or for people who know what they are doing.
> 
> 
> Umm I think we're agreeing, no? I'm trying to leave the swappiness knob 
> in for those who (think?) they know what they're doing. Somehow it needs 
> to be turned to "manual" again.
> 

No. Fold your all "autoswappiness" stuff directly into the
reclaim_mapped calculation that was previously keyed off swappiness.
Don't have it modify vm_swappiness at all: work directly on
reclaim_mapped.

Then, you should be able to retain the user's vm_swappiness input
into the system as well. If you can't figure out a good place to
put this in, don't worry about it to start with.

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

Andrew Morton writes:

> Con Kolivas <kernel@kolivas.org> wrote:
>>
>> Andrew Morton writes:
>> 
>> > Con Kolivas <kernel@kolivas.org> wrote:
>> >>
>> >>  Ah what the heck. They can only be knocked back to where they already are.
>> > 
>> > hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
>> > nice numbers though.
>> > 
>> > How come vm_swappiness gets squared?  That's the mysterious "bias
>> > downwards", yes?  What's the theory there?
>> 
>> No real world feedback mechanism is linear. As the pressure grows the 
>> positive/negative feedback grows exponentially.
> 
> That takes me back.  The classic control system is PID:
> Proportional/Integral/Derivative - they refer to the way in which the error
> term (output-desired output) is fed back to the input:
> 
> Proportional: the bigger the error, the more input drive
> 
> Integral: feeding back a bit of the integral of the error prevents
> permanent output skew due to non-infinite forward gain.
> 
> Derivative: feeding back -(rate of change) provides damping.
> 
> You can live without I and D - the main thing is to feed back the -error.
> 
> IOW: linear works just fine :)

/me hides

Umm sorry the control systems I look at are physiological and tend to be 
exponential, so ignore me.

> Your answer didn't help me understand the design though.

Ok I'll just describe it. I should have just said that when mapped pages are 
low the best seems to be a very low swappiness, but not zero as zero seems 
to get bogged down easily (kswapd gets busy and basic things take longer) as 
occasionally slipping some pages onto swap helps. Generally it's when what I 
called the application pages (blush) get to around 75% that allowing things 
to swap at the rate equivalent to swappiness==60 is helpful. Once the mapped 
pages went greater than 75% the machine would start bogging down again if 
the swappiness remained at 60. I guess I made up the maths to fill the way 
the design worked best. Linear brought up the swappiness too easily and 
under swap thrash made things worse. It looked nicer but didn't really 
behave well.

>> > Please define this new term "application pages"?
>> 
>> errm it's fuzzy to say the least. It's the closest I can come to 
>> representing what end users understand as "non-cached" pages.
> 
> Isn't that mapped pages?

I'm all ears.

>> > Those si_swapinfo() and si_meminfo() calls need to come out of there.
>> 
>> I'm game. I had the idea but not the skill. Anyone wanna help me with that?
> 
> Need to work out what cen be removed first.  The freeswap/totalswap can go.
>  That leaves us needing what?  totalram and freeram.  If the algorithm can
> be flipped over to use nr_mapped, we'd be looking good.

Ok. I need some time to clean up this mess and try and figure out what to do 
then.

Con

Re: Autoregulate swappiness & inactivation

[FabF]

On Thu, 2004-07-08 at 10:27, Con Kolivas wrote:
> Andrew Morton writes:
> 
> > Con Kolivas <kernel@kolivas.org> wrote:
> >>
> >> Andrew Morton writes:
> >> 
> >> > Con Kolivas <kernel@kolivas.org> wrote:
> >> >>
> >> >>  Ah what the heck. They can only be knocked back to where they already are.
> >> > 
> >> > hm.  You get an eGrump for sending two patchs in one email.  Surprisingly
> >> > nice numbers though.
> >> > 
> >> > How come vm_swappiness gets squared?  That's the mysterious "bias
> >> > downwards", yes?  What's the theory there?
> >> 
> >> No real world feedback mechanism is linear. As the pressure grows the 
> >> positive/negative feedback grows exponentially.
> > 
> > That takes me back.  The classic control system is PID:
> > Proportional/Integral/Derivative - they refer to the way in which the error
> > term (output-desired output) is fed back to the input:
> > 
> > Proportional: the bigger the error, the more input drive
> > 
> > Integral: feeding back a bit of the integral of the error prevents
> > permanent output skew due to non-infinite forward gain.
> > 
> > Derivative: feeding back -(rate of change) provides damping.
> > 
> > You can live without I and D - the main thing is to feed back the -error.
> > 
> > IOW: linear works just fine :)
> 
> /me hides
> 
> Umm sorry the control systems I look at are physiological and tend to be 
> exponential, so ignore me.
> 
> > Your answer didn't help me understand the design though.
> 
> Ok I'll just describe it. I should have just said that when mapped pages are 
> low the best seems to be a very low swappiness, but not zero as zero seems 
> to get bogged down easily (kswapd gets busy and basic things take longer) as 
> occasionally slipping some pages onto swap helps. Generally it's when what I 
> called the application pages (blush) get to around 75% that allowing things 
> to swap at the rate equivalent to swappiness==60 is helpful. Once the mapped 
> pages went greater than 75% the machine would start bogging down again if 
> the swappiness remained at 60. I guess I made up the maths to fill the way 
> the design worked best. Linear brought up the swappiness too easily and 
> under swap thrash made things worse. It looked nicer but didn't really 
> behave well.
> 
> >> > Please define this new term "application pages"?
> >> 
> >> errm it's fuzzy to say the least. It's the closest I can come to 
> >> representing what end users understand as "non-cached" pages.
> > 
> > Isn't that mapped pages?
> 
> I'm all ears.
> 
> >> > Those si_swapinfo() and si_meminfo() calls need to come out of there.
> >> 
> >> I'm game. I had the idea but not the skill. Anyone wanna help me with that?
> > 
> > Need to work out what cen be removed first.  The freeswap/totalswap can go.
> >  That leaves us needing what?  totalram and freeram.  If the algorithm can
> > be flipped over to use nr_mapped, we'd be looking good.
> 
> Ok. I need some time to clean up this mess and try and figure out what to do 
> then.
Con,
	What's interesting is try_to_free_pages comment :

" the zone may be full of dirty or under-writeback pages, which this
 * caller can't do much about.  We kick pdflush and take explicit naps
in the
 * hope that some of these pages can be written.  But if the allocating
task..."

	I mean do we have high activity profile of that side of the kernel when
bringing up some big application to life ?
	Does work consist here in 50% out, 50% in (time) ? Your anticipation
algorithm can help the "in" side but maybe we can optimize yet the "out"
side.btw, I'm surprised to see autoswappiness so far in fx tree:

page_reclaim
	try_to_free_pages
		shrink_caches
			shrink_zone
				refill_inactive_zone
					auto_swap calculation


IOW, does such parameter could not involve more decisions ?

Regards,
FabF

> 
> Con
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Autoregulate swappiness & inactivation

[Timothy Miller]

Con Kolivas wrote:

> /me hides
> 
> Umm sorry the control systems I look at are physiological and tend to be 
> exponential, so ignore me.

No.  I see no reason to disregard your understanding of biological 
control systems.  Millions of years of evolution have fine-tuned some 
very complex and robust control feedback systems.  While I wouldn't 
suggest that they're the only way to do the job, they're something that 
we should definately pay attention to.

Frankly, I think the cross-pollination that you bring from your 
background in medicine can do nothing but help us.

Re: Autoregulate swappiness & inactivation

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Nick Piggin wrote:
| Con Kolivas wrote:
|
|> Nick Piggin writes:
|> Umm I think we're agreeing, no? I'm trying to leave the swappiness
|> knob in for those who (think?) they know what they're doing. Somehow
|> it needs to be turned to "manual" again.
|>
|
| No. Fold your all "autoswappiness" stuff directly into the
| reclaim_mapped calculation that was previously keyed off swappiness.
| Don't have it modify vm_swappiness at all: work directly on
| reclaim_mapped.
|
| Then, you should be able to retain the user's vm_swappiness input
| into the system as well. If you can't figure out a good place to
| put this in, don't worry about it to start with.
|

Wasn't the point of this patch to allow the machine to tweak the
swappiness knob itself according to what it thinks is best, unless the
user tells it not to?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7X8RhDd4aOud5P8RAkNBAJ99+wIoTY1sHTTwOdO5fH8lggBpPgCfVFuv
Db7yGOwZjB+nTd6GxnM8KdM=
=/eIv
-----END PGP SIGNATURE-----

Re: Autoregulate swappiness & inactivation

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Timothy Miller wrote:
|
|
| Con Kolivas wrote:
|
|> /me hides
|>
|> Umm sorry the control systems I look at are physiological and tend to
|> be exponential, so ignore me.
|
|
| No.  I see no reason to disregard your understanding of biological
| control systems.  Millions of years of evolution have fine-tuned some
| very complex and robust control feedback systems.  While I wouldn't
| suggest that they're the only way to do the job, they're something that
| we should definately pay attention to.
|
| Frankly, I think the cross-pollination that you bring from your
| background in medicine can do nothing but help us.
|

Con is a medic?  . . . . shit.  I'm 5 years overdue for my
measles/mumps/rhubella shot

/me hides *cough*

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7YB6hDd4aOud5P8RApYOAJ46eBZIWctU6vZ1fJWVGOlX+HvhagCdEp8L
Ved/y9Wwb+8wJodTkZY4spY=
=wNu5
-----END PGP SIGNATURE-----

Re: Autoregulate swappiness & inactivation

[Timothy Miller]

John Richard Moser wrote:

> 
> Con is a medic?  . . . . shit.  I'm 5 years overdue for my
> measles/mumps/rhubella shot
> 
> /me hides *cough*


I'm not sure the term 'medic' quite does him justice.

Re: Autoregulate swappiness & inactivation

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Timothy Miller wrote:
|
| I'm not sure the term 'medic' quite does him justice.
|

Heh.  I didn't mean any offense.  :P

On a side note, Thunderbird keeps changing > to | for inline forwarding.
~ Any idea how to disable this?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA7b85hDd4aOud5P8RAk/4AJ9urtDoFyxaLyUk8WD9X4JryHl0IQCeNdyD
i0cUhDcTwjqd6uJiN6MIO/o=
=+zqR
-----END PGP SIGNATURE-----

Re: Autoregulate swappiness & inactivation

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

FabF wrote:
> Con,
> 	What's interesting is try_to_free_pages comment :
> 
> " the zone may be full of dirty or under-writeback pages, which this
>  * caller can't do much about.  We kick pdflush and take explicit naps
> in the
>  * hope that some of these pages can be written.  But if the allocating
> task..."
> 
> 	I mean do we have high activity profile of that side of the kernel when
> bringing up some big application to life ?
> 	Does work consist here in 50% out, 50% in (time) ? Your anticipation
> algorithm can help the "in" side but maybe we can optimize yet the "out"
> side.btw, I'm surprised to see autoswappiness so far in fx tree:
> 
> page_reclaim
> 	try_to_free_pages
> 		shrink_caches
> 			shrink_zone
> 				refill_inactive_zone
> 					auto_swap calculation
> 
> 
> IOW, does such parameter could not involve more decisions ?

If you put it that way, yes - it would classify as duct tape. However 
the code already acted based upon mapped_ratio which is pretty much all 
this patch does. Folded in in that sample patch I sent out earlier you 
can see that all it does is acted on mapped_ratio in a different manner 
so it's not really an extra layer at all.

-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	vm_swappiness = mapped_ratio * 150 / 100;
+	vm_swappiness = vm_swappiness * vm_swappiness / 150;
+	swap_tendency = distress + vm_swappiness;

Con

> Regards,
> FabF

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Autoregulate swappiness & inactivation

[Felipe Alfaro Solana]

On Thu, 2004-07-08 at 13:12 -0400, John Richard Moser wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> 
> Timothy Miller wrote:
> |
> |
> | Con Kolivas wrote:
> |
> |> /me hides
> |>
> |> Umm sorry the control systems I look at are physiological and tend to
> |> be exponential, so ignore me.
> |
> |
> | No.  I see no reason to disregard your understanding of biological
> | control systems.  Millions of years of evolution have fine-tuned some
> | very complex and robust control feedback systems.  While I wouldn't
> | suggest that they're the only way to do the job, they're something that
> | we should definately pay attention to.
> |
> | Frankly, I think the cross-pollination that you bring from your
> | background in medicine can do nothing but help us.
> |
> 
> Con is a medic?  . . . . shit.  I'm 5 years overdue for my
> measles/mumps/rhubella shot

He's a Doctor, IIRC.

Re: Autoregulate swappiness & inactivation

[FabF]

On Fri, 2004-07-09 at 03:05, Con Kolivas wrote:
> FabF wrote:
> > Con,
> > 	What's interesting is try_to_free_pages comment :
> > 
> > " the zone may be full of dirty or under-writeback pages, which this
> >  * caller can't do much about.  We kick pdflush and take explicit naps
> > in the
> >  * hope that some of these pages can be written.  But if the allocating
> > task..."
> > 
> > 	I mean do we have high activity profile of that side of the kernel when
> > bringing up some big application to life ?
> > 	Does work consist here in 50% out, 50% in (time) ? Your anticipation
> > algorithm can help the "in" side but maybe we can optimize yet the "out"
> > side.btw, I'm surprised to see autoswappiness so far in fx tree:
> > 
> > page_reclaim
> > 	try_to_free_pages
> > 		shrink_caches
> > 			shrink_zone
> > 				refill_inactive_zone
> > 					auto_swap calculation
> > 
> > 
> > IOW, does such parameter could not involve more decisions ?
> 
> If you put it that way, yes - it would classify as duct tape. However 
> the code already acted based upon mapped_ratio which is pretty much all 
> this patch does. Folded in in that sample patch I sent out earlier you 
> can see that all it does is acted on mapped_ratio in a different manner 
> so it's not really an extra layer at all.
> 
> -	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
> +	vm_swappiness = mapped_ratio * 150 / 100;
> +	vm_swappiness = vm_swappiness * vm_swappiness / 150;
> +	swap_tendency = distress + vm_swappiness;

Here's an easy benchmark to demonstrate problem :
1.Run Mozilla
2.Minimize
3=>Mozilla Resident Size (mrs) : 24Mb
4.Run updatedb
5.=>mrs : 15Mb
6.updatedb ends up
7.mrs doesn't move at all (yes, it goes down as I'm typing this msg :)).

So my question is :
Don't we have a way to say "whose pages were reclaimed from and
 reattribute its" ? (having in mind memory status per se).
IOW flushing (I guess it's pdflush relevant ? ) do work for dead
processes but doesn't care about applications alive...

Regards,
FabF

> 
> Con
> 
> > Regards,
> > FabF

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

FabF wrote:

> 
> Here's an easy benchmark to demonstrate problem :
> 1.Run Mozilla
> 2.Minimize
> 3=>Mozilla Resident Size (mrs) : 24Mb
> 4.Run updatedb
> 5.=>mrs : 15Mb
> 6.updatedb ends up
> 7.mrs doesn't move at all (yes, it goes down as I'm typing this msg :)).
> 

How much RAM do you have? Does this happen with and without Con's
patch?

I don't have a problem here with your problem, however I'm running
my -np patchset, which has different use-once heuristics.

> So my question is :
> Don't we have a way to say "whose pages were reclaimed from and
>  reattribute its" ? (having in mind memory status per se).
> IOW flushing (I guess it's pdflush relevant ? ) do work for dead
> processes but doesn't care about applications alive...
> 

Page reclaim doesn't really know or care about processes, it
basically works on a global page pool.

pdflush is used to perform writeout of dirty data, so it has
no part in reducing Mozilla's RSS.

I don't really understand what you are asking though. Your basic
problem is that mozilla's resident memory gets evicted too easily,
is that right?

Re: Autoregulate swappiness & inactivation

[FabF]

On Fri, 2004-07-09 at 12:43, Nick Piggin wrote:
> FabF wrote:
> 
> > 
> > Here's an easy benchmark to demonstrate problem :
> > 1.Run Mozilla
> > 2.Minimize
> > 3=>Mozilla Resident Size (mrs) : 24Mb
> > 4.Run updatedb
> > 5.=>mrs : 15Mb
> > 6.updatedb ends up
> > 7.mrs doesn't move at all (yes, it goes down as I'm typing this msg :)).
> > 
> 
> How much RAM do you have? Does this happen with and without Con's
> patch?
Hi Nick,

256Mb with Con's patch 1 (autoswappiness activated) mm6
but it's general behaviour on my box :(

> 
> I don't have a problem here with your problem, however I'm running
> my -np patchset, which has different use-once heuristics.
> 
> > So my question is :
> > Don't we have a way to say "whose pages were reclaimed from and
> >  reattribute its" ? (having in mind memory status per se).
> > IOW flushing (I guess it's pdflush relevant ? ) do work for dead
> > processes but doesn't care about applications alive...
> > 
> 
> Page reclaim doesn't really know or care about processes, it
> basically works on a global page pool.
That's exactly the nerve center of the problem I guess.
When we swap we don't care about different processes but when some of
its is going in, we _quickly_ need to refresh memory but isn't it too
late ? I mean what do we do here ? We recover pages and "get application
to life".Desktop side of the story reminds me about some oses giving
_impression_ all was alright.I mean there must be a way to anticipate
 such trouble without renice -xx all GUI relevant processes
 in order to have both server/client cfg synergy.
 
> 
> pdflush is used to perform writeout of dirty data, so it has
> no part in reducing Mozilla's RSS.
Oops ... kswapd then ?

> 
> I don't really understand what you are asking though. Your basic
> problem is that mozilla's resident memory gets evicted too easily,
> is that right?
> 
Not at all.My problem is mozilla has some MB to recover when
reactivating; meanwhile, I consider there was sufficient resource to
share with it _before_ reactivation as I'm waiting some minutes after an
heavy process (e.g updatedb) to be done and over.

AFAICS, Con's patches are about auto-regulation, not about anticipation
(?)

Regards,
FabF

Re: Autoregulate swappiness & inactivation

[Nick Piggin]

FabF wrote:
> On Fri, 2004-07-09 at 12:43, Nick Piggin wrote:

>>pdflush is used to perform writeout of dirty data, so it has
>>no part in reducing Mozilla's RSS.
> 
> Oops ... kswapd then ?
> 

Yep.

> 
>>I don't really understand what you are asking though. Your basic
>>problem is that mozilla's resident memory gets evicted too easily,
>>is that right?
>>
> 
> Not at all.My problem is mozilla has some MB to recover when
> reactivating; meanwhile, I consider there was sufficient resource to
> share with it _before_ reactivation as I'm waiting some minutes after an
> heavy process (e.g updatedb) to be done and over.
> 

You could try my -np7 patch, which would hopefully fix the problem
for you.

It may take some time and work before (if ever) the memory management
patches in it are merged though.

> AFAICS, Con's patches are about auto-regulation, not about anticipation
> (?)
> 

Yep. Con's patch changes the conditions that are required to start
reclaiming mapped pages. Basically: when to start swapping.

Re: Autoregulate swappiness & inactivation

[FabF]

On Fri, 2004-07-09 at 13:24, Nick Piggin wrote:
> FabF wrote:
> > On Fri, 2004-07-09 at 12:43, Nick Piggin wrote:
> 
> >>pdflush is used to perform writeout of dirty data, so it has
> >>no part in reducing Mozilla's RSS.
> > 
> > Oops ... kswapd then ?
> > 
> 
> Yep.
> 
> > 
> >>I don't really understand what you are asking though. Your basic
> >>problem is that mozilla's resident memory gets evicted too easily,
> >>is that right?
> >>
> > 
> > Not at all.My problem is mozilla has some MB to recover when
> > reactivating; meanwhile, I consider there was sufficient resource to
> > share with it _before_ reactivation as I'm waiting some minutes after an
> > heavy process (e.g updatedb) to be done and over.
> > 
> 
> You could try my -np7 patch, which would hopefully fix the problem
> for you.

Hi Nick:)
	 I've been busy benchmarking Con's autoregulation which _is_ proved
interesting in middle pressure.
AFAICS mm7np works that way : it cleans up memory brightly so we do see
attractive free memory available _but_ relevant rss doesn't move 1 byte
:( So we'll have foregrounding elevation for sure (no cleaning required)
but application still have to 'emerge' and that's the heavier side of
the story I'm afraid.

Regards,
FabF