Re: [2.4.17/18pre] VM and swap - it's really unusable

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Aviv Shavit]

Sorry for posting to the thread so late...

Stephan von Krawczynski Writes (Thu Jan 03 2002
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0201.0/0809.html):
>> A) VM has major issues 
>On all boxes I run currently (all 1GB or below RAM),
>I cannot find 
>_major_ issues. 

Ken Brownfield's A) and B) hit me
regularly(http://www.uwsg.indiana.edu/hypermail/linux/kernel/0201.0/0740.html)
(Thanks Ken for starting this enlightening thread and
to all those that contributed)

Running software I developed:
- running on 2.4.17
- accessing a large number of files
- 2GB memory
- large multiple partitions - ext2

I saw references to patches by Martin and
M.H.vanLeeuwen on this thread. Where can I get those
(hopefully with a bit of info) ?

I also saw references to 'rmap' on the 2.4.18
changelog. Is that related ?

pls. cc' me on your posts back.

Thanks
Aviv



"And all this science I don't understand,
      It's just my job five days a week"
Rocket Man, E.J.




__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ken Brownfield]

On Mon, Feb 25, 2002 at 03:51:43AM -0800, Aviv Shavit wrote:
| Ken Brownfield's A) and B) hit me
| regularly(http://www.uwsg.indiana.edu/hypermail/linux/kernel/0201.0/0740.html)
| (Thanks Ken for starting this enlightening thread and
| to all those that contributed)

No problem.  You might be coming in late, but nothing's changed. :-/
Although the APIC thing might be MPS1.4 related, I'm finding.  I don't
remember if that thread had my workaround to the APIC issue, so I'll
attach that if you're feeling adventurous.  Works flawlessly for me in
production, for the last few months.

| Running software I developed:
| - running on 2.4.17
| - accessing a large number of files
| - 2GB memory
| - large multiple partitions - ext2
| 
| I saw references to patches by Martin and
| M.H.vanLeeuwen on this thread. Where can I get those
| (hopefully with a bit of info) ?

I've attached his patch; hopefully he doesn't mind.  His patch is
actually similar to code that's in rmap currently.

It does seem to temper the effects of [id]cache bloat for some of my
more common load patterns.  But even with that patch, under VM load the
mainline kernel collapses.  For instance, I have a large parallel task
that takes 5 minutes under -rmap or -aa, but I get bored and kill it
after 4 *hours* on mainline.  It dips into swap, but only by about 50MB.

Also, -aa doesn't seem to make as large an impact on shrinking the
[id]caches.  I imagine that without returning values,
shrink_[id]cache_memory() behavior is difficult to tune appropriately.

In any case, rmap-12f has been running fine for me.  I'm going to create
three kernels to distribute in production -- 2.4.18 more or less
vanilla, 2.4.18 with random debugging (thanks Andreas) to try to resolve
the /dev/random death issue, and 2.4.18+O(1)K3+rmap12f.

rmap is a tremendous improvement over mainline -- really the only data I
still need is stability, both of O(1) and rmap.  Porting Andrea's 10_vm
was a pain the last time I did it, and it didn't have all of the
positives of rmap.

Clearly I'm not in the majority, though.  While I get bitten on a weekly
basis by the 2.4 VM, very few other people mention it.  Maybe they just
assume that it's normal behavior?  Scary.

Anyway, short of the long is that I would suggest rmap if you're having
problems.  Rmap actually obviates three other patches I typically apply.
But be aware that rmap is still a bit of a work in progress, at least in
terms of tuning.

Cheers,
-- 
Ken.
brownfld@irridia.com


| I also saw references to 'rmap' on the 2.4.18
| changelog. Is that related ?
| 
| pls. cc' me on your posts back.
| 
| Thanks
| Aviv
| 
| "And all this science I don't understand,
|       It's just my job five days a week"
| Rocket Man, E.J.


--- linux.virgin/mm/vmscan.c	Mon Dec 31 12:46:25 2001
+++ linux/mm/vmscan.c	Fri Jan 11 18:03:05 2002
@@ -394,9 +394,9 @@
 		if (PageDirty(page) && is_page_cache_freeable(page) && page->mapping) {
 			/*
 			 * It is not critical here to write it only if
-			 * the page is unmapped beause any direct writer
+			 * the page is unmapped because any direct writer
 			 * like O_DIRECT would set the PG_dirty bitflag
-			 * on the phisical page after having successfully
+			 * on the physical page after having successfully
 			 * pinned it and after the I/O to the page is finished,
 			 * so the direct writes to the page cannot get lost.
 			 */
@@ -480,11 +480,14 @@
 
 			/*
 			 * Alert! We've found too many mapped pages on the
-			 * inactive list, so we start swapping out now!
+			 * inactive list.
+			 * Move referenced pages to the active list.
 			 */
-			spin_unlock(&pagemap_lru_lock);
-			swap_out(priority, gfp_mask, classzone);
-			return nr_pages;
+			if (PageReferenced(page) && !PageLocked(page)) {
+				del_page_from_inactive_list(page);
+				add_page_to_active_list(page);
+			}
+			continue;
 		}
 
 		/*
@@ -521,6 +524,9 @@
 	}
 	spin_unlock(&pagemap_lru_lock);
 
+	if (max_mapped <= 0 && (nr_pages > 0 || priority < DEF_PRIORITY))
+		swap_out(priority, gfp_mask, classzone);
+
 	return nr_pages;
 }
 

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ken Brownfield]

I've been playing with 2.4.18 with the following primary patches with
very good results:

	O(1)-K3
	Andrew Morton's low-latency and read-latency2
	Martin's vmscan patch

In the past, I've been applying low-latency without enabling the CONFIG
option, silly me.  Why haven't Andrew's low latency patches gone into
2.4 or 2.5?  Let's pretty please with sugar on top and a cherry get
those into 2.4.19 even if -aa doesn't make it -- it's making a world of
difference in the stuff I'm throwing at the kernel, and I'm thinking -aa
and -rmap are giving improvements partially because they incorporate
some of these changes.

I think adding 10_vm would put the finishing touch on bridging the gap until
2.5, IMHO.  O(1) would just be the proverbial icing on the cake at a later
stage, since there still seems to be some work left to do on it (per-CPU
stuff?).

Now if TUX2 would only stop going into 99% CPU mode. ;)
-- 
Ken.
brownfld@irridia.com

On Mon, Feb 25, 2002 at 06:09:34PM -0600, Ken Brownfield wrote:
| On Mon, Feb 25, 2002 at 03:51:43AM -0800, Aviv Shavit wrote:
| | Ken Brownfield's A) and B) hit me
| | regularly(http://www.uwsg.indiana.edu/hypermail/linux/kernel/0201.0/0740.html)
| | (Thanks Ken for starting this enlightening thread and
| | to all those that contributed)
| 
| No problem.  You might be coming in late, but nothing's changed. :-/
| Although the APIC thing might be MPS1.4 related, I'm finding.  I don't
| remember if that thread had my workaround to the APIC issue, so I'll
| attach that if you're feeling adventurous.  Works flawlessly for me in
| production, for the last few months.
| 
| | Running software I developed:
| | - running on 2.4.17
| | - accessing a large number of files
| | - 2GB memory
| | - large multiple partitions - ext2
| | 
| | I saw references to patches by Martin and
| | M.H.vanLeeuwen on this thread. Where can I get those
| | (hopefully with a bit of info) ?
| 
| I've attached his patch; hopefully he doesn't mind.  His patch is
| actually similar to code that's in rmap currently.
| 
| It does seem to temper the effects of [id]cache bloat for some of my
| more common load patterns.  But even with that patch, under VM load the
| mainline kernel collapses.  For instance, I have a large parallel task
| that takes 5 minutes under -rmap or -aa, but I get bored and kill it
| after 4 *hours* on mainline.  It dips into swap, but only by about 50MB.
| 
| Also, -aa doesn't seem to make as large an impact on shrinking the
| [id]caches.  I imagine that without returning values,
| shrink_[id]cache_memory() behavior is difficult to tune appropriately.
| 
| In any case, rmap-12f has been running fine for me.  I'm going to create
| three kernels to distribute in production -- 2.4.18 more or less
| vanilla, 2.4.18 with random debugging (thanks Andreas) to try to resolve
| the /dev/random death issue, and 2.4.18+O(1)K3+rmap12f.
| 
| rmap is a tremendous improvement over mainline -- really the only data I
| still need is stability, both of O(1) and rmap.  Porting Andrea's 10_vm
| was a pain the last time I did it, and it didn't have all of the
| positives of rmap.
| 
| Clearly I'm not in the majority, though.  While I get bitten on a weekly
| basis by the 2.4 VM, very few other people mention it.  Maybe they just
| assume that it's normal behavior?  Scary.
| 
| Anyway, short of the long is that I would suggest rmap if you're having
| problems.  Rmap actually obviates three other patches I typically apply.
| But be aware that rmap is still a bit of a work in progress, at least in
| terms of tuning.
| 
| Cheers,
| -- 
| Ken.
| brownfld@irridia.com
| 
| 
| | I also saw references to 'rmap' on the 2.4.18
| | changelog. Is that related ?
| | 
| | pls. cc' me on your posts back.
| | 
| | Thanks
| | Aviv
| | 
| | "And all this science I don't understand,
| |       It's just my job five days a week"
| | Rocket Man, E.J.
| 
| 
| --- linux.virgin/mm/vmscan.c	Mon Dec 31 12:46:25 2001
| +++ linux/mm/vmscan.c	Fri Jan 11 18:03:05 2002
| @@ -394,9 +394,9 @@
|  		if (PageDirty(page) && is_page_cache_freeable(page) && page->mapping) {
|  			/*
|  			 * It is not critical here to write it only if
| -			 * the page is unmapped beause any direct writer
| +			 * the page is unmapped because any direct writer
|  			 * like O_DIRECT would set the PG_dirty bitflag
| -			 * on the phisical page after having successfully
| +			 * on the physical page after having successfully
|  			 * pinned it and after the I/O to the page is finished,
|  			 * so the direct writes to the page cannot get lost.
|  			 */
| @@ -480,11 +480,14 @@
|  
|  			/*
|  			 * Alert! We've found too many mapped pages on the
| -			 * inactive list, so we start swapping out now!
| +			 * inactive list.
| +			 * Move referenced pages to the active list.
|  			 */
| -			spin_unlock(&pagemap_lru_lock);
| -			swap_out(priority, gfp_mask, classzone);
| -			return nr_pages;
| +			if (PageReferenced(page) && !PageLocked(page)) {
| +				del_page_from_inactive_list(page);
| +				add_page_to_active_list(page);
| +			}
| +			continue;
|  		}
|  
|  		/*
| @@ -521,6 +524,9 @@
|  	}
|  	spin_unlock(&pagemap_lru_lock);
|  
| +	if (max_mapped <= 0 && (nr_pages > 0 || priority < DEF_PRIORITY))
| +		swap_out(priority, gfp_mask, classzone);
| +
|  	return nr_pages;
|  }
|  
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to majordomo@vger.kernel.org
| More majordomo info at  http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at  http://www.tux.org/lkml/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Aviv Shavit]

late as usual ....

thanks Ken.

Just got to testng rmap-12f on 2.4.17. While I was
able to throw at it more sessions then at the vannila
kernel, it crashed with a kernel panic. Something that
has not happend for main-line .17 nor for .18.

I use the term sessions since the bandwidth I throw at
the machine is more or less a constant flooded
fastether link at 80Mb/s. The more the sessions - the
more the memory used and more files accessed.

18aa works great ! I love seeing how the cached memory
actually is reduced as the number of sessions grow. ;)

I've been trying to isolate what it is in aa that
helps me so. Tried the _vm_ batch - but they didn't
seem to score. 

Perhaps this info helps someone.

Thanks again.



--- Ken Brownfield <brownfld@irridia.com> wrote:
> I've been playing with 2.4.18 with the following
> primary patches with
> very good results:
> 
> 	O(1)-K3
> 	Andrew Morton's low-latency and read-latency2
> 	Martin's vmscan patch
> 
> In the past, I've been applying low-latency without
> enabling the CONFIG
> option, silly me.  Why haven't Andrew's low latency
> patches gone into
> 2.4 or 2.5?  Let's pretty please with sugar on top
> and a cherry get
> those into 2.4.19 even if -aa doesn't make it --
> it's making a world of
> difference in the stuff I'm throwing at the kernel,
> and I'm thinking -aa
> and -rmap are giving improvements partially because
> they incorporate
> some of these changes.
> 
> I think adding 10_vm would put the finishing touch
> on bridging the gap until
> 2.5, IMHO.  O(1) would just be the proverbial icing
> on the cake at a later
> stage, since there still seems to be some work left
> to do on it (per-CPU
> stuff?).
> 
> Now if TUX2 would only stop going into 99% CPU mode.
> ;)
> -- 
> Ken.
> brownfld@irridia.com
> 
> On Mon, Feb 25, 2002 at 06:09:34PM -0600, Ken
> Brownfield wrote:
> | On Mon, Feb 25, 2002 at 03:51:43AM -0800, Aviv
> Shavit wrote:
> | | Ken Brownfield's A) and B) hit me
> | |
>
regularly(http://www.uwsg.indiana.edu/hypermail/linux/kernel/0201.0/0740.html)
> | | (Thanks Ken for starting this enlightening
> thread and
> | | to all those that contributed)
> | 
> | No problem.  You might be coming in late, but
> nothing's changed. :-/
> | Although the APIC thing might be MPS1.4 related,
> I'm finding.  I don't
> | remember if that thread had my workaround to the
> APIC issue, so I'll
> | attach that if you're feeling adventurous.  Works
> flawlessly for me in
> | production, for the last few months.
> | 
> | | Running software I developed:
> | | - running on 2.4.17
> | | - accessing a large number of files
> | | - 2GB memory
> | | - large multiple partitions - ext2
> | | 
> | | I saw references to patches by Martin and
> | | M.H.vanLeeuwen on this thread. Where can I get
> those
> | | (hopefully with a bit of info) ?
> | 
> | I've attached his patch; hopefully he doesn't
> mind.  His patch is
> | actually similar to code that's in rmap currently.
> | 
> | It does seem to temper the effects of [id]cache
> bloat for some of my
> | more common load patterns.  But even with that
> patch, under VM load the
> | mainline kernel collapses.  For instance, I have a
> large parallel task
> | that takes 5 minutes under -rmap or -aa, but I get
> bored and kill it
> | after 4 *hours* on mainline.  It dips into swap,
> but only by about 50MB.
> | 
> | Also, -aa doesn't seem to make as large an impact
> on shrinking the
> | [id]caches.  I imagine that without returning
> values,
> | shrink_[id]cache_memory() behavior is difficult to
> tune appropriately.
> | 
> | In any case, rmap-12f has been running fine for
> me.  I'm going to create
> | three kernels to distribute in production --
> 2.4.18 more or less
> | vanilla, 2.4.18 with random debugging (thanks
> Andreas) to try to resolve
> | the /dev/random death issue, and
> 2.4.18+O(1)K3+rmap12f.
> | 
> | rmap is a tremendous improvement over mainline --
> really the only data I
> | still need is stability, both of O(1) and rmap. 
> Porting Andrea's 10_vm
> | was a pain the last time I did it, and it didn't
> have all of the
> | positives of rmap.
> | 
> | Clearly I'm not in the majority, though.  While I
> get bitten on a weekly
> | basis by the 2.4 VM, very few other people mention
> it.  Maybe they just
> | assume that it's normal behavior?  Scary.
> | 
> | Anyway, short of the long is that I would suggest
> rmap if you're having
> | problems.  Rmap actually obviates three other
> patches I typically apply.
> | But be aware that rmap is still a bit of a work in
> progress, at least in
> | terms of tuning.
> | 
> | Cheers,
> | -- 
> | Ken.
> | brownfld@irridia.com
> | 
> | 
> | | I also saw references to 'rmap' on the 2.4.18
> | | changelog. Is that related ?
> | | 
> | | pls. cc' me on your posts back.
> | | 
> | | Thanks
> | | Aviv
> | | 
> | | "And all this science I don't understand,
> | |       It's just my job five days a week"
> | | Rocket Man, E.J.
> | 
> | 
> | --- linux.virgin/mm/vmscan.c	Mon Dec 31 12:46:25
> 2001
> | +++ linux/mm/vmscan.c	Fri Jan 11 18:03:05 2002
> | @@ -394,9 +394,9 @@
> |  		if (PageDirty(page) &&
> is_page_cache_freeable(page) && page->mapping) {
> |  			/*
> |  			 * It is not critical here to write it only if
> | -			 * the page is unmapped beause any direct
> writer
> | +			 * the page is unmapped because any direct
> writer
> |  			 * like O_DIRECT would set the PG_dirty
> bitflag
> | -			 * on the phisical page after having
> successfully
> | +			 * on the physical page after having
> successfully
> |  			 * pinned it and after the I/O to the page is
> finished,
> |  			 * so the direct writes to the page cannot get
> lost.
> |  			 */
> | @@ -480,11 +480,14 @@
> |  
> |  			/*
> |  			 * Alert! We've found too many mapped pages on
> the
> | -			 * inactive list, so we start swapping out
> now!
> | +			 * inactive list.
> | +			 * Move referenced pages to the active list.
> |  			 */
> | -			spin_unlock(&pagemap_lru_lock);
> | -			swap_out(priority, gfp_mask, classzone);
> | -			return nr_pages;
> | +			if (PageReferenced(page) && !PageLocked(page))
> {
> | +				del_page_from_inactive_list(page);
> | +				add_page_to_active_list(page);
> | +			}
> | +			continue;
> |  		}
> |  
> |  		/*
> | @@ -521,6 +524,9 @@
> |  	}
> |  	spin_unlock(&pagemap_lru_lock);
> |  
> | +	if (max_mapped <= 0 && (nr_pages > 0 || priority
> < DEF_PRIORITY))
> | +		swap_out(priority, gfp_mask, classzone);
> | +
> |  	return nr_pages;
> |  }
> |  
> | -
> | To unsubscribe from this list: send the line
> "unsubscribe linux-kernel" in
> | the body of a message to majordomo@vger.kernel.org
> | More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> 
=== message truncated ===



__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rik van Riel]

On Tue, 9 Apr 2002, Aviv Shavit wrote:

> Just got to testng rmap-12f on 2.4.17. While I was
> able to throw at it more sessions then at the vannila
> kernel, it crashed with a kernel panic. Something that
> has not happend for main-line .17 nor for .18.

Something I haven't seen either.  Do you have the message
your kernel spit out so I can have a go at fixing the problem ?

kind regards,

Rik
-- 
Will hack the VM for food.

http://www.surriel.com/		http://distro.conectiva.com/

vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

I recommend everybody to never use a 2.4 kernel without first applying
this vm patch:

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/vm-33.gz

It applies cleanly to both 2.4.19pre5 and 2.4.19pre6. Andrew splitted it
into orthogonal pieces for easy merging from Marcelo's side (modulo
-rest that is important too but that it's still quite monolithic, but
it's pointless to invest further effort at this time until we are
certain Marcelo will do its job and eventually merge it in mainline):

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/

So far a first part of those patches is been merged into mainline into
pre5 (not any previous kernel, if you've some problem reproducible with
pre4 pre3 pre2 and pre1 or any previous kernel that's not related to the
async flushing changes, I seen a bogus report floating around to Marcelo
about pre1 pointing to the vm changes, it can't be the vm changes if
it's pre[1234]).

This VM is under heavy stressing for weeks on my SMP highmem machine
with a real life DBMS workload in a real life setup with huge VM
pressure with mem=1024m and 1.2G of shm pushed in swap constantly by the
kernel, performance of the workload is now very good and exactly
reproducible and constant, so I recommend it for all production systems
(both lowmem desktops and highend servers).

Alternatively you can use the whole -aa patchkit, to get all the other
critical highend features like pte-highmem, highio etc...

I haven't bugreports pending on the vm patch.

Thanks,

Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Richard Gooch]

Andrea Arcangeli writes:
> I recommend everybody to never use a 2.4 kernel without first applying
> this vm patch:
[...]

The way you write this makes it sound that the unpatched kernel is
very dangerous. Is this actually true? Or do you really just mean "the
patched kernel has better handling under extreme loads"?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

On Tue, Apr 09, 2002 at 06:07:50PM -0600, Richard Gooch wrote:
> Andrea Arcangeli writes:
> > I recommend everybody to never use a 2.4 kernel without first applying
> > this vm patch:
> [...]
> 
> The way you write this makes it sound that the unpatched kernel is
> very dangerous. Is this actually true? Or do you really just mean "the
> patched kernel has better handling under extreme loads"?

The unpatched kernel isn't dangerous in the sense it won't destroy data,
it won't corrupt memory and finally it won't deadlock on smp locks, but
it can theoretically deadlock with oom and it has various other runtime
issues starting from highmem balancing, too much swapping, lru list
balancing, related-bhs in highmem, numa broken with += min etc... so
IMHO it is better to _always_ use the patched kernel that takes care of
all problems that I know of at the moment, plus it has further
optimizations. OTOH for lots of workloads mainline is just fine, the
deadlocks never trigger and the runtime behaviour is ok, but unless you
are certain you don't need the vm-33.gz patch, I recommend to apply it.

Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[shane]

Hi,

I too can recommend the -aa kernels. I have been using the -aa kernels
since 2.4.10, and IMHO they are just as stable if not more stable than
the vanilla kernels. The difference is in the performance, especially
the swapping as Andrea noted. Specifically they are better at I/O, both
throughput and interactive responsiveness during heavy I/O. I have also
seen the best utilization on my little 100/switched network using these
kernels. I also use -aa on my Thinkpad and it works great there too. I
haven't tried the XFS but I did try tux and it just smokes!

Anyways just my 2 cents,

Shane

BTW: Big thanks to Andrew Morton for stepping up and splitting the VM
pieces for merging.  

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Christoph Hellwig]

On Wed, Apr 10, 2002 at 01:36:09AM +0200, Andrea Arcangeli wrote:
> I recommend everybody to never use a 2.4 kernel without first applying
> this vm patch:
> 
> 	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/vm-33.gz
> 
> It applies cleanly to both 2.4.19pre5 and 2.4.19pre6. Andrew splitted it
> into orthogonal pieces for easy merging from Marcelo's side (modulo
> -rest that is important too but that it's still quite monolithic, but
> it's pointless to invest further effort at this time until we are
> certain Marcelo will do its job and eventually merge it in mainline):

Maybe you could start by explaining the questions asked by akpm ontop of
many of the split pages that you even merged into -aa?  Or by explaining why
you reverse all functional changes akpm did in the -rest patch?

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Richard Gooch]

Andrea Arcangeli writes:
> On Tue, Apr 09, 2002 at 06:07:50PM -0600, Richard Gooch wrote:
> > Andrea Arcangeli writes:
> > > I recommend everybody to never use a 2.4 kernel without first applying
> > > this vm patch:
> > [...]
> > 
> > The way you write this makes it sound that the unpatched kernel is
> > very dangerous. Is this actually true? Or do you really just mean "the
> > patched kernel has better handling under extreme loads"?
> 
> The unpatched kernel isn't dangerous in the sense it won't destroy
> data, it won't corrupt memory and finally it won't deadlock on smp
> locks, but it can theoretically deadlock with oom and it has various
> other runtime issues starting from highmem balancing, too much
> swapping, lru list balancing, related-bhs in highmem, numa broken
> with += min etc... so IMHO it is better to _always_ use the patched
> kernel that takes care of all problems that I know of at the moment,
> plus it has further optimizations. OTOH for lots of workloads
> mainline is just fine, the deadlocks never trigger and the runtime
> behaviour is ok, but unless you are certain you don't need the
> vm-33.gz patch, I recommend to apply it.

So, in other words, 99.99% of users don't need to apply the patch.
They should, in order to have a better system, but 99.99% of them
won't notice the difference. That seems to be a more honest
recommendation than the "panic stations" alert that you posted.

Just because you want people to apply your patches, doesn't mean you
should resort to alarmist-sounding messages. Let's at least have truth
in advertising in one small corner of the world :-)

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

On Wed, Apr 10, 2002 at 04:40:22PM -0600, Richard Gooch wrote:
> Just because you want people to apply your patches, doesn't mean you
> should resort to alarmist-sounding messages. Let's at least have truth
> in advertising in one small corner of the world :-)

It's not a 99.99% that doesn't need it, I'd say a 5% would strictly need
it. There are various known problems, and they're all fixed by such
patch AFIK so unless you want to risk you _should_ apply it, that's the
whole point of the previous email.

Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Ken Brownfield]

This sounds great, but I still have concerns with using -aa, or subsets
of same.

How much of the improved behavior that you're seeing is due to the vm-33
tweaks and not pte-highmem, block-highmem, or any of the 100 or so other
2.4.19-pre6aa1 patches?

For production use, I prefer to divert from mainline only with my
specific needs (or trivial fixes).  Using -aa would introduce a large
array of (to me) unknowns.  How many of the -aa patches are "ready" for
mainline and production?  vm is currently being debated on the floor --
but what about pte-highmem and block-highmem?  How many of the other
patches are as widely tested as the vm patch?  For some reason, applying
a patch called "00_readahead-got-broken-somewhere-1" doesn't give me the
utmost confidence in production.  Call it a failed bag check.

While 2.4.x is a stable kernel, it needs to be a working* kernel until
2.5 can sort out these and the vast array of other issues.  IMHO.
*Admittedly, "working" in this case only applies to larger servers, but
it would be quite tragic to delay the spread of Linux to hardware that's
been available and used in production for _years_.  Maybe 5% of the
installed base has relevant hardware, but the benefit to Linux _far_
outstrips that seemingly anemic number.  I've probably rehashed that
point too much as it is, but...

What I'd like to hear (and what I suspect many admins trying to get
higher-end hardware working optimally in a production environment would
like to hear) is what specific patches applied to mainline are needed to
correct the current VM and I/O issues in the 2.4 tree?

If it's vm, pte-highmem, and block-highmem, that's fine -- and separable
from -aa.  Otherwise it's difficult to get people to test, use, and
provide feedback that isn't polluted by unnecessary variables.

Thanks,
-- 
Ken.
ken@irridia.com


On Wed, Apr 10, 2002 at 01:36:09AM +0200, Andrea Arcangeli wrote:
| I recommend everybody to never use a 2.4 kernel without first applying
| this vm patch:
| 
| 	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/vm-33.gz
| 
| It applies cleanly to both 2.4.19pre5 and 2.4.19pre6. Andrew splitted it
| into orthogonal pieces for easy merging from Marcelo's side (modulo
| -rest that is important too but that it's still quite monolithic, but
| it's pointless to invest further effort at this time until we are
| certain Marcelo will do its job and eventually merge it in mainline):
| 
| 	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.19pre5/
| 
| So far a first part of those patches is been merged into mainline into
| pre5 (not any previous kernel, if you've some problem reproducible with
| pre4 pre3 pre2 and pre1 or any previous kernel that's not related to the
| async flushing changes, I seen a bogus report floating around to Marcelo
| about pre1 pointing to the vm changes, it can't be the vm changes if
| it's pre[1234]).
| 
| This VM is under heavy stressing for weeks on my SMP highmem machine
| with a real life DBMS workload in a real life setup with huge VM
| pressure with mem=1024m and 1.2G of shm pushed in swap constantly by the
| kernel, performance of the workload is now very good and exactly
| reproducible and constant, so I recommend it for all production systems
| (both lowmem desktops and highend servers).
| 
| Alternatively you can use the whole -aa patchkit, to get all the other
| critical highend features like pte-highmem, highio etc...
| 
| I haven't bugreports pending on the vm patch.
| 
| Thanks,
| 
| Andrea

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Aviv Shavit]

This goes back to my previous post - 
Applying only the vm patches didn't get me far.

I'm still trying to pin point what it is thats helping
me out in -aa

--- Ken Brownfield <ken@irridia.com> wrote: 
:
:
> 
> How much of the improved behavior that you're seeing
> is due to the vm-33
> tweaks and not pte-highmem, block-highmem, or any of
> the 100 or so other
> 2.4.19-pre6aa1 patches?
:
:
> 
> What I'd like to hear (and what I suspect many
> admins trying to get
> higher-end hardware working optimally in a
> production environment would
> like to hear) is what specific patches applied to
> mainline are needed to
> correct the current VM and I/O issues in the 2.4
> tree?
> 
> If it's vm, pte-highmem, and block-highmem, that's
> fine -- and separable
> from -aa.  Otherwise it's difficult to get people to
> test, use, and
> provide feedback that isn't polluted by unnecessary
> variables.
> 
> Thanks,
> -- 
> Ken.
> ken@irridia.com
> 



__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/

Re: vm-33, strongly recommended [Re: [2.4.17/18pre] VM and swap - it's really unusable]

[Andrea Arcangeli]

On Thu, Apr 11, 2002 at 04:50:15PM -0700, Aviv Shavit wrote:
> This goes back to my previous post - 
> Applying only the vm patches didn't get me far.
> 
> I'm still trying to pin point what it is thats helping
> me out in -aa

For the level of cache during your workload (you mentioned that variable
in the previous email) what matters mostly is the vm-33 patch. Do you
get significantly different levels of cache with only the vm-33 patch
compared to the whole latest -aa? (there are other variables too that
could influence the level of cache, the readahead boost for example, but
they're much less likely to influence the cache levels than the vm-33
patch)

It maybe the benefit you see is not only in the VM part, but it could
came also the dozen of other fixes and improvements. For example
starting from the lowlatency fixes from Andrew (note: _fixes_) to highio
(from Jens) if you've highmem, to the dyn-sched (from Davide) if you've
tons of sleeping tasks or interactive processes etc...

The reason I maintain the main patches like the vm one also against
mainline, is exactly to address Ken's concern about being able to apply
just one patch if he's not confortable with the whole patchkit, and
secondly to be able to test it separately without the pollution. I feel
the vm patch is one of the most important and that's why I mentioned it
in particular, but the lowlatency fixes and lots of other stuff is
important too. But I've also the feeling the other stuff [modulo the
major things like pte-highmem and highio that at least affects only the
x86 high-end and not that much desktops or little server] is much much
easier to get integrated and that's why I worry much less about it.

Thanks for all the feedback and testing!

Andrea