hints at modifying kswapd params in 2.4.16

hints at modifying kswapd params in 2.4.16

[Sven Heinicke]

We have been having many problems on a Dell PowerEdge 4400 with 4G of
memory.  We are willing to get new hardware if, perhaps, there is
something known to not work with that hardwares memory.  If there are
known hardware configurations that work will in high load, high IO
situations.  We really want to stay with Linux, but my boss is getting
increasingly agitated with issues.

Our application has, on the order of 300 network sockets open at any
one time primarily for input.  And only 1 or 2 for output.  We
constantly malloc/free 100k of memory, I mean a lot.  Plus do a fair
amount of SCSI IO.  The threads never use more then like 300M of
memory at a time though caching memory fills up to like 2G.  I believe
it would work on a system with 1G of memory.

With kernels 2.4.14 and before, including some AA kernels, the cache
would fill up memory and do a little swapping to disk and then start
the just-in-time free memory stuff.  We where very happy when 2.4.16
seems smarter about this then previous version and never uses more
then ~2G of cachememory.

But, after running for a time, kswapd starts taking more CPU time then
the threads we are running and slowing down the processing.  Is this
something wrong with kswapd or might modifying files in /proc/sys/vm/
fix this?

If modifying files in /proc/sys/vm/ can fix it, what should one
consider when modifying the /proc/sys/vm/kswapd and other files to
suit my applications?  Also, in my /proc/sys/vm directory there is
only the following files:

$ ls /proc/sys/vm
bdflush  max-readahead  overcommit_memory  pagetable_cache
kswapd   min-readahead  page-cluster

With 2.4.16 I though I should have others as described in the proc.txt
file, like `buffermem', `pagecache' and others.  How do I get these,
and once I do have them some hints on how to modify there values to
optimize my applications might be useful.

	 Sven

Re: hints at modifying kswapd params in 2.4.16

[Ken Brownfield]

Yes, 2.4.16 fixed a major source of kswapd thrashing for (at least)
mmapped memory, but there are still some corner cases left which I've
also seen.  For instance, in the new case I'm seeing kswapd thrashing
even when freepages hasn't been reached under heavy I/O.  I'm under the
impression that this behavior is more of a mis-behavior rather than a
mis-tuning.

I don't think I've seen a response to my recent post go by; hopefully
the VM issues can still be debugged soon even after the handoff to
Marcelo... :-)
-- 
Ken.
brownfld@irridia.com

PS: The hardware in question on my end is an HP LH6000r, 6-way Xeon, 4GB.

On Mon, Dec 03, 2001 at 01:00:41PM -0500, Sven Heinicke wrote:
| 
| We have been having many problems on a Dell PowerEdge 4400 with 4G of
| memory.  We are willing to get new hardware if, perhaps, there is
| something known to not work with that hardwares memory.  If there are
| known hardware configurations that work will in high load, high IO
| situations.  We really want to stay with Linux, but my boss is getting
| increasingly agitated with issues.
| 
| Our application has, on the order of 300 network sockets open at any
| one time primarily for input.  And only 1 or 2 for output.  We
| constantly malloc/free 100k of memory, I mean a lot.  Plus do a fair
| amount of SCSI IO.  The threads never use more then like 300M of
| memory at a time though caching memory fills up to like 2G.  I believe
| it would work on a system with 1G of memory.
| 
| With kernels 2.4.14 and before, including some AA kernels, the cache
| would fill up memory and do a little swapping to disk and then start
| the just-in-time free memory stuff.  We where very happy when 2.4.16
| seems smarter about this then previous version and never uses more
| then ~2G of cachememory.
| 
| But, after running for a time, kswapd starts taking more CPU time then
| the threads we are running and slowing down the processing.  Is this
| something wrong with kswapd or might modifying files in /proc/sys/vm/
| fix this?
| 
| If modifying files in /proc/sys/vm/ can fix it, what should one
| consider when modifying the /proc/sys/vm/kswapd and other files to
| suit my applications?  Also, in my /proc/sys/vm directory there is
| only the following files:
| 
| $ ls /proc/sys/vm
| bdflush  max-readahead  overcommit_memory  pagetable_cache
| kswapd   min-readahead  page-cluster
| 
| With 2.4.16 I though I should have others as described in the proc.txt
| file, like `buffermem', `pagecache' and others.  How do I get these,
| and once I do have them some hints on how to modify there values to
| optimize my applications might be useful.
| 
| 	 Sven
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to majordomo@vger.kernel.org
| More majordomo info at  http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at  http://www.tux.org/lkml/

Re: hints at modifying kswapd params in 2.4.16

[Sven Heinicke]

Does the AC kernel still have the old VM?  I really wanna stick the
the new stuff but but a need a stable system.  Older kernels 2.4.8 had
highmem issues, not the 2.4.16 has kswap issues.

Ken Brownfield writes:
 > Yes, 2.4.16 fixed a major source of kswapd thrashing for (at least)
 > mmapped memory, but there are still some corner cases left which I've
 > also seen.  For instance, in the new case I'm seeing kswapd thrashing
 > even when freepages hasn't been reached under heavy I/O.  I'm under the
 > impression that this behavior is more of a mis-behavior rather than a
 > mis-tuning.
 > 
 > I don't think I've seen a response to my recent post go by; hopefully
 > the VM issues can still be debugged soon even after the handoff to
 > Marcelo... :-)
 > -- 
 > Ken.
 > brownfld@irridia.com
 > 
 > PS: The hardware in question on my end is an HP LH6000r, 6-way Xeon, 4GB.
 > 
 > On Mon, Dec 03, 2001 at 01:00:41PM -0500, Sven Heinicke wrote:
 > | 
 > | We have been having many problems on a Dell PowerEdge 4400 with 4G of
 > | memory.  We are willing to get new hardware if, perhaps, there is
 > | something known to not work with that hardwares memory.  If there are
 > | known hardware configurations that work will in high load, high IO
 > | situations.  We really want to stay with Linux, but my boss is getting
 > | increasingly agitated with issues.
 > | 
 > | Our application has, on the order of 300 network sockets open at any
 > | one time primarily for input.  And only 1 or 2 for output.  We
 > | constantly malloc/free 100k of memory, I mean a lot.  Plus do a fair
 > | amount of SCSI IO.  The threads never use more then like 300M of
 > | memory at a time though caching memory fills up to like 2G.  I believe
 > | it would work on a system with 1G of memory.
 > | 
 > | With kernels 2.4.14 and before, including some AA kernels, the cache
 > | would fill up memory and do a little swapping to disk and then start
 > | the just-in-time free memory stuff.  We where very happy when 2.4.16
 > | seems smarter about this then previous version and never uses more
 > | then ~2G of cachememory.
 > | 
 > | But, after running for a time, kswapd starts taking more CPU time then
 > | the threads we are running and slowing down the processing.  Is this
 > | something wrong with kswapd or might modifying files in /proc/sys/vm/
 > | fix this?
 > | 
 > | If modifying files in /proc/sys/vm/ can fix it, what should one
 > | consider when modifying the /proc/sys/vm/kswapd and other files to
 > | suit my applications?  Also, in my /proc/sys/vm directory there is
 > | only the following files:
 > | 
 > | $ ls /proc/sys/vm
 > | bdflush  max-readahead  overcommit_memory  pagetable_cache
 > | kswapd   min-readahead  page-cluster
 > | 
 > | With 2.4.16 I though I should have others as described in the proc.txt
 > | file, like `buffermem', `pagecache' and others.  How do I get these,
 > | and once I do have them some hints on how to modify there values to
 > | optimize my applications might be useful.
 > | 
 > | 	 Sven
 > | -
 > | To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
 > | the body of a message to majordomo@vger.kernel.org
 > | More majordomo info at  http://vger.kernel.org/majordomo-info.html
 > | Please read the FAQ at  http://www.tux.org/lkml/
 > -
 > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
 > the body of a message to majordomo@vger.kernel.org
 > More majordomo info at  http://vger.kernel.org/majordomo-info.html
 > Please read the FAQ at  http://www.tux.org/lkml/

Re: hints at modifying kswapd params in 2.4.16

[Sven Heinicke]

hmm, i did some snooping aroudn and I concluded that I worded my
questions wrong (not including bad grammar).  My question should be:

What is the latest kernel with the old VM?  The 2.4.13-ac8?


Sven Heinicke writes:
 > 
 > Does the AC kernel still have the old VM?  I really wanna stick the
 > the new stuff but but a need a stable system.  Older kernels 2.4.8 had
 > highmem issues, not the 2.4.16 has kswap issues.
 > 
 > Ken Brownfield writes:
 >  > Yes, 2.4.16 fixed a major source of kswapd thrashing for (at least)
 >  > mmapped memory, but there are still some corner cases left which I've
 >  > also seen.  For instance, in the new case I'm seeing kswapd thrashing
 >  > even when freepages hasn't been reached under heavy I/O.  I'm under the
 >  > impression that this behavior is more of a mis-behavior rather than a
 >  > mis-tuning.
 >  > 
 >  > I don't think I've seen a response to my recent post go by; hopefully
 >  > the VM issues can still be debugged soon even after the handoff to
 >  > Marcelo... :-)
 >  > -- 
 >  > Ken.
 >  > brownfld@irridia.com
 >  > 
 >  > PS: The hardware in question on my end is an HP LH6000r, 6-way Xeon, 4GB.
 >  > 
 >  > On Mon, Dec 03, 2001 at 01:00:41PM -0500, Sven Heinicke wrote:
 >  > | 
 >  > | We have been having many problems on a Dell PowerEdge 4400 with 4G of
 >  > | memory.  We are willing to get new hardware if, perhaps, there is
 >  > | something known to not work with that hardwares memory.  If there are
 >  > | known hardware configurations that work will in high load, high IO
 >  > | situations.  We really want to stay with Linux, but my boss is getting
 >  > | increasingly agitated with issues.
 >  > | 
 >  > | Our application has, on the order of 300 network sockets open at any
 >  > | one time primarily for input.  And only 1 or 2 for output.  We
 >  > | constantly malloc/free 100k of memory, I mean a lot.  Plus do a fair
 >  > | amount of SCSI IO.  The threads never use more then like 300M of
 >  > | memory at a time though caching memory fills up to like 2G.  I believe
 >  > | it would work on a system with 1G of memory.
 >  > | 
 >  > | With kernels 2.4.14 and before, including some AA kernels, the cache
 >  > | would fill up memory and do a little swapping to disk and then start
 >  > | the just-in-time free memory stuff.  We where very happy when 2.4.16
 >  > | seems smarter about this then previous version and never uses more
 >  > | then ~2G of cachememory.
 >  > | 
 >  > | But, after running for a time, kswapd starts taking more CPU time then
 >  > | the threads we are running and slowing down the processing.  Is this
 >  > | something wrong with kswapd or might modifying files in /proc/sys/vm/
 >  > | fix this?
 >  > | 
 >  > | If modifying files in /proc/sys/vm/ can fix it, what should one
 >  > | consider when modifying the /proc/sys/vm/kswapd and other files to
 >  > | suit my applications?  Also, in my /proc/sys/vm directory there is
 >  > | only the following files:
 >  > | 
 >  > | $ ls /proc/sys/vm
 >  > | bdflush  max-readahead  overcommit_memory  pagetable_cache
 >  > | kswapd   min-readahead  page-cluster
 >  > | 
 >  > | With 2.4.16 I though I should have others as described in the proc.txt
 >  > | file, like `buffermem', `pagecache' and others.  How do I get these,
 >  > | and once I do have them some hints on how to modify there values to
 >  > | optimize my applications might be useful.
 >  > | 
 >  > | 	 Sven
 >  > | -
 >  > | To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
 >  > | the body of a message to majordomo@vger.kernel.org
 >  > | More majordomo info at  http://vger.kernel.org/majordomo-info.html
 >  > | Please read the FAQ at  http://www.tux.org/lkml/
 >  > -
 >  > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
 >  > the body of a message to majordomo@vger.kernel.org
 >  > More majordomo info at  http://vger.kernel.org/majordomo-info.html
 >  > Please read the FAQ at  http://www.tux.org/lkml/

Re: hints at modifying kswapd params in 2.4.16

[Alan Cox]

> Does the AC kernel still have the old VM?  I really wanna stick the
> the new stuff but but a need a stable system.  Older kernels 2.4.8 had
> highmem issues, not the 2.4.16 has kswap issues.

There's a riel vm patch for 2.4.16 if you want to see if the vm thing is
the problem case

Alan

Re: hints at modifying kswapd params in 2.4.16

[Sven Heinicke]

We have actually moved our code to another Dell 4400 we got at the
same time (in theory the same exact same system) and I rebuilt the
kernel with the same .config file and I am getting different memory
usage patterns.

The first system I tried was Red Hat 7.1, it never used more then 2G
of cache memory leaving the other 2G free.

The other system, Mandrake 8.0, sucks up all the 4G of memory with
cache but has not yet shown any signs of thrashing.  Though the code
has only been running a few hours.

Could this be a C compiler issue?

Alan Cox writes:
 > > Does the AC kernel still have the old VM?  I really wanna stick the
 > > the new stuff but but a need a stable system.  Older kernels 2.4.8 had
 > > highmem issues, not the 2.4.16 has kswap issues.
 > 
 > There's a riel vm patch for 2.4.16 if you want to see if the vm thing is
 > the problem case
 > 
 > Alan

Re: hints at modifying kswapd params in 2.4.16

[Alan Cox]

> The first system I tried was Red Hat 7.1, it never used more then 2G
> of cache memory leaving the other 2G free.
> 
> The other system, Mandrake 8.0, sucks up all the 4G of memory with
> cache but has not yet shown any signs of thrashing.  Though the code
> has only been running a few hours.

The RH 7.1 tree is 2.4.2-ac based and certainly wont behave well under some
loads.

Re: hints at modifying kswapd params in 2.4.16

[Sven Heinicke]

I'm not sure if I was understood here.  On both Mandrake 8.0 and Red
Hat 7.1 I'm running the 2.4.16 kernel with the same .confg file, and
built on there respective kernel.  And I'm getting the below different
memory usage patterns with the same processes running.  Seems
something external to the kernel is causing the differences.

As of this morning, December 5th, the Mandrake systems has run longer
then the same kernel ever did on Red Hat.  I say again, with the same
kernel source and same .confg file.

    Sven


Alan Cox writes:
 > > The first system I tried was Red Hat 7.1, it never used more then 2G
 > > of cache memory leaving the other 2G free.
 > > 
 > > The other system, Mandrake 8.0, sucks up all the 4G of memory with
 > > cache but has not yet shown any signs of thrashing.  Though the code
 > > has only been running a few hours.
 > 
 > The RH 7.1 tree is 2.4.2-ac based and certainly wont behave well under some
 > loads.

Re: hints at modifying kswapd params in 2.4.16

[Mike Fedyk]

On Wed, Dec 05, 2001 at 09:57:19AM -0500, Sven Heinicke wrote:
> 
> I'm not sure if I was understood here.  On both Mandrake 8.0 and Red
> Hat 7.1 I'm running the 2.4.16 kernel with the same .confg file, and
> built on there respective kernel.  And I'm getting the below different
> memory usage patterns with the same processes running.  Seems
> something external to the kernel is causing the differences.
> 
> As of this morning, December 5th, the Mandrake systems has run longer
> then the same kernel ever did on Red Hat.  I say again, with the same
> kernel source and same .confg file.
> 

What about compiler version, glibc version, and hardware revision (think
lspci) differences?