Memory management in Kernel 2.4.x

Memory management in Kernel 2.4.x

[Andreas Hartmann]

Hello all!


Unfortunately, the memory management of kernel 2.4.x didn't get better until 
today. It is very easy to make a machine dead. Take the following script:

http://groups.google.com/groups?q=malloc+bestie&hl=de&lr=&selm=slrn8aiglm.tqd.pfk@c.zeiss.de&rnum=2


The result with kernel 2.4.19pre8ac4:

May 27 11:04:21 athlon kernel: Out of Memory: Killed process 763 (knode).
May 27 11:04:22 athlon kernel: Out of Memory: Killed process 764 (knode).
May 27 11:04:22 athlon kernel: Out of Memory: Killed process 765 (knode).
May 27 11:04:22 athlon kernel: Out of Memory: Killed process 766 (knode).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 322 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 324 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 325 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 326 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 327 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 755 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 899 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 902 
(mozilla-bin).
May 27 11:04:40 athlon kernel: Out of Memory: Killed process 903 
(mozilla-bin).
May 27 11:04:54 athlon kernel: Out of Memory: Killed process 295 (kdeinit).
May 27 11:05:00 athlon kernel: Out of Memory: Killed process 293 (kdeinit).
May 27 11:05:05 athlon kernel: Out of Memory: Killed process 283 (kdeinit).
May 27 11:05:13 athlon kernel: Out of Memory: Killed process 932 (kdeinit).
May 27 11:05:16 athlon kernel: Out of Memory: Killed process 287 (kdeinit).
May 27 11:05:31 athlon kernel: Out of Memory: Killed process 300 (kdeinit).
May 27 11:05:31 athlon kernel: Out of Memory: Killed process 303 (korgac).
May 27 11:05:37 athlon kernel: Out of Memory: Killed process 297 (kwikdisk).
May 27 11:06:05 athlon kernel: Out of Memory: Killed process 292 (kdeinit).
May 27 11:06:46 athlon kernel: Out of Memory: Killed process 289 (kdeinit).
May 27 11:06:52 athlon kernel: Out of Memory: Killed process 269 (kdeinit).
May 27 11:06:57 athlon kernel: Out of Memory: Killed process 304 (kalarmd).
May 27 11:07:18 athlon kernel: Out of Memory: Killed process 286 (kdeinit).
May 27 11:07:21 athlon kernel: Out of Memory: Killed process 1103 (malloc).
May 27 11:07:21 athlon kernel: Out of Memory: Killed process 1103 (malloc).
May 27 11:07:34 athlon kernel: Out of Memory: Killed process 1104 (malloc).
May 27 11:07:52 athlon kernel: Out of Memory: Killed process 1104 (malloc).
May 27 11:08:04 athlon kernel: Out of Memory: Killed process 1105 (malloc).
May 27 11:09:12 athlon last message repeated 5 times
May 27 11:09:16 athlon kernel: Out of Memory: Killed process 1106 (malloc).
May 27 11:09:24 athlon kernel: Out of Memory: Killed process 1107 (malloc).
May 27 11:09:51 athlon kernel: Out of Memory: Killed process 1108 (malloc).
May 27 11:10:01 athlon kernel: Out of Memory: Killed process 1111 (malloc).
May 27 11:10:07 athlon kernel: Out of Memory: Killed process 1112 (malloc).
May 27 11:10:12 athlon kernel: Out of Memory: Killed process 1115 (malloc).
May 27 11:10:58 athlon kernel: Out of Memory: Killed process 1118 (malloc).
May 27 11:10:59 athlon kernel: Out of Memory: Killed process 1118 (malloc).
May 27 11:11:05 athlon kernel: Out of Memory: Killed process 1124 (malloc).
May 27 11:11:10 athlon kernel: Out of Memory: Killed process 1121 (malloc).
May 27 11:11:18 athlon kernel: Out of Memory: Killed process 1127 (malloc).
May 27 11:11:24 athlon kernel: Out of Memory: Killed process 1130 (malloc).
May 27 11:11:33 athlon kernel: Out of Memory: Killed process 1133 (malloc).
May 27 11:11:44 athlon kernel: Out of Memory: Killed process 1134 (malloc).
May 27 11:11:45 athlon kernel: Out of Memory: Killed process 1135 (malloc).
May 27 11:11:50 athlon kernel: Out of Memory: Killed process 1136 (malloc).
May 27 11:11:59 athlon kernel: Out of Memory: Killed process 1137 (malloc).
May 27 11:12:28 athlon kernel: Out of Memory: Killed process 1138 (malloc).
May 27 11:12:32 athlon kernel: Out of Memory: Killed process 1138 (malloc).
May 27 11:13:04 athlon kernel: Out of Memory: Killed process 1139 (malloc).
May 27 11:13:04 athlon kernel: Out of Memory: Killed process 1139 (malloc).


KDE was down. The machine didn't respond during this fork bomb.



I have got the same problem with an remote rsync via ssh when rsnc breaks 
down because of an data-error when rsyncing about 30 G of datas at the end 
of the session. With 2.4.x-kernels I need a lot of more memory for this 
process than with kernel 2.2.


Kind regards,
Andreas Hartmann

Re: Memory management in Kernel 2.4.x

[Florian Weimer]

Andreas Hartmann <andihartmann@freenet.de> writes:

> Unfortunately, the memory management of kernel 2.4.x didn't get
> better until today. It is very easy to make a machine dead. Take the
> following script:

Use an -ac kernel with disabled overcommitment.

-- 
Florian Weimer 	                  Weimer@CERT.Uni-Stuttgart.DE
University of Stuttgart           http://CERT.Uni-Stuttgart.DE/people/fw/
RUS-CERT                          +49-711-685-5973/fax +49-711-685-5898

Re: Memory management in Kernel 2.4.x

[Zwane Mwaikambo]

On Mon, 27 May 2002, Andreas Hartmann wrote:

> rsync allocates all of the memory the machine has (256 MB RAM, 128 MB swap). 
> When this occures, processes get killed like described in the posting 
> before. The machine doesn't respond as long as the rsync - process isn't 
> killed, because it fetches all the memory which gets free after a process 
> has been killed.

And the rsync process never gets singled out? nice!

	Zwane Mwaikambo

-- 
http://function.linuxpower.ca
		

Re: Memory management in Kernel 2.4.x

[Alan Cox]

On Mon, 2002-05-27 at 10:40, Andreas Hartmann wrote:
> Unfortunately, the memory management of kernel 2.4.x didn't get better until 
> today. It is very easy to make a machine dead. Take the following script:
> 
> http://groups.google.com/groups?q=malloc+bestie&hl=de&lr=&selm=slrn8aiglm.tqd.pfk@c.zeiss.de&rnum=2
> 
> 
> The result with kernel 2.4.19pre8ac4:
> 
> May 27 11:04:21 athlon kernel: Out of Memory: Killed process 763 (knode).

This appears to be correct behaviour. You allocated astronomical amounts
of memory and had memory overcommit enabled so it broke. Thats
configurable and you can if you wish disable overcommit in the -ac
kernel by setting /proc/sys/vm/overcommit_memory to 2 (thats not
supported by the base 2.4 kernels yet). If you can make it OOM in that
mode then I am interested indeed. 

The rsync one is more interesting, it could be something doing huge
amounts of memory allocations it could also be excessive kernel usage or
wrong OOM semantics somewhere.

Re: Memory management in Kernel 2.4.x

[Andreas Hartmann]

Alan Cox wrote:

> On Mon, 2002-05-27 at 10:40, Andreas Hartmann wrote:
>> Unfortunately, the memory management of kernel 2.4.x didn't get better
>> until today. It is very easy to make a machine dead. Take the following
>> script:
>> 
>> 
http://groups.google.com/groups?q=malloc+bestie&hl=de&lr=&selm=slrn8aiglm.tqd.pfk@c.zeiss.de&rnum=2
>> 
>> 
>> The result with kernel 2.4.19pre8ac4:
>> 
>> May 27 11:04:21 athlon kernel: Out of Memory: Killed process 763 (knode).
> 
> This appears to be correct behaviour. You allocated astronomical amounts
> of memory and had memory overcommit enabled so it broke. Thats
> configurable and you can if you wish disable overcommit in the -ac
> kernel by setting /proc/sys/vm/overcommit_memory to 2 (thats not
> supported by the base 2.4 kernels yet). If you can make it OOM in that
> mode then I am interested indeed.

I tested it and got in the result the same problem. Well, no processes got 
killed by the kernel and the virtual memory didn't grow up to the end (as 
far as I could see it).
But every (other, regular) process crashed when it wanted to have some 
memory because it didn't get the memory. Or I wasn't able to start any new 
process - I even couldn't do a logon via ssh or a local login at the 
console.


May 27 13:37:12 athlon login: PAM unable to 
dlopen(/lib/security/pam_pwdb.so)
May 27 13:37:12 athlon login: PAM [dlerror: libpwdb.so.0: failed to map 
segment from shared object: Cannot allocate memory]
May 27 13:37:12 athlon login: PAM adding faulty module: 
/lib/security/pam_pwdb.so
May 27 13:37:13 athlon out of memory [198out of me]
May 27 13:37:15 athlon login: PAM unable to 
dlopen(/lib/security/pam_pwdb.so)
May 27 13:37:15 athlon login: PAM [dlerror: libpwdb.so.0: failed to map 
segment from shared object: Cannot allocate memory]
May 27 13:37:15 athlon login: PAM adding faulty module: 
/lib/security/pam_pwdb.so
May 27 13:37:15 athlon out of memory [2617out of m]
May 27 13:37:15 athlon last message repeated 29 times
May 27 13:37:23 athlon login: PAM unable to 
dlopen(/lib/security/pam_pwdb.so)
May 27 13:37:23 athlon login: PAM [dlerror: libpwdb.so.0: failed to map 
segment from shared object: Cannot allocate memory]
May 27 13:37:23 athlon login: PAM adding faulty module: 
/lib/security/pam_pwdb.so
May 27 13:37:23 athlon out of memory [2620out of m]
May 27 13:37:23 athlon last message repeated 29 times
May 27 13:37:27 athlon login: PAM unable to 
dlopen(/lib/security/pam_pwdb.so)
May 27 13:37:27 athlon login: PAM [dlerror: libpwdb.so.0: failed to map 
segment from shared object: Cannot allocate memory]
May 27 13:37:27 athlon login: PAM adding faulty module: 
/lib/security/pam_pwdb.so
May 27 13:37:27 athlon out of memory [199out of me]
May 27 13:39:36 athlon login: PAM unable to 
dlopen(/lib/security/pam_pwdb.so)
May 27 13:39:36 athlon login: PAM [dlerror: libpwdb.so.0: failed to map 
segment from shared object: Cannot allocate memory]
May 27 13:39:36 athlon login: PAM adding faulty module: 
/lib/security/pam_pwdb.so
May 27 13:39:36 athlon out of memory [2629out of m]
May 27 13:39:36 athlon last message repeated 29 times
May 27 13:42:01 athlon sshd[124]: error: fork: Cannot allocate memory

All in all, the overcommitment - option seems to work fine (the machine 
responded well :-)), but it should restrict just the process which leads 
the machine to death. All other processes should be as free as before.

> The rsync one is more interesting, it could be something doing huge
> amounts of memory allocations it could also be excessive kernel usage or
> wrong OOM semantics somewhere.

rsync allocates all of the memory the machine has (256 MB RAM, 128 MB swap). 
When this occures, processes get killed like described in the posting 
before. The machine doesn't respond as long as the rsync - process isn't 
killed, because it fetches all the memory which gets free after a process 
has been killed.


Regards,
Andreas Hartmann

Re: Memory management in Kernel 2.4.x

[Andreas Hartmann]

Zwane Mwaikambo wrote:

> On Mon, 27 May 2002, Andreas Hartmann wrote:
> 
>> rsync allocates all of the memory the machine has (256 MB RAM, 128 MB
>> swap). When this occures, processes get killed like described in the
>> posting before. The machine doesn't respond as long as the rsync -
>> process isn't killed, because it fetches all the memory which gets free
>> after a process has been killed.
> 
> And the rsync process never gets singled out? nice!

Until it's killed by the kernel (if overcommitment isn't deactivated). If 
overcommitment is deactivated, the services of the machine are dead 
forever. There will be nothing, which kills such a process. Or am I wrong?


Regards,
Andreas Hartmann

Re: Memory management in Kernel 2.4.x

[Peter Wächtler]

Andreas Hartmann wrote:
> Zwane Mwaikambo wrote:
> 
> 
>>On Mon, 27 May 2002, Andreas Hartmann wrote:
>>
>>
>>>rsync allocates all of the memory the machine has (256 MB RAM, 128 MB
>>>swap). When this occures, processes get killed like described in the
>>>posting before. The machine doesn't respond as long as the rsync -
>>>process isn't killed, because it fetches all the memory which gets free
>>>after a process has been killed.
>>>
>>And the rsync process never gets singled out? nice!
>>
> 
> Until it's killed by the kernel (if overcommitment isn't deactivated). If 
> overcommitment is deactivated, the services of the machine are dead 
> forever. There will be nothing, which kills such a process. Or am I wrong?
> 

There is still the oom killer (Out Of Memory).
But it doesn't trigger and the machine pages "forever".
Usually kswapd eats the CPU then, discarding and reloading pages,
searching lists for pages to evict and so on.

Re: Memory management in Kernel 2.4.x

[Peter Wächtler]

Alan Cox wrote:
> On Mon, 2002-05-27 at 14:45, Peter Wächtler wrote:
> 
>>There is still the oom killer (Out Of Memory).
>>But it doesn't trigger and the machine pages "forever".
>>Usually kswapd eats the CPU then, discarding and reloading pages,
>>searching lists for pages to evict and so on.
>>
> 
> On a -ac kernel with mode 2 or 3 set for overcommit you have to run out
> of kernel resources to hang the box. It won't go OOM because it can't.
> That wouldn't be a VM bug but a leak or poor handling of kernel
> allocations somewhere. Sadly the changes needed to do that (beancounter
> patch) were things Linus never accepted for 2.4
> 

I heard of the "beancounter patch" several times now.
Where is it - that beancounter patch? What does it besides bean counting ;-)
ah, googled it:

ftp://ftp.sw.com.sg/pub/Linux/people/saw/kernel/user_beancounter/UserBeancounter.html

I think we will get another candidate for beancounting: the futexes are locking
user pages.. and I already thought about accounting with
setrlimit/ulimit and "max locked memory (kbytes)"

Re: Memory management in Kernel 2.4.x

[Alan Cox]

On Mon, 2002-05-27 at 14:45, Peter Wächtler wrote:
> There is still the oom killer (Out Of Memory).
> But it doesn't trigger and the machine pages "forever".
> Usually kswapd eats the CPU then, discarding and reloading pages,
> searching lists for pages to evict and so on.

On a -ac kernel with mode 2 or 3 set for overcommit you have to run out
of kernel resources to hang the box. It won't go OOM because it can't.
That wouldn't be a VM bug but a leak or poor handling of kernel
allocations somewhere. Sadly the changes needed to do that (beancounter
patch) were things Linus never accepted for 2.4

Re: Memory management in Kernel 2.4.x

[H. Peter Anvin]

Followup to:  <1022513156.1126.289.camel@irongate.swansea.linux.org.uk>
By author:    Alan Cox <alan@lxorguk.ukuu.org.uk>
In newsgroup: linux.dev.kernel
> 
> On a -ac kernel with mode 2 or 3 set for overcommit you have to run out
> of kernel resources to hang the box. It won't go OOM because it can't.
> That wouldn't be a VM bug but a leak or poor handling of kernel
> allocations somewhere. Sadly the changes needed to do that (beancounter
> patch) were things Linus never accepted for 2.4
> 

Well, if you can't fork a new process because that would push you into
overcommit, then you usually can't actually do anything useful on the
machine.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Memory management in Kernel 2.4.x

[Benjamin LaHaise]

On Mon, May 27, 2002 at 02:22:22PM -0700, H. Peter Anvin wrote:
> Well, if you can't fork a new process because that would push you into
> overcommit, then you usually can't actually do anything useful on the
> machine.

Just use vfork or clone + exec.  It's faster and uses less memory.

		-ben
-- 
"You will be reincarnated as a toad; and you will be much happier."

Re: Memory management in Kernel 2.4.x

[H. Peter Anvin]

Benjamin LaHaise wrote:
> On Mon, May 27, 2002 at 02:22:22PM -0700, H. Peter Anvin wrote:
> 
>>Well, if you can't fork a new process because that would push you into
>>overcommit, then you usually can't actually do anything useful on the
>>machine.
> 
> 
> Just use vfork or clone + exec.  It's faster and uses less memory.
> 

Doesn't help.  exec() should fail.

	-hpa

Re: Memory management in Kernel 2.4.x

[Benjamin LaHaise]

On Mon, May 27, 2002 at 02:34:00PM -0700, H. Peter Anvin wrote:
> Doesn't help.  exec() should fail.

Only if you're out of memory (any sane app nowadays does not allocate 
memory from data/bss).

		-ben
-- 
"You will be reincarnated as a toad; and you will be much happier."

Re: Memory management in Kernel 2.4.x

[H. Peter Anvin]

Benjamin LaHaise wrote:
> On Mon, May 27, 2002 at 02:34:00PM -0700, H. Peter Anvin wrote:
> 
>>Doesn't help.  exec() should fail.
> 
> 
> Only if you're out of memory (any sane app nowadays does not allocate 
> memory from data/bss).
> 

.data/.bss is writable, and will need to be allocated to avoid 
overcommitment, so the exec() will fail.

	-hpa

Re: Memory management in Kernel 2.4.x

[William Lee Irwin III]

On Mon, May 27, 2002 at 02:22:22PM -0700, H. Peter Anvin wrote:
>>> Well, if you can't fork a new process because that would push you into
>>> overcommit, then you usually can't actually do anything useful on the
>>> machine.

On Mon, 2002-05-27 at 22:33, Benjamin LaHaise wrote:
>> Just use vfork or clone + exec.  It's faster and uses less memory.

On Mon, May 27, 2002 at 11:50:31PM +0100, Alan Cox wrote:
> In the general case a fork doesn't cause too much overcommit. Most of
> the binary is mapped read-only as is a lot of the library space. Since
> its read only and backed by a file it has zero cost. If you mprotect it
> then you pay at mprotect time

If you're willing to take a feature request, I'd be much obliged if the
pagetable memory were also accounted.


Thanks,
Bill

Re: Memory management in Kernel 2.4.x

[Alan Cox]

On Mon, 2002-05-27 at 22:22, H. Peter Anvin wrote:
> Followup to:  <1022513156.1126.289.camel@irongate.swansea.linux.org.uk>
> By author:    Alan Cox <alan@lxorguk.ukuu.org.uk>
> In newsgroup: linux.dev.kernel
> > 
> > On a -ac kernel with mode 2 or 3 set for overcommit you have to run out
> > of kernel resources to hang the box. It won't go OOM because it can't.
> > That wouldn't be a VM bug but a leak or poor handling of kernel
> > allocations somewhere. Sadly the changes needed to do that (beancounter
> > patch) were things Linus never accepted for 2.4
> > 
> 
> Well, if you can't fork a new process because that would push you into
> overcommit, then you usually can't actually do anything useful on the
> machine.

Thats actually easy to deal with and on my list for modes 4 and 5 (2 and
3 with root granted a reserved fraction)

Re: Memory management in Kernel 2.4.x

[Alan Cox]

On Mon, 2002-05-27 at 22:33, Benjamin LaHaise wrote:
> On Mon, May 27, 2002 at 02:22:22PM -0700, H. Peter Anvin wrote:
> > Well, if you can't fork a new process because that would push you into
> > overcommit, then you usually can't actually do anything useful on the
> > machine.
> 
> Just use vfork or clone + exec.  It's faster and uses less memory.

In the general case a fork doesn't cause too much overcommit. Most of
the binary is mapped read-only as is a lot of the library space. Since
its read only and backed by a file it has zero cost. If you mprotect it
then you pay at mprotect time

Re: Memory management in Kernel 2.4.x

[Alan Cox]

On Mon, 2002-05-27 at 22:56, William Lee Irwin III wrote:
> If you're willing to take a feature request, I'd be much obliged if the
> pagetable memory were also accounted.

I suspect that is doable providing you are willing to handwave a few
assumptions and be a fraction conservative. However I'm accounting
against ram+swap. You need to account against ram only.

Re: Memory management in Kernel 2.4.x

[Andreas Hartmann]

H. Peter Anvin wrote:

> Followup to:  <1022513156.1126.289.camel@irongate.swansea.linux.org.uk>
> By author:    Alan Cox <alan@lxorguk.ukuu.org.uk>
> In newsgroup: linux.dev.kernel
>> 
>> On a -ac kernel with mode 2 or 3 set for overcommit you have to run out
>> of kernel resources to hang the box. It won't go OOM because it can't.
>> That wouldn't be a VM bug but a leak or poor handling of kernel
>> allocations somewhere. Sadly the changes needed to do that (beancounter
>> patch) were things Linus never accepted for 2.4
>> 
> 
> Well, if you can't fork a new process because that would push you into
> overcommit, then you usually can't actually do anything useful on the
> machine.

>From my experience with mode 2:

you can't do _anything_, if the overcommitment range has been reached. Even 
running programms are crashing if they want to have some more memory. So, 
new processes cannot be started and old processes cannot run and are 
crashing as far as they want to have more memory. At last, there will be 
only one user-process on the machine running - the bad programm, eating up 
all the memory.


Regards,
Andreas Hartmann

Re: Memory management in Kernel 2.4.x

[Alan Cox]

On Tue, 2002-05-28 at 06:42, Andreas Hartmann wrote:
> > Well, if you can't fork a new process because that would push you into
> > overcommit, then you usually can't actually do anything useful on the
> > machine.
> 
> >From my experience with mode 2:
> 
> you can't do _anything_, if the overcommitment range has been reached. Even 
> running programms are crashing if they want to have some more memory. So, 
> new processes cannot be started and old processes cannot run and are 
> crashing as far as they want to have more memory. At last, there will be 
> only one user-process on the machine running - the bad programm, eating up 
> all the memory.

Are you sure you have it properly configured. Programs will only crash
if they demand more memory and don't properly handle out of memory
errors. No OOM kills occur. I've run simulated sets with large databases
and I don't see the problem you are reporting. There is a not quite
theoretical case where everything continues running but nothing quits
because it all has the memory it wants and there is not enough more. 

Also if an old program crashes, it frees memory so you have room for a
new one. With your fork bomb there is a likelyhood that a new fork will
beat others to the memory. 

Ultimately something has to die off or give back resources when it finds
it can't get any. There is a finite resource, you used it all. Going
beyond the current stuff to doing definable chargable subgroups with per
subgroup resource billing and optional overcommit rules is doable (thats
beancounter stuff again) but does require we add setluid/getluid
syscalls, tweak the PAM code in user space and merge the beancounter
stuff. At that point you can do really *cool* stuff like

	Staff have a guaranteed 75% of memory
	Students have a guaranteed 25% of memory but can take 50 if its 		there

And sit contented in the sure knowledge that if you fire up emacs on a
giant file as a staff member you'll only kick students off the box


Alan

Re: Memory management in Kernel 2.4.x

[Dmitry Volkoff]

Hello,

> Unfortunately, the memory management of kernel 2.4.x didn't get better until 
> today. It is very easy to make a machine dead. Take the following script: 
> 
> http://groups.google.com/groups?q=malloc+bestie&hl=de&lr=&selm=slrn8aiglm.tqd.pfk@c.zeiss.de&rnum=2 
> 
> The result with kernel 2.4.19pre8ac4: 

What about results with latest -aa kernels? 
I'm writing this letter while running your killer-test ;)
My machine is perfectly responsive.

$ uname -a
Linux localhost 2.4.19-pre6vm33 #2 Sat Apr 13 00:56:55 MSD 2002 i686 unknown
This is vanilla 2.4.19-pre6 + vm33 from Andrea Arcangelly.

Funny stats:

bash-2.05$ free
total       used       free     shared    buffers     cached
Mem:        516496     512804       3692          0       2080      10064
-/+ buffers/cache:     500660      15836
Swap:      1028120    1028120          0

I'm running X, netscape, mutt, top, 5 xterms, web, mysql, ftp, samba etc.
Lots of "0-order allocation failed" in the log. Anyway it is usable!
The good thing is VM is killing right process most of the time: 

VM: killing process memory-killer

Netscape got killed. Big deal, I can start it again and it works... 

P.S. Running it already for 20 minutes.

-- 

    DV

Re: Memory management in Kernel 2.4.x

[Andreas Hartmann]

Alan Cox wrote:
> On Tue, 2002-05-28 at 06:42, Andreas Hartmann wrote:
> 
>>>Well, if you can't fork a new process because that would push you into
>>>overcommit, then you usually can't actually do anything useful on the
>>>machine.
>>
>>>From my experience with mode 2:
>>
>>you can't do _anything_, if the overcommitment range has been reached. Even 
>>running programms are crashing if they want to have some more memory. So, 
>>new processes cannot be started and old processes cannot run and are 
>>crashing as far as they want to have more memory. At last, there will be 
>>only one user-process on the machine running - the bad programm, eating up 
>>all the memory.
> 
> 
> Are you sure you have it properly configured. Programs will only crash
> if they demand more memory and don't properly handle out of memory
> errors.

I don't know, if they properly handle out of memory errors. It have been 
X-programms.
Logon wasn't able either with login nor with ssh from remote (sshd 
couldn't fork).

> No OOM kills occur.

That's definitly right.

> I've run simulated sets with large databases
> and I don't see the problem you are reporting. There is a not quite
> theoretical case where everything continues running but nothing quits
> because it all has the memory it wants and there is not enough more. 

A process which doesn't need any more memory is running well. In 
consoles, on which I have been logged in, I could roll through the last 
commands without any problem - but I couldn't start a new process.

> Also if an old program crashes, it frees memory so you have room for a
> new one.

The memory wich has been freed is eaten up from the malicious program. 
It's very easy to reproduce it with the program I linked to in my first 
posting.

> With your fork bomb there is a likelyhood that a new fork will
> beat others to the memory. 

That's it. And if there is a program running with such a malfunction the 
machine can't do anything more. If a hacker wants to block your machine, 
he can do it without beeing root.

> Ultimately something has to die off or give back resources when it finds
> it can't get any. There is a finite resource, you used it all. Going
> beyond the current stuff to doing definable chargable subgroups with per
> subgroup resource billing and optional overcommit rules is doable (thats
> beancounter stuff again) but does require we add setluid/getluid
> syscalls, tweak the PAM code in user space and merge the beancounter
> stuff. At that point you can do really *cool* stuff like
> 
> 	Staff have a guaranteed 75% of memory
> 	Students have a guaranteed 25% of memory but can take 50 if its	there
> 
> And sit contented in the sure knowledge that if you fire up emacs on a
> giant file as a staff member you'll only kick students off the box

I think, that's what would be best. Ressources are given to groups and 
they must work with. There isn't any more for them - the rest is for 
sysadmins to ensure that they always can successfully control the 
machine and can kill malicious processes. No external reachable daemon 
must be running in the sysadmin ressource-group.

I think it's something like ressource-management on IBM's S390.


Regards,
Andreas Hartmann

Re: Memory management in Kernel 2.4.x

[Andrea Arcangeli]

On Mon, May 27, 2002 at 03:45:55PM +0200, Peter Wächtler wrote:
> Andreas Hartmann wrote:
> >Zwane Mwaikambo wrote:
> >
> >
> >>On Mon, 27 May 2002, Andreas Hartmann wrote:
> >>
> >>
> >>>rsync allocates all of the memory the machine has (256 MB RAM, 128 MB
> >>>swap). When this occures, processes get killed like described in the
> >>>posting before. The machine doesn't respond as long as the rsync -
> >>>process isn't killed, because it fetches all the memory which gets free
> >>>after a process has been killed.
> >>>
> >>And the rsync process never gets singled out? nice!
> >>
> >
> >Until it's killed by the kernel (if overcommitment isn't deactivated). If 
> >overcommitment is deactivated, the services of the machine are dead 
> >forever. There will be nothing, which kills such a process. Or am I wrong?
> >
> 
> There is still the oom killer (Out Of Memory).
> But it doesn't trigger and the machine pages "forever".
> Usually kswapd eats the CPU then, discarding and reloading pages,
> searching lists for pages to evict and so on.

can you reproduce with 2.4.19pre9aa2? I expect at least the deadlock
(if it's a deadlock and not a livelock) should go away.

Also I read in another email that somebody grown the per-socket buffer
to hundred mbytes, if you do that on a 32bit arch with highmem you'll
run into troubles, too much ZONE_NORMAL ram will be constantly pinned
for the tcp pipeline and the machine can enter livelocks.

Andrea

Re: Memory management in Kernel 2.4.x

[Andreas Hartmann]

Hello Andrea,

Andrea Arcangeli wrote:
> On Mon, May 27, 2002 at 03:45:55PM +0200, Peter Wächtler wrote:
> 
>>Andreas Hartmann wrote:
>>
>>>Zwane Mwaikambo wrote:
>>>
>>>
>>>
>>>>On Mon, 27 May 2002, Andreas Hartmann wrote:
>>>>
>>>>
>>>>
>>>>>rsync allocates all of the memory the machine has (256 MB RAM, 128 MB
>>>>>swap). When this occures, processes get killed like described in the
>>>>>posting before. The machine doesn't respond as long as the rsync -
>>>>>process isn't killed, because it fetches all the memory which gets free
>>>>>after a process has been killed.
>>>>>
>>>>
>>>>And the rsync process never gets singled out? nice!
>>>>
>>>
>>>Until it's killed by the kernel (if overcommitment isn't deactivated). If 
>>>overcommitment is deactivated, the services of the machine are dead 
>>>forever. There will be nothing, which kills such a process. Or am I wrong?
>>>
>>
>>There is still the oom killer (Out Of Memory).
>>But it doesn't trigger and the machine pages "forever".
>>Usually kswapd eats the CPU then, discarding and reloading pages,
>>searching lists for pages to evict and so on.
> 
> 
> can you reproduce with 2.4.19pre9aa2? I expect at least the deadlock
> (if it's a deadlock and not a livelock) should go away.
> 
> Also I read in another email that somebody grown the per-socket buffer
> to hundred mbytes, if you do that on a 32bit arch with highmem you'll
> run into troubles, too much ZONE_NORMAL ram will be constantly pinned
> for the tcp pipeline and the machine can enter livelocks.

I tested it and the error-situation occured again (thanks to rsync :-)).

How did the kernel react?
Well, I waited some seconds until all the memory was in use (256 MB RAM 
and 128 MB swap). I have to say, nearly all the memory, because there 
was allways some MB's free in RAM. So I was able to login via ssh as 
root. As I wanted to do something, the machine didn't react. Until this 
point, the machine seemed to work fine - xosview through ssh showed the 
activity very well. But then it crashed, because it couldn't open 
/proc/stat:
	Can not open file : /proc/stat
Some seconds later, the machine was working normal again. What happened? 
  /var/log/messages says:

Jun  1 14:03:46 susi squid[271]: Squid Parent: child process 273 exited 
due to signal 9
Jun  1 14:03:48 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1d2/0)
Jun  1 14:03:48 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1d2/0)
Jun  1 14:03:48 susi kernel: VM: killing process squid
Jun  1 14:04:05 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1f0/0)
Jun  1 14:04:07 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0xf0/0)
Jun  1 14:04:11 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1f0/0)
Jun  1 14:04:25 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1d2/0)
Jun  1 14:04:25 susi kernel: VM: killing process rsync


The kernel killed the maliciuos rsync-process.
Not nice was, that the kernel killed squid, too. If it didn't kill squid 
it would have been a very good result. On the other hand, squid had 
already problems - so it was probably a good idea to kill it completely 
to make room for other running programs.

I would like to know, if the killing of rsync was pure chance or if it 
was systematically. I will try it again :-).


Regards,
Andreas Hartmann

Re: Memory management in Kernel 2.4.x

[Andreas Hartmann]

Hello Andrea,

 >> Andreas Hartmann wrote:
 > Andrea Arcangeli wrote:

[...]

 >> can you reproduce with 2.4.19pre9aa2? I expect at least the deadlock
 >> (if it's a deadlock and not a livelock) should go away.
 >>
 >> Also I read in another email that somebody grown the per-socket
 >> buffer
 >> to hundred mbytes, if you do that on a 32bit arch with highmem you'll
 >> run into troubles, too much ZONE_NORMAL ram will be constantly pinned
 >> for the tcp pipeline and the machine can enter livelocks.

 > I tested it and the error-situation occured again (thanks to rsync
 > :-)).

 > How did the kernel react?
 > Well, I waited some seconds until all the memory was in use (256 MB
 > RAM
 > and 128 MB swap). I have to say, nearly all the memory, because there
 > was allways some MB's free in RAM. So I was able to login via ssh as
 > root. As I wanted to do something, the machine didn't react. Until
 > this
 > point, the machine seemed to work fine - xosview through ssh showed
 > the
 > activity very well. But then it crashed, because it couldn't open
 > /proc/stat:
 >	Can not open file : /proc/stat
 > Some seconds later, the machine was working normal again. What
 > happened?
 >  /var/log/messages says:

[...]

 > I would like to know, if the killing of rsync was pure chance or if it
 > was systematically. I will try it again :-).

I did now two more tests. The kernel reacted nearly as described above. 
The malicious process was killed but one time unfortunately xosview, 
too. I always tried to open a new ssh session, wich always worked. Even 
  innd was able to server some news-postings :-).

If the kernel always only would kill the malicious process, it would be 
really great and probably the solution of a really big problem of 
2.4.-kernels!

1. try
Jun  1 14:40:58 susi PAM_unix[2650]: (sshd) session closed for user root
Jun  1 14:42:08 susi sshd[2660]: Accepted publickey for root from 
192.168.1.3 port 32987 ssh2
Jun  1 14:42:20 susi PAM_unix[2660]: (sshd) session opened for user root 
by (uid=0)
Jun  1 14:44:36 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1d2/0)
Jun  1 14:44:36 susi kernel: VM: killing process rsync


2. try
Jun  1 14:49:00 susi PAM_unix[2804]: (sshd) session closed for user root
Jun  1 14:50:21 susi sshd[2813]: Accepted publickey for andreas from 
192.168.1.3 port 32991 ssh2
Jun  1 14:50:27 susi PAM_unix[2813]: (sshd) session opened for user 
andreas by (uid=0)
Jun  1 14:51:37 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1f0/0)
Jun  1 14:51:46 susi last message repeated 2 times
Jun  1 14:51:50 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1d2/0)
Jun  1 14:51:50 susi kernel: VM: killing process xosview
Jun  1 14:52:03 susi kernel: __alloc_pages: 0-order allocation failed 
(gfp=0x1d2/0)
Jun  1 14:52:03 susi kernel: VM: killing process rsync


The problems of resync always occure if the rsync-process on the other 
side is stoped with CTRL-C e.g. At this moment, the ssh-connection to 
the host susi is closed, which rsync obviously can't handle correctly.
That's why it's very easy for me to reproduce the situation and how the 
kernel is acting.


Regards,
Andreas Hartmann

Re: Memory management in Kernel 2.4.x

[Andrea Arcangeli]

On Sat, Jun 01, 2002 at 02:35:20PM +0200, Andreas Hartmann wrote:
> Hello Andrea,
> 
> Andrea Arcangeli wrote:
> >On Mon, May 27, 2002 at 03:45:55PM +0200, Peter Wächtler wrote:
> >
> >>Andreas Hartmann wrote:
> >>
> >>>Zwane Mwaikambo wrote:
> >>>
> >>>
> >>>
> >>>>On Mon, 27 May 2002, Andreas Hartmann wrote:
> >>>>
> >>>>
> >>>>
> >>>>>rsync allocates all of the memory the machine has (256 MB RAM, 128 MB
> >>>>>swap). When this occures, processes get killed like described in the
> >>>>>posting before. The machine doesn't respond as long as the rsync -
> >>>>>process isn't killed, because it fetches all the memory which gets free
> >>>>>after a process has been killed.
> >>>>>
> >>>>
> >>>>And the rsync process never gets singled out? nice!
> >>>>
> >>>
> >>>Until it's killed by the kernel (if overcommitment isn't deactivated). 
> >>>If overcommitment is deactivated, the services of the machine are dead 
> >>>forever. There will be nothing, which kills such a process. Or am I 
> >>>wrong?
> >>>
> >>
> >>There is still the oom killer (Out Of Memory).
> >>But it doesn't trigger and the machine pages "forever".
> >>Usually kswapd eats the CPU then, discarding and reloading pages,
> >>searching lists for pages to evict and so on.
> >
> >
> >can you reproduce with 2.4.19pre9aa2? I expect at least the deadlock
> >(if it's a deadlock and not a livelock) should go away.
> >
> >Also I read in another email that somebody grown the per-socket buffer
> >to hundred mbytes, if you do that on a 32bit arch with highmem you'll
> >run into troubles, too much ZONE_NORMAL ram will be constantly pinned
> >for the tcp pipeline and the machine can enter livelocks.
> 
> I tested it and the error-situation occured again (thanks to rsync :-)).
> 
> How did the kernel react?
> Well, I waited some seconds until all the memory was in use (256 MB RAM 
> and 128 MB swap). I have to say, nearly all the memory, because there 
> was allways some MB's free in RAM. So I was able to login via ssh as 
> root. As I wanted to do something, the machine didn't react. Until this 
> point, the machine seemed to work fine - xosview through ssh showed the 
> activity very well. But then it crashed, because it couldn't open 
> /proc/stat:
> 	Can not open file : /proc/stat
> Some seconds later, the machine was working normal again. What happened? 
>  /var/log/messages says:
> 
> Jun  1 14:03:46 susi squid[271]: Squid Parent: child process 273 exited 
> due to signal 9
> Jun  1 14:03:48 susi kernel: __alloc_pages: 0-order allocation failed 
> (gfp=0x1d2/0)
> Jun  1 14:03:48 susi kernel: __alloc_pages: 0-order allocation failed 
> (gfp=0x1d2/0)
> Jun  1 14:03:48 susi kernel: VM: killing process squid
> Jun  1 14:04:05 susi kernel: __alloc_pages: 0-order allocation failed 
> (gfp=0x1f0/0)
> Jun  1 14:04:07 susi kernel: __alloc_pages: 0-order allocation failed 
> (gfp=0xf0/0)
> Jun  1 14:04:11 susi kernel: __alloc_pages: 0-order allocation failed 
> (gfp=0x1f0/0)
> Jun  1 14:04:25 susi kernel: __alloc_pages: 0-order allocation failed 
> (gfp=0x1d2/0)
> Jun  1 14:04:25 susi kernel: VM: killing process rsync
> 
> 
> The kernel killed the maliciuos rsync-process.
> Not nice was, that the kernel killed squid, too. If it didn't kill squid 
> it would have been a very good result. On the other hand, squid had 

squid or any other task could as well have more memory allocated than
rsync, so the oom killer heuristic could get wrong it too. There's
simply no way that the kernel will be able to do the right choice always
without user intervention, no-way. Adding the allocation rate of tasks
would be a good start, much better than the current oom killer heuristic
that will fail on any real non-desktop server usage with 90% of virtual
memory always allocated in non malicious tasks.

> already problems - so it was probably a good idea to kill it completely 
> to make room for other running programs.
> 
> I would like to know, if the killing of rsync was pure chance or if it 
> was systematically. I will try it again :-).

it was systemaltically. If will eventually be killed as you expect. The
only problem is that something else may be killed too in the meantime,
but the kernel can always do the wrong choice anyways, and being
guaranteed of no-deadlock is much better than the current status of
mainline.

we had some discussion with Andrew on l-k last month on how to be safe
and to still allow an heuristic to make a first-choice based on some
statistic, but that's low prio for me. no matter what we do the kernel
can always do the wrong thing and have to kill task without the
userspace being aware of the error condition.

Andrea

Re: Memory management in Kernel 2.4.x

[Peter Wächtler]

Andrea Arcangeli wrote:
> On Mon, May 27, 2002 at 03:45:55PM +0200, Peter Wächtler wrote:
> 
>>Andreas Hartmann wrote:
>>
>>>Zwane Mwaikambo wrote:
>>>
>>>
>>>
>>>>On Mon, 27 May 2002, Andreas Hartmann wrote:
>>>>
>>>>
>>>>
>>>>>rsync allocates all of the memory the machine has (256 MB RAM, 128 MB
>>>>>swap). When this occures, processes get killed like described in the
>>>>>posting before. The machine doesn't respond as long as the rsync -
>>>>>process isn't killed, because it fetches all the memory which gets free
>>>>>after a process has been killed.
>>>>>
>>>>>
>>>>And the rsync process never gets singled out? nice!
>>>>
>>>>
>>>Until it's killed by the kernel (if overcommitment isn't deactivated). If 
>>>overcommitment is deactivated, the services of the machine are dead 
>>>forever. There will be nothing, which kills such a process. Or am I wrong?
>>>
>>>
>>There is still the oom killer (Out Of Memory).
>>But it doesn't trigger and the machine pages "forever".
>>Usually kswapd eats the CPU then, discarding and reloading pages,
>>searching lists for pages to evict and so on.
>>
> 
> can you reproduce with 2.4.19pre9aa2? I expect at least the deadlock
> (if it's a deadlock and not a livelock) should go away.
> 
> Also I read in another email that somebody grown the per-socket buffer
> to hundred mbytes, if you do that on a 32bit arch with highmem you'll
> run into troubles, too much ZONE_NORMAL ram will be constantly pinned
> for the tcp pipeline and the machine can enter livelocks.
> 

Sorry for the confusion.
I was just trying to explain that without overcommitment there would be
the "normal" OOM handling. But I don't know this feature from the -ac
kernels. Am I wrong?

It's Andreas who has the problem with rsync being killed and the machine
seems to hang.
But I still think that the buffer cache has to be better restricted.
The vm is caching far too aggressively (but I never tried with -aa)

Re: Memory management in Kernel 2.4.x

[Bill Davidsen]

On 27 May 2002, Alan Cox wrote:

> On Mon, 2002-05-27 at 22:22, H. Peter Anvin wrote:
    <_snip_>
> > Well, if you can't fork a new process because that would push you into
> > overcommit, then you usually can't actually do anything useful on the
> > machine.
> 
> Thats actually easy to deal with and on my list for modes 4 and 5 (2 and
> 3 with root granted a reserved fraction)

It seems to me that selectively limiting the number of processes in a
pgroup, or selectively killing large RSS programs in a large pgroup
(non-root) would be one way to identify the processes which were either
clone/fork looping, or have children begetting children. After than
perhaps killing or restricting from the high numerical uid down. That
might tend to spare system and/or well-behaved processes.

Comment?

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.