Is it possible to give the user the option to cancel forkbombs?

Is it possible to give the user the option to cancel forkbombs?

[Martin Olsson]

Dear kernel hackers,

This is a message from below 0x7FFFFFFF. Please look at this bug (it's 
not a new concept but still):
https://bugs.launchpad.net/ubuntu/+bug/163185

I'm no expert but I'd guess the "complete freeze" part of the bug has to 
do with the kernel, no? It would be nice to have a system which always 
lets me choose to abort stuff regardless of what kind of program mess I 
accidently started. Sort of like how you can "always" count on 
CTRL-ALT-DEL to work. Maybe it's possible to set some kind of 
MAX_PROCESS_COUNT for each user (I don't know) but it looks like this is 
not done by default in many distros today?

I mean, this effectively took down both my laptop and made the server of 
my shared web hosting company unresponsive for >10 mins (maybe it had 
some auto-reboot mechanism, I'm not sure).



		Martin

Re: Is it possible to give the user the option to cancel forkbombs?

[Alan Cox]

On Fri, 16 Nov 2007 21:51:27 -0800
Martin Olsson <mnemo@minimum.se> wrote:

> Dear kernel hackers,
> 
> This is a message from below 0x7FFFFFFF. Please look at this bug (it's 
> not a new concept but still):
> https://bugs.launchpad.net/ubuntu/+bug/163185

It seems to want people to register to view it. I guess Ubuntu should fix
launchpad then we can see the bug report

Alan

Re: Is it possible to give the user the option to cancel forkbombs?

[Martin Olsson]

Sorry about that, I checked the "has security impact" checkbox and that 
marked it as private by default. This is a very well known problem 
though so keeping secret certainly does not make sense. I have manually 
removed the "private" flag now.

The content of the bug report was as follows:
---------------------------------------------

Repro steps:

1. Install gutsy gibbon (or probably any ubuntu)
2. Start a gnome terminal
3. Run this command:

    :(){ :|:& };:

4. Ubuntu starts to work furiously, after less than a second terminal 
gets flooded with "low resources" message, and within a few seconds the 
whole machine breaks down complete to the point where no a single pixel 
is updated and the mouse cannot be moved at all. It's not possible to 
escape to a ALT-Fn console terminal and CTRL-ALT-DEL does not work.

Okay, so this is not as bad as winnuke.exe because it's not remote but I 
just did it on my shared hosting co and their server went down. And I 
mean seriously, there should be a way for a user to abort stuff that 
hogs resources this type of complete breakdown is NEVER acceptible. I 
had to power of the machine and my file system got royally screwed (long 
fsck etc).

Some of you might say this is like the oldest trick in the book, yada 
yada yada...


		Martin



Alan Cox wrote:
> On Fri, 16 Nov 2007 21:51:27 -0800
> Martin Olsson <mnemo@minimum.se> wrote:
> 
>> Dear kernel hackers,
>>
>> This is a message from below 0x7FFFFFFF. Please look at this bug (it's 
>> not a new concept but still):
>> https://bugs.launchpad.net/ubuntu/+bug/163185
> 
> It seems to want people to register to view it. I guess Ubuntu should fix
> launchpad then we can see the bug report
> 
> Alan
> -

Re: Is it possible to give the user the option to cancel forkbombs?

[Alan Cox]

> 3. Run this command:
> 
>     :(){ :|:& };:
> 
> 4. Ubuntu starts to work furiously, after less than a second terminal 
> gets flooded with "low resources" message, 

Set your system with suitable process and user resource limits. Also I'd
suggest setting memory overcommit to strict.

Our default behaviour is relaxed as it is usually the right thing for a
generic desktop box.

Alan

Re: Is it possible to give the user the option to cancel forkbombs?

[Dane Mutters]

I thought you might find this helpful.  (I brought this issue up with
the Slackware folks once, and they told me basically this.)

http://wiki.craz1.homelinux.com/index.php/Linux:Security:Forkbomb

I was also told that the ability to spawn such rampant forks/processes
is controlled by default in Debian.  Is this the case?

Here is an LQ thread where I brought it up:

http://www.linuxquestions.org/questions/linux-security-4/how-can-i-prevent-forkbombs-338560/

I would like to see something done about this, with Ubuntu as popular as
it is, even as a server in some cases.  Is there a way that in the
future, one could simply download a package or click a box or something
and have a limit set, like the links suggest?  That would make things
just "that much" more convenient for system administrators (and might
help them/us to remember to set these limits, too...).

Thanks.

-Dane


On Fri, 2007-11-16 at 23:04 -0800, Martin Olsson wrote:
> Sorry about that, I checked the "has security impact" checkbox and that 
> marked it as private by default. This is a very well known problem 
> though so keeping secret certainly does not make sense. I have manually 
> removed the "private" flag now.
> 
> The content of the bug report was as follows:
> ---------------------------------------------
> 
> Repro steps:
> 
> 1. Install gutsy gibbon (or probably any ubuntu)
> 2. Start a gnome terminal
> 3. Run this command:
> 
>     :(){ :|:& };:
> 
> 4. Ubuntu starts to work furiously, after less than a second terminal 
> gets flooded with "low resources" message, and within a few seconds the 
> whole machine breaks down complete to the point where no a single pixel 
> is updated and the mouse cannot be moved at all. It's not possible to 
> escape to a ALT-Fn console terminal and CTRL-ALT-DEL does not work.
> 
> Okay, so this is not as bad as winnuke.exe because it's not remote but I 
> just did it on my shared hosting co and their server went down. And I 
> mean seriously, there should be a way for a user to abort stuff that 
> hogs resources this type of complete breakdown is NEVER acceptible. I 
> had to power of the machine and my file system got royally screwed (long 
> fsck etc).
> 
> Some of you might say this is like the oldest trick in the book, yada 
> yada yada...
> 
> 
> 		Martin
> 
> 
> 
> Alan Cox wrote:
> > On Fri, 16 Nov 2007 21:51:27 -0800
> > Martin Olsson <mnemo@minimum.se> wrote:
> > 
> >> Dear kernel hackers,
> >>
> >> This is a message from below 0x7FFFFFFF. Please look at this bug (it's 
> >> not a new concept but still):
> >> https://bugs.launchpad.net/ubuntu/+bug/163185
> > 
> > It seems to want people to register to view it. I guess Ubuntu should fix
> > launchpad then we can see the bug report
> > 
> > Alan
> > -
> 
> 

Re: Is it possible to give the user the option to cancel forkbombs?

[Peter Zijlstra]

On Fri, 2007-11-16 at 22:45 -0800, Dane Mutters wrote:
> I thought you might find this helpful.  (I brought this issue up with
> the Slackware folks once, and they told me basically this.)
> 
> http://wiki.craz1.homelinux.com/index.php/Linux:Security:Forkbomb
> 
> I was also told that the ability to spawn such rampant forks/processes
> is controlled by default in Debian.  Is this the case?
> 
> Here is an LQ thread where I brought it up:
> 
> http://www.linuxquestions.org/questions/linux-security-4/how-can-i-prevent-forkbombs-338560/
> 
> I would like to see something done about this, with Ubuntu as popular as
> it is, even as a server in some cases.  Is there a way that in the
> future, one could simply download a package or click a box or something
> and have a limit set, like the links suggest?  That would make things
> just "that much" more convenient for system administrators (and might
> help them/us to remember to set these limits, too...).

If you don't know which limits to set and need a package for them, your
job title should not be system administrator.

Re: Is it possible to give the user the option to cancel forkbombs?

[Alan Cox]

> > I would like to see something done about this, with Ubuntu as popular as
> > it is, even as a server in some cases.  Is there a way that in the
> > future, one could simply download a package or click a box or something
> > and have a limit set, like the links suggest?  That would make things
> > just "that much" more convenient for system administrators (and might
> > help them/us to remember to set these limits, too...).
> 
> If you don't know which limits to set and need a package for them, your
> job title should not be system administrator.

Thats a very arrogant viewpoint. I don't have to be a TV engineer to use
my television.

Distributions should be providing sensible defaults out of the box. The
kernel already provides them the mechanisms.

Alan

Re: Is it possible to give the user the option to cancel forkbombs?

[Andi Kleen]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

>> > I would like to see something done about this, with Ubuntu as popular as
>> > it is, even as a server in some cases.  Is there a way that in the
>> > future, one could simply download a package or click a box or something
>> > and have a limit set, like the links suggest?  That would make things
>> > just "that much" more convenient for system administrators (and might
>> > help them/us to remember to set these limits, too...).
>> 
>> If you don't know which limits to set and need a package for them, your
>> job title should not be system administrator.
>
> Thats a very arrogant viewpoint. I don't have to be a TV engineer to use
> my television.
>
> Distributions should be providing sensible defaults out of the box. The
> kernel already provides them the mechanisms.

If you mean ulimits -- the current ulimits are not very useful imho.   Or 
at least not for handling the general resource resumption problem. They
work in some limited circumstances for well known special purpose workloads,
but not more. 

I don't know why people here always claiming they are (have they ever
tried to use them on your desktop in general?).

The equation to limit resources is something like:

MAX_PROCESSES_PER_UID*(MAX_MEM_PER_PROCESS+MAX_FD_PER_PROCESS+...)=TOTAL_RESOURCES_UID

To get an effective limit you either need MAX_PROCESSES or MAX_MEM to unusable
low numbers breaking a lot of applications. And you cannot generally predict 
in advance if the workload will need high MAX_MEM or high MAX_PROCESSES
or a combination of both.

That is why distributions usually don't set them. Or at least not for
in a setup to protect the system fully because that is not possible.

e.g. SUSE supports a optional default limit that sets the max memory
per process to below the available memory, but the user can still
easily circumvent that by starting multiple processes.

And I'm not really talking about a general bean counter for all kernel
objects here, just the "ordinary easy resources" like processes and virtual
memory.

Pretty much all the per process limits would need to be per uid to be really
useful in general. I'm hoping that we'll get some of that out of the recent
container work. e.g. if there was a "max mem per uid" then you could actually
set it to a sensible value. Even better than max mem per uid would be probably
"max total memory used as fraction of the system" or something like that --
that would also handle things like memory hotplug etc. well. Probably
would also need to be separated into swap space and real memory; at least
as long as the Linux swap algorithms are so slow.

Regarding the fork bomb problem: 

the uid cgroup scheduler that went in recently should already help a little,
although to be really effective against fork bombs for a desktop user you
would probably need multiple cgroups per uid (so e.g. that the window
manager is also protected against other processes running on the same
uid and you can then still kill the nasty processes from it).

That is why I didn't like hardcoding the uid in the scheduler too much.

-Andi

Re: Is it possible to give the user the option to cancel forkbombs?

[Herbert Xu]

Andi Kleen <andi@firstfloor.org> wrote:
>
> Pretty much all the per process limits would need to be per uid to be really
> useful in general. I'm hoping that we'll get some of that out of the recent

You've hit the nail on the head.  I've always liked to have
per-uid limits on network sockets too as otherwise you either
have to cope with some rogue user taking all your TCP memory
away or worse all of your kernel memory.  Luckily the former
isn't fatal because each TCP socket has a guaranteed 4K so they
will still operate but it's still suboptimal.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Is it possible to give the user the option to cancel forkbombs?

[Martin Olsson]

(taking this reply offline, so this is mostly intended for Alan)

Peter Zijlstra wrote:
> 
> If you don't know which limits to set and need a package for them, your
> job title should not be system administrator.
> 

In theory I agree with you Peter but in practice a lot of low-skill 
people have to install and administer servers. I think the pragmatic and 
realistic approach would be to accept that many admins are not perfect, 
far from it.

Also, this bug involves many aspects and it's possible to argue what 
exactly is the bug. Even if we don't change any default values, we can 
still change the kernel so that it survives a fork bomb. After all, the 
Vista kernel does survive a fork bomb although the system as a whole 
gets unusable.

### On Ubuntu I currently see:
1. Launch fork bomb
2. System gets to a complete freeze (I can't move mouse and not a single 
pixel is updated on the screen).

### On Vista I currently see:
1. Launch fork bomb
2. System gets very slow to the point where the system is unusable (but 
I can always mouse around the mouse freely with no sluggishness and the 
screen updates continue to make progress even though it takes 1-3 
seconds between each update). I can still open and close the start menu 
if I got the patience to wait >45 seconds for it each time.

I don't think that setting a max process count by default is a 
good/viable solution. But the kernel could still be changed so that it 
doesn't hose itself completely.



		Martin

Re: Is it possible to give the user the option to cancel forkbombs?

[Peter Zijlstra]

On Sat, 2007-11-17 at 09:42 -0800, Martin Olsson wrote:
> (taking this reply offline, so this is mostly intended for Alan)
> 
> Peter Zijlstra wrote:
> > 
> > If you don't know which limits to set and need a package for them, your
> > job title should not be system administrator.
> > 
> 
> In theory I agree with you Peter but in practice a lot of low-skill 
> people have to install and administer servers. I think the pragmatic and 
> realistic approach would be to accept that many admins are not perfect, 
> far from it.

Sure, I know that. But providing a package to tune a system setting
doesn't sound like a sane option either. This low-skill admin will
likely not know about this package either.

And between: 'install this package and magic happens', or 'If you have
problem A, fiddle know B so and so' I prefer the latter.

So my suggestion would be to provide good, simple and direct sys-ad 101
documentation for your specific distro.

Of course, your -server distro might have different default settings
from your desktop line.

> Also, this bug involves many aspects and it's possible to argue what 
> exactly is the bug. Even if we don't change any default values, we can 
> still change the kernel so that it survives a fork bomb. After all, the 
> Vista kernel does survive a fork bomb although the system as a whole 
> gets unusable.
> 
> ### On Ubuntu I currently see:
> 1. Launch fork bomb
> 2. System gets to a complete freeze (I can't move mouse and not a single 
> pixel is updated on the screen).
> 
> ### On Vista I currently see:
> 1. Launch fork bomb
> 2. System gets very slow to the point where the system is unusable (but 
> I can always mouse around the mouse freely with no sluggishness and the 
> screen updates continue to make progress even though it takes 1-3 
> seconds between each update). I can still open and close the start menu 
> if I got the patience to wait >45 seconds for it each time.
> 
> I don't think that setting a max process count by default is a 
> good/viable solution. But the kernel could still be changed so that it 
> doesn't hose itself completely.

Agreed, the kernel could perhaps handle it a little more graceful. Do
you happen to know what makes current Linux suck? Is it the excess of
tasks? (I've only ran a few thousand loops on CFS and that seemed to
work). Or is it the memory trashing that causes most problems?

My guess would be memory, and the problem is that the current overcommit
settings allow for a good experience for a lot of things, the downside
is this DoS scenario.

Our OOM killer often killing totally irrelevant processes doesn't help
either.

Re: Is it possible to give the user the option to cancel forkbombs?

[Diego Calleja]

El Sat, 17 Nov 2007 09:42:51 -0800, Martin Olsson <mnemo@minimum.se> escribió:

> I don't think that setting a max process count by default is a 
> good/viable solution. 


I don't see why...OS X had a default limit of 100 processes per uid (increased
to 266 in 10.5) and "it works" (many people notices it, but it's not surprising
since the limit is too restrictive).

If you don't have limits, you can't avoid starvation easily. From my experience,
since I use CFS, fork/compile bombs (forgetting to put a number after make -j...)
are very sluggish mainly because the whole graphic subsystem is paged out.

Re: Is it possible to give the user the option to cancel forkbombs?

[Dane Mutters]

On Sat, 2007-11-17 at 16:53 +0100, Diego Calleja wrote:
> El Sat, 17 Nov 2007 09:42:51 -0800, Martin Olsson <mnemo@minimum.se> escribió:
> 
> > I don't think that setting a max process count by default is a 
> > good/viable solution. 
> 
> 
> I don't see why...OS X had a default limit of 100 processes per uid (increased
> to 266 in 10.5) and "it works" (many people notices it, but it's not surprising
> since the limit is too restrictive).
> 
> If you don't have limits, you can't avoid starvation easily. From my experience,
> since I use CFS, fork/compile bombs (forgetting to put a number after make -j...)
> are very sluggish mainly because the whole graphic subsystem is paged out.

I don't know if this is at all feasible, but is it possible to have a
mechanism that would detect a fork bomb in progress and either stop the
fork, or allow the user to cancel the operation?  For example, are there
any legitimate processes (i.e. ones that really need to fork like crazy)
that would need to generate 200+ processes in less than 1 second?

(Note: I'm not a programmer; I'm just throwing out the idea.)

-Dane

Re: Is it possible to give the user the option to cancel forkbombs?

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

On Sat, 17 Nov 2007 09:55:01 -0800
Dane Mutters <dmutters@gmail.com> wrote:

> I don't know if this is at all feasible, but is it possible to have a
> mechanism that would detect a fork bomb in progress and either stop the
> fork, or allow the user to cancel the operation?  For example, are there
> any legitimate processes (i.e. ones that really need to fork like crazy)
> that would need to generate 200+ processes in less than 1 second?
> 
> (Note: I'm not a programmer; I'm just throwing out the idea.)
> 

If the parent PID of the new task is exported through TASKSTATS, you can
do it already in userspace. If not, that data should be exported.

Then you could write a root daemon using netlink, set it to RT priority
and create an inheritable counter in it to thwart binary forking.
The counter would be cleared every x seconds.
No need to do it in the kernel.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Is it possible to give the user the option to cancel forkbombs?

[(``-_-´´) -- Fernando]

[-- Attachment #1: Type: text/plain, Size: 1810 bytes --]

On Saturday 17 November 2007 07:04:48 Martin Olsson wrote:
> Repro steps:
> 
> 1. Install gutsy gibbon (or probably any ubuntu)
> 2. Start a gnome terminal
> 3. Run this command:
> 
>     :(){ :|:& };:
> 
> 4. Ubuntu starts to work furiously, after less than a second terminal 
> gets flooded with "low resources" message, and within a few seconds the 
> whole machine breaks down complete to the point where no a single pixel 
> is updated and the mouse cannot be moved at all. It's not possible to 
> escape to a ALT-Fn console terminal and CTRL-ALT-DEL does not work.
> 
> Okay, so this is not as bad as winnuke.exe because it's not remote but I 
> just did it on my shared hosting co and their server went down. And I 
> mean seriously, there should be a way for a user to abort stuff that 
> hogs resources this type of complete breakdown is NEVER acceptible. I 
> had to power of the machine and my file system got royally screwed (long 
> fsck etc).
> 
> Some of you might say this is like the oldest trick in the book, yada 
> yada yada...
> 
> 		Martin

I used to see stuff like this happening on my University students test servers. Once they started doing forks inside for(;;), the server would go down.
Then they replaced the servers by vwmare machines, and now reboots are faster.

But I would like to see installed by default some script like AND(Auto Nice Daemon) or CpuLimit locking any and every process to a maximum of 90% after X seconds. That way one could allways launch a new command and kill it.
Of course, this is assuming that disk I/O is not our main prob, 'cause if that happens there's no way out.

-- 
BUGabundo  :o)
(``-_-´´)	http://Ubuntu.BUGabundo.net
Linux user #443786    GPG key 1024D/A1784EBB
My new micro-blog @ http://BUGabundo.net

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Is it possible to give the user the option to cancel forkbombs?

[David Newall]

(``-_-´´) -- Fernando wrote:
> I used to see stuff like this happening on my University students test servers. Once they started doing forks inside for(;;), the server would go down.
> Then they replaced the servers by vwmare machines, and now reboots are faster.
>   

UNIX (and Linux) already has a feature for use in case of forkbombs, 
namely keeping a shell available, at a high priority, from which you 
manage a wide range of undesired situations.  There isn't, that I know 
of, any program which responds to a magic key and does the work, but I 
think it can be written without requiring any additional kernel support.

Re: Is it possible to give the user the option to cancel forkbombs?

[Diego Calleja]

El Fri, 16 Nov 2007 21:51:27 -0800, Martin Olsson <mnemo@minimum.se> escribió:

> Dear kernel hackers,
> 
> This is a message from below 0x7FFFFFFF. Please look at this bug (it's 
> not a new concept but still):
> https://bugs.launchpad.net/ubuntu/+bug/163185

Can't see that page:
----------------------
Not allowed here

Sorry, you don't have permission to access this page. 
----------------


> CTRL-ALT-DEL to work. Maybe it's possible to set some kind of 
> MAX_PROCESS_COUNT for each user (I don't know) but it looks like this is 
> not done by default in many distros today?

Per-user process limits have been there for a long time, most of distros don't
use them and don't offer GUIs to configure them.