can I bring Linux down by running "renice -20 cpu_intensive_process"?

can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Tomasz Chmielewski]

I have a Linux server (kernel 2.6.8.1 + Linux RAID1) which is a "backup" 
machine: it gets the files from other servers, compresses it, writes to 
the tape, checks md5sums etc.

It's been running for quite a bit, no problems with stability so far.

Yesterday, something happened though.

I was logged in remotely, and the system was running md5sum against a 30 
GB file.

I wanted the things to speed up a bit, and made "renice -20 <md5sum_pid>".

Few minutes after that I couldn't start any process, so I thought I made 
the system so busy with renice -20, that my SSH session probably 
disconnected.

In the morning, the system was still unavailable - I could ping it, I 
could telnet to any of the ports opened, but nothing more happened.

SSH was waiting forever after:

debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1


Nothing was displayed on the monitor (all black).

As I restarted the machine, I saw that the logging ends few minutes 
after I changed the priority of md5sum to -20.


So here is my question: is it possible to bring down the machine by 
simply doing "renice -20 cpu_intensive_process"?

As I said, this machine does heavy compression and md5sum calculations 
of big files every day, and was stable all the time - but stopped 
responding after I changed the priority of a CPU-intensive process to -20.

Coincidence and a hardware failure?

-- 
Tomasz Chmielewski
http://wpkg.org

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Jan Engelhardt]

>Subject: can I bring Linux down by running "renice -20 cpu_intensive_process"?
>
Depends on what the cpu_intensive_process does. If it tries to allocate 
lots of memory, maybe. If it's _just_ CPU (as in `perl -e '1 while 1'`), 
you get a chance that you can input some commands on a terminal to kill it.
SCHED_FIFO'ing or SCHED_RR'ing such a process is sudden death of course.

> I have a Linux server (kernel 2.6.8.1 + Linux RAID1) which is a "backup"
> machine: it gets the files from other servers, compresses it, writes to the
> tape, checks md5sums etc.
>
> It's been running for quite a bit, no problems with stability so far.
>
Why would you need it to run at -20 anyway?

> As I restarted the machine, I saw that the logging ends few minutes after I
> changed the priority of md5sum to -20.
>
> So here is my question: is it possible to bring down the machine by simply
> doing "renice -20 cpu_intensive_process"?
>
In case of md5sum: it should not be. At least you should have been able to 
unblank the console pressing any key, or have sysrq available.

> As I said, this machine does heavy compression and md5sum calculations of big
> files every day, and was stable all the time - but stopped responding after I
> changed the priority of a CPU-intensive process to -20.
>
> Coincidence and a hardware failure?
>
Sysrq+T (and/or +P) will tell you where the CPU is running.


Jan Engelhardt
-- 

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Tomasz Chmielewski]

Jan Engelhardt wrote:
>> Subject: can I bring Linux down by running "renice -20 cpu_intensive_process"?
>>
> Depends on what the cpu_intensive_process does. If it tries to allocate 
> lots of memory, maybe. If it's _just_ CPU (as in `perl -e '1 while 1'`), 
> you get a chance that you can input some commands on a terminal to kill it.
> SCHED_FIFO'ing or SCHED_RR'ing such a process is sudden death of course.
> 
>> I have a Linux server (kernel 2.6.8.1 + Linux RAID1) which is a "backup"
>> machine: it gets the files from other servers, compresses it, writes to the
>> tape, checks md5sums etc.
>>
>> It's been running for quite a bit, no problems with stability so far.
>>
> Why would you need it to run at -20 anyway?

Hmm, I hoped md5sum 30_gig would finish before I finish work to start 
writing new data on tape...


>> As I restarted the machine, I saw that the logging ends few minutes after I
>> changed the priority of md5sum to -20.
>>
>> So here is my question: is it possible to bring down the machine by simply
>> doing "renice -20 cpu_intensive_process"?
>>
> In case of md5sum: it should not be. At least you should have been able to 
> unblank the console pressing any key, or have sysrq available.

So in my case it just died for some reason (the console didn't unblank; 
the md5sum process should have ended long time ago).
On the other hand, the machine was responding to pings, and the ports 
were open, so it wasn't totally dead.

Hmm, so we can just speculate what it was.


-- 
Tomasz Chmielewski

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Måns Rullgård]

Jan Engelhardt <jengelh@linux01.gwdg.de> writes:

>>Subject: can I bring Linux down by running "renice -20
>>cpu_intensive_process"?
>>
> Depends on what the cpu_intensive_process does. If it tries to allocate 
> lots of memory, maybe. If it's _just_ CPU (as in `perl -e '1 while 1'`), 
> you get a chance that you can input some commands on a terminal to kill it.
> SCHED_FIFO'ing or SCHED_RR'ing such a process is sudden death of course.

Sysrq+n changes all realtime tasks to normal priority.

-- 
Måns Rullgård
mru@inprovide.com

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Jeffrey Hundstad]

Måns Rullgård wrote:

>Jan Engelhardt <jengelh@linux01.gwdg.de> writes:
>
>  
>
>>>Subject: can I bring Linux down by running "renice -20
>>>cpu_intensive_process"?
>>>
>>>      
>>>
>>Depends on what the cpu_intensive_process does. If it tries to allocate 
>>lots of memory, maybe. If it's _just_ CPU (as in `perl -e '1 while 1'`), 
>>you get a chance that you can input some commands on a terminal to kill it.
>>SCHED_FIFO'ing or SCHED_RR'ing such a process is sudden death of course.
>>    
>>
>
>Sysrq+n changes all realtime tasks to normal priority.
>
>  
>

Patient: "Doctor When I poke myself in the eye it hurts."
Doctor "Don't do that then."

-- 
Jeffrey Hundstad

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Måns Rullgård]

Jeffrey Hundstad <jeffrey.hundstad@mnsu.edu> writes:

> Måns Rullgård wrote:
>
>>Jan Engelhardt <jengelh@linux01.gwdg.de> writes:
>>
>>>>Subject: can I bring Linux down by running "renice -20
>>>>cpu_intensive_process"?
>>>>
>>> Depends on what the cpu_intensive_process does. If it tries to
>>> allocate lots of memory, maybe. If it's _just_ CPU (as in `perl -e
>>> '1 while 1'`), you get a chance that you can input some commands on
>>> a terminal to kill it.
>>>SCHED_FIFO'ing or SCHED_RR'ing such a process is sudden death of course.
>>
>>Sysrq+n changes all realtime tasks to normal priority.
>
> Patient: "Doctor When I poke myself in the eye it hurts."
> Doctor "Don't do that then."

A bug might cause an otherwise well-behaved realtime process to start
spinning in a loop or something.  Having a way to stop it is good,
IMHO.

-- 
Måns Rullgård
mru@inprovide.com

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Luke-Jr]

On Friday 10 March 2006 22:01, Måns Rullgård wrote:
> Sysrq+n changes all realtime tasks to normal priority.

Would the kernel's main loop (where I presume Sysreq is handled) get a chance 
to run with a constantly busy realtime task?

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Lee Revell]

On Sun, 2006-03-12 at 01:41 +0000, Luke-Jr wrote:
> On Friday 10 March 2006 22:01, Måns Rullgård wrote:
> > Sysrq+n changes all realtime tasks to normal priority.
> 
> Would the kernel's main loop (where I presume Sysreq is handled) get a chance 
> to run with a constantly busy realtime task?

No, other tasks will not get to run at all.  A SCHED_FIFO task that
spins forever is a fatal bug.  Making it less dangerous would require
making it less useful.

Lee

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Lee Revell]

On Sun, 2006-03-12 at 01:41 +0000, Luke-Jr wrote:
> On Friday 10 March 2006 22:01, Måns Rullgård wrote:
> > Sysrq+n changes all realtime tasks to normal priority.
> 
> Would the kernel's main loop (where I presume Sysreq is handled) get a chance 
> to run with a constantly busy realtime task?

Sorry I was thinking of the -rt kernel in my previous post - in mainline
this would be effective.  In the -rt kernel you are screwed if the
spinning RT task is higher priority than the keyboard IRQ thread.

Lee

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Lee Revell]

On Fri, 2006-03-10 at 22:01 +0000, Måns Rullgård wrote:
> Jan Engelhardt <jengelh@linux01.gwdg.de> writes:
> 
> >>Subject: can I bring Linux down by running "renice -20
> >>cpu_intensive_process"?
> >>
> > Depends on what the cpu_intensive_process does. If it tries to allocate 
> > lots of memory, maybe. If it's _just_ CPU (as in `perl -e '1 while 1'`), 
> > you get a chance that you can input some commands on a terminal to kill it.
> > SCHED_FIFO'ing or SCHED_RR'ing such a process is sudden death of course.
> 
> Sysrq+n changes all realtime tasks to normal priority.
> 

A nice -20 SCHED_OTHER task is not realtime, only SCHED_FIFO and
SCHED_RR.

Lee

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Måns Rullgård]

Lee Revell <rlrevell@joe-job.com> writes:

> On Fri, 2006-03-10 at 22:01 +0000, Måns Rullgård wrote:
>> Jan Engelhardt <jengelh@linux01.gwdg.de> writes:
>> 
>> >>Subject: can I bring Linux down by running "renice -20
>> >>cpu_intensive_process"?
>> >>
>> > Depends on what the cpu_intensive_process does. If it tries to
>> > allocate lots of memory, maybe. If it's _just_ CPU (as in `perl
>> > -e '1 while 1'`), you get a chance that you can input some
>> > commands on a terminal to kill it.  SCHED_FIFO'ing or
>> > SCHED_RR'ing such a process is sudden death of course.
>> 
>> Sysrq+n changes all realtime tasks to normal priority.
>> 
>
> A nice -20 SCHED_OTHER task is not realtime, only SCHED_FIFO and
> SCHED_RR.

Maybe extending sysrq+n to lower the priority of -20 tasks would be a
good idea.

-- 
Måns Rullgård
mru@inprovide.com

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Bill Davidsen]

Måns Rullgård wrote:
> Lee Revell <rlrevell@joe-job.com> writes:
> 
>> On Fri, 2006-03-10 at 22:01 +0000, Måns Rullgård wrote:
>>> Jan Engelhardt <jengelh@linux01.gwdg.de> writes:
>>>
>>>>> Subject: can I bring Linux down by running "renice -20
>>>>> cpu_intensive_process"?
>>>>>
>>>> Depends on what the cpu_intensive_process does. If it tries to
>>>> allocate lots of memory, maybe. If it's _just_ CPU (as in `perl
>>>> -e '1 while 1'`), you get a chance that you can input some
>>>> commands on a terminal to kill it.  SCHED_FIFO'ing or
>>>> SCHED_RR'ing such a process is sudden death of course.
>>> Sysrq+n changes all realtime tasks to normal priority.
>>>
>> A nice -20 SCHED_OTHER task is not realtime, only SCHED_FIFO and
>> SCHED_RR.
> 
> Maybe extending sysrq+n to lower the priority of -20 tasks would be a
> good idea.
> 
If it runs before the keyboard thread it doesn't matter... But why 
should this hang anything, when there should be enough i/o to get out of 
the user process. There's a good fix for this, don't give this guy root 
any more ;-)

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Lee Revell]

On Thu, 2006-03-16 at 16:15 -0500, Bill Davidsen wrote:
> There's a good fix for this, don't give this guy root 
> any more ;-) 

Minor nit: s/root/realtime privileges/.  Since 2.6.12 these have been
decoupled.  No official distro release supports it OOTB yet (the
upcoming Ubuntu Dapper will).

Lee

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Måns Rullgård]

Bill Davidsen <davidsen@tmr.com> writes:

> Måns Rullgård wrote:
>> Lee Revell <rlrevell@joe-job.com> writes:
>>
>>> On Fri, 2006-03-10 at 22:01 +0000, Måns Rullgård wrote:
>>>> Jan Engelhardt <jengelh@linux01.gwdg.de> writes:
>>>>
>>>>>> Subject: can I bring Linux down by running "renice -20
>>>>>> cpu_intensive_process"?
>>>>>>
>>>>> Depends on what the cpu_intensive_process does. If it tries to
>>>>> allocate lots of memory, maybe. If it's _just_ CPU (as in `perl
>>>>> -e '1 while 1'`), you get a chance that you can input some
>>>>> commands on a terminal to kill it.  SCHED_FIFO'ing or
>>>>> SCHED_RR'ing such a process is sudden death of course.
>>>> Sysrq+n changes all realtime tasks to normal priority.
>>>>
>>> A nice -20 SCHED_OTHER task is not realtime, only SCHED_FIFO and
>>> SCHED_RR.
>> Maybe extending sysrq+n to lower the priority of -20 tasks would be a
>> good idea.
>>
> If it runs before the keyboard thread it doesn't matter...

Of course not, but that's not generally the case.

> But why should this hang anything, when there should be enough i/o
> to get out of the user process. There's a good fix for this, don't
> give this guy root any more ;-)

Ever heard of bugs?  Anyone developing a program can make a mistake.
If the program runs with realtime scheduling a bug that makes it enter
an infinite loop (or do something else that hogs the CPU) can be
difficult to find since it rather efficiently locks you out.

-- 
Måns Rullgård
mru@inprovide.com

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Mike Galbraith]

On Thu, 2006-03-16 at 22:51 +0000, Måns Rullgård wrote:
> Bill Davidsen <davidsen@tmr.com> writes:
> 
> > Måns Rullgård wrote:
> >> Maybe extending sysrq+n to lower the priority of -20 tasks would be a
> >> good idea.
> >>
> > If it runs before the keyboard thread it doesn't matter...
> 
> Of course not, but that's not generally the case.
> 
> > But why should this hang anything, when there should be enough i/o
> > to get out of the user process. There's a good fix for this, don't
> > give this guy root any more ;-)
> 
> Ever heard of bugs?  Anyone developing a program can make a mistake.
> If the program runs with realtime scheduling a bug that makes it enter
> an infinite loop (or do something else that hogs the CPU) can be
> difficult to find since it rather efficiently locks you out.

Given that someone has already determined that installing a safety valve
for RT tasks was worth while, and given that there is practically no
difference between a nice -20 and the lowest RT priority, seems to me
that extending that safety valve to cover reniced tasks is the
obviously-correct thing to do.  I think you should submit a patch.

	-Mike

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Måns Rullgård]

Mike Galbraith <efault@gmx.de> writes:

> On Thu, 2006-03-16 at 22:51 +0000, Måns Rullgård wrote:
>> Bill Davidsen <davidsen@tmr.com> writes:
>> 
>> > Måns Rullgård wrote:
>> >> Maybe extending sysrq+n to lower the priority of -20 tasks would be a
>> >> good idea.
>> >>
>> > If it runs before the keyboard thread it doesn't matter...
>> 
>> Of course not, but that's not generally the case.
>> 
>> > But why should this hang anything, when there should be enough i/o
>> > to get out of the user process. There's a good fix for this, don't
>> > give this guy root any more ;-)
>> 
>> Ever heard of bugs?  Anyone developing a program can make a mistake.
>> If the program runs with realtime scheduling a bug that makes it enter
>> an infinite loop (or do something else that hogs the CPU) can be
>> difficult to find since it rather efficiently locks you out.
>
> Given that someone has already determined that installing a safety valve
> for RT tasks was worth while, and given that there is practically no
> difference between a nice -20 and the lowest RT priority, seems to me
> that extending that safety valve to cover reniced tasks is the
> obviously-correct thing to do.  I think you should submit a patch.

Something like this ought to do it (untested):

--- kernel/sched.c.orig 2006-02-09 23:41:57.000000000 +0000
+++ kernel/sched.c      2006-03-17 22:16:46.257298014 +0000
@@ -5681,21 +5681,22 @@
 
        read_lock_irq(&tasklist_lock);
        for_each_process (p) {
-               if (!rt_task(p))
-                       continue;
+               if (rt_task(p)) {
+                       rq = task_rq_lock(p, &flags);
 
-               rq = task_rq_lock(p, &flags);
-
-               array = p->array;
-               if (array)
-                       deactivate_task(p, task_rq(p));
-               __setscheduler(p, SCHED_NORMAL, 0);
-               if (array) {
-                       __activate_task(p, task_rq(p));
-                       resched_task(rq->curr);
+                       array = p->array;
+                       if (array)
+                               deactivate_task(p, task_rq(p));
+                       __setscheduler(p, SCHED_NORMAL, 0);
+                       if (array) {
+                               __activate_task(p, task_rq(p));
+                               resched_task(rq->curr);
+                       }
+
+                       task_rq_unlock(rq, &flags);
+               } else if (TASK_NICE(p) == -20) {
+                       set_user_nice(p, 0);
                }
-
-               task_rq_unlock(rq, &flags);
        }
        read_unlock_irq(&tasklist_lock);
 }


-- 
Måns Rullgård
mru@inprovide.com

Re: can I bring Linux down by running "renice -20 cpu_intensive_process"?

[Bill Davidsen]

Måns Rullgård wrote:

>Bill Davidsen <davidsen@tmr.com> writes:
>
>  
>
>>Måns Rullgård wrote:
>>    
>>
>>>Lee Revell <rlrevell@joe-job.com> writes:
>>>
>>>      
>>>
>>>>On Fri, 2006-03-10 at 22:01 +0000, Måns Rullgård wrote:
>>>>        
>>>>
>>>>>Jan Engelhardt <jengelh@linux01.gwdg.de> writes:
>>>>>
>>>>>          
>>>>>
>>>>>>>Subject: can I bring Linux down by running "renice -20
>>>>>>>cpu_intensive_process"?
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>Depends on what the cpu_intensive_process does. If it tries to
>>>>>>allocate lots of memory, maybe. If it's _just_ CPU (as in `perl
>>>>>>-e '1 while 1'`), you get a chance that you can input some
>>>>>>commands on a terminal to kill it.  SCHED_FIFO'ing or
>>>>>>SCHED_RR'ing such a process is sudden death of course.
>>>>>>            
>>>>>>
>>>>>Sysrq+n changes all realtime tasks to normal priority.
>>>>>
>>>>>          
>>>>>
>>>>A nice -20 SCHED_OTHER task is not realtime, only SCHED_FIFO and
>>>>SCHED_RR.
>>>>        
>>>>
>>>Maybe extending sysrq+n to lower the priority of -20 tasks would be a
>>>good idea.
>>>
>>>      
>>>
>>If it runs before the keyboard thread it doesn't matter...
>>    
>>
>
>Of course not, but that's not generally the case.
>
>  
>
>>But why should this hang anything, when there should be enough i/o
>>to get out of the user process. There's a good fix for this, don't
>>give this guy root any more ;-)
>>    
>>
>
>Ever heard of bugs?  Anyone developing a program can make a mistake.
>If the program runs with realtime scheduling a bug that makes it enter
>an infinite loop (or do something else that hogs the CPU) can be
>difficult to find since it rather efficiently locks you out.
>
>  
>
Please google "emoticons" and find out what those funny characters at 
the end of the of the paragraph you quoted really mean. Sheesh!

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979