RE: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

RE: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Ian Collinson]

Andi Kleen wrote:
> Fixing it would require boosting keventd's priority either globally 
> or temporarily. E.g. if the original reporter could put this
> (untested/uncompiled) at the beginning of
kernel/context.c:context_thread():
>
>     current->policy = SCHED_RR;
>     current->rt_priority = 99;
>
> it could fix his problem.

OK, I've tried the above fix on 2.4.17, and it seems to work.  Thanks Andi.
We can now get into the box on a high priority console after provoking a
lockup with a lower priority, CPU hogging realtime process.

Are there any potentially negative consequences of this fix, apart from
those already mentioned?

I certainly vote for this feature being preserved, as it is extremely useful
for debugging realtime priority apps.  FYI, we narrowed it down to breaking
in either 2.4.10-pre11 or pre12. 

Cheers
Ian


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Imerge Limited                          Tel :- +44 (0)1954 783600 
Unit 6 Bar Hill Business Park           Fax :- +44 (0)1954 783601 
Saxon Way                               Web :- http://www.imerge.co.uk 
Bar Hill 
Cambridge 
CB3 8SL 
United Kingdom 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Andi Kleen]

On Wed, Jun 05, 2002 at 01:53:06PM +0200, Ian Collinson wrote:

> 
> Are there any potentially negative consequences of this fix, apart from
> those already mentioned?

I don't think so. 

It could still fail when you install a prio=99, SCHED_FIFO process.

> I certainly vote for this feature being preserved, as it is extremely useful
> for debugging realtime priority apps.  FYI, we narrowed it down to breaking
> in either 2.4.10-pre11 or pre12. 

That was when the low latency console changes went in. Before that console
switches could interrupt scheduling for a long time, causing problems
for other realtime people. The change was to move the expensive parts
of the console switch to keventd.

-Andi

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[george anzinger]

Andi Kleen wrote:
> 
> On Wed, Jun 05, 2002 at 01:53:06PM +0200, Ian Collinson wrote:
> 
> >
> > Are there any potentially negative consequences of this fix, apart from
> > those already mentioned?
> 
> I don't think so.
> 
> It could still fail when you install a prio=99, SCHED_FIFO process.
> 
> > I certainly vote for this feature being preserved, as it is extremely useful
> > for debugging realtime priority apps.  FYI, we narrowed it down to breaking
> > in either 2.4.10-pre11 or pre12.
> 
> That was when the low latency console changes went in. Before that console
> switches could interrupt scheduling for a long time, causing problems
> for other realtime people. The change was to move the expensive parts
> of the console switch to keventd.

So that means that, with the above change to prio 99, we
reintroduce the latency problem, only now it is in a task
(keventd) and not an interrupt?  (I know, I know, the work
has to be done somewhere.  At least this way we can control
what priority level it is done at.  I.e. this is a step in
the right direction.  I just what folks to be aware of the
latency issue and where it is.)

For what its worth, you can change the priority of keventd
AFTER a system is up.  Robert Love's real time tools contain
a program (rt I think) that will do this for you.  Just
follow the URL for preemption in my sig. file and look
around.

-- 
George Anzinger   george@mvista.com
High-res-timers: 
http://sourceforge.net/projects/high-res-timers/
Real time sched:  http://sourceforge.net/projects/rtsched/
Preemption patch:
http://www.kernel.org/pub/linux/kernel/people/rml

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Robert Love]

On Wed, 2002-06-05 at 11:05, george anzinger wrote:

> So that means that, with the above change to prio 99, we
> reintroduce the latency problem, only now it is in a task
> (keventd) and not an interrupt?  (I know, I know, the work
> has to be done somewhere.  At least this way we can control
> what priority level it is done at.  I.e. this is a step in
> the right direction.  I just what folks to be aware of the
> latency issue and where it is.)

Good point.

> For what its worth, you can change the priority of keventd
> AFTER a system is up.  Robert Love's real time tools contain
> a program (rt I think) that will do this for you.  Just
> follow the URL for preemption in my sig. file and look
> around.

Due to kernel.org's policy that all content must be kernel-related (has
something to do with the domain name, I suspect) my scheduler tools are
not there.  You can get them at

	http://tech9.net/rml/

however :)

	Robert Love

realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Ian Collinson]

	

	We're having problems with realtime scheduling (SCHED_RR and
SCHED_FIFO), on 2.4 kernels >= 2.4.10 (built for i386, no SMP).  We have an
app that uses real-time scheduled threads. To aid debugging, in case of
realtime threads spinning and locking the system, we always keep a bash
running on a (text) console, at SCHED_RR, priority 99 (a higher priority
than any threads in our app).  We test that this is a valid approach by
running a lower priority realtime app, on another console, that sits in an
infinite busy loop.  This has always worked, and we've been able to
successfully use the high-priority bash to run gdb, and so on.  This is also
what the man page for sched_setscheduler suggests, to avoid total system
lock up.

	But, it seems to have stopped working with 2.4 kernels >= 2.4.10,
(have tried up to 2.4.19-pre9).  Once the lower priority realtime process is
running, the higher priority shell doesn't pre-empt it, and locks up.

	Details of the test:

	I have a "realtime" app, that sets the scheduling policy and
priority, then forks a specified app, and "eat_cpu", that just sits in an
infinite loop.  Both are attached.
	
	On one console (as root), I do

	> realtime -rr 99 bash
	
	Which gives me a priorty 99, SCHED_RR bash. Then, on another console
(also as root), I do:

	> sleep 10; realtime -rr 50 eat_cpu

	Then I switch back to the first console, with its priority 99 bash.
I am able to type away for 10 seconds, until the priority 50 process starts,
at which point the shell locks up.   I can get the same effect on one
console with:

	> ( sleep 10; realtime -rr 50 eat_cpu ) & realtime -rr 99 bash

	Previously, the high-priority shell would never lock up.  Now it
does.

	Any idea why this doesn't work?

	Cheers,
	Ian

Code:

/* realtime
 * To build:
 *   g++ -o realtime realtime.cxx
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sched.h>
#include <unistd.h>
#include <string>


void usage()
{
  printf("usage: realtime -get | {-rr | -fifo} <pri> cmd ...\n");
  exit(1);
}

const char *policyString(int policy)
{
  switch(policy)
  {
  case SCHED_OTHER:   return "SCHED_OTHER";
  case SCHED_FIFO:    return "SCHED_FIFO";
  case SCHED_RR:      return "SCHED_RR";
  }
  return "BAD POLICY";
}

int main(int argc, char *argv[])
{
  seteuid(getuid());

  if (argc < 2)
    usage();

  if (strcmp(argv[1], "-get") == 0)
  {
    int policy = sched_getscheduler(0);
    if (policy < 0)
    {
      perror("sched_getscheduler failed:");
      return 1;
    }

    sched_param param;
    if (sched_getparam(0, &param) < 0)
    {
      perror("sched_getparam failed");
      return 1;
    }

    printf("policy = %d = %s\n", policy, policyString(policy));

    if (policy != SCHED_OTHER)
      printf("priority=%d\n", param.sched_priority);

    int max = sched_get_priority_max(policy);
    if (max < 0) {
      perror("sched_get_priority_max");
      return 1;
    }
    int min = sched_get_priority_min(policy);
    if (min < 0) {
      perror("sched_get_priority_min");
      return 1;
    }                                                        
    printf("prorities: min = %d, max = %d\n", min, max);

    return 0;

  }

  // set

  if (argc < 4)
     usage();

  int policy;

  if (!strcmp(argv[1], "-rr"))
  {
    policy = SCHED_RR;
  }
  else if (!strcmp(argv[1], "-fifo"))
  {
    policy = SCHED_FIFO;
  }
  else {
    printf("bad policy: choose -rr (round robin) or -fifo\n");
    usage();
  }
    
  struct sched_param param;
  param.sched_priority = atoi(argv[2]);

  printf("policy = %d = %s\n", policy, policyString(policy));
  printf("priority=%d\n", param.sched_priority);

  // become root
  seteuid(0);

  if (sched_setscheduler(0, policy, &param) < 0) {
    perror("sched_setscheduler");
    return 1;
  }

  // back down
  seteuid(getuid());

  string cmd;
  for (int i=3; i<argc; i++)
    cmd = cmd + argv[i] + " ";

  printf("cmd = %s\n", cmd.c_str());

  system(cmd.c_str());

  return 0;
}

/* eof */

/* eat_cpu.c */
int main(int argc, char *argv[])
{
  for (;;)
    ;
}    
/* eof */



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Imerge Limited                          Tel :- +44 (0)1954 783600 
Unit 6 Bar Hill Business Park           Fax :- +44 (0)1954 783601 
Saxon Way                               Web :- http://www.imerge.co.uk 
Bar Hill 
Cambridge 
CB3 8SL 
United Kingdom 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Mike Kravetz]

On Thu, May 30, 2002 at 06:54:46PM +0100, Ian Collinson wrote:
> 
> 	We're having problems with realtime scheduling (SCHED_RR and
> SCHED_FIFO), on 2.4 kernels >= 2.4.10 (built for i386, no SMP).  We have an
> app that uses real-time scheduled threads. To aid debugging, in case of
> realtime threads spinning and locking the system, we always keep a bash
> running on a (text) console, at SCHED_RR, priority 99 (a higher priority
> than any threads in our app).  We test that this is a valid approach by
> running a lower priority realtime app, on another console, that sits in an
> infinite busy loop.  This has always worked, and we've been able to
> successfully use the high-priority bash to run gdb, and so on.  This is also
> what the man page for sched_setscheduler suggests, to avoid total system
> lock up.

<snip>

> 	Then I switch back to the first console, with its priority 99 bash.
> I am able to type away for 10 seconds, until the priority 50 process starts,
> at which point the shell locks up.   I can get the same effect on one
> console with:
> 
> 	> ( sleep 10; realtime -rr 50 eat_cpu ) & realtime -rr 99 bash
> 
> 	Previously, the high-priority shell would never lock up.  Now it
> does.

This works fine for me on 2.4.17 with a SERIAL console.  Could this
be related to some differences (new features) in the VGA console?
I am totally ignorant of how the consoles work.

-- 
Mike

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Andrea Arcangeli]

On Fri, May 31, 2002 at 11:28:47AM -0700, Mike Kravetz wrote:
> On Thu, May 30, 2002 at 06:54:46PM +0100, Ian Collinson wrote:
> > 
> > 	We're having problems with realtime scheduling (SCHED_RR and
> > SCHED_FIFO), on 2.4 kernels >= 2.4.10 (built for i386, no SMP).  We have an
> > app that uses real-time scheduled threads. To aid debugging, in case of
> > realtime threads spinning and locking the system, we always keep a bash
> > running on a (text) console, at SCHED_RR, priority 99 (a higher priority
> > than any threads in our app).  We test that this is a valid approach by
> > running a lower priority realtime app, on another console, that sits in an
> > infinite busy loop.  This has always worked, and we've been able to
> > successfully use the high-priority bash to run gdb, and so on.  This is also
> > what the man page for sched_setscheduler suggests, to avoid total system
> > lock up.
> 
> <snip>
> 
> > 	Then I switch back to the first console, with its priority 99 bash.
> > I am able to type away for 10 seconds, until the priority 50 process starts,
> > at which point the shell locks up.   I can get the same effect on one
> > console with:
> > 
> > 	> ( sleep 10; realtime -rr 50 eat_cpu ) & realtime -rr 99 bash
> > 
> > 	Previously, the high-priority shell would never lock up.  Now it
> > does.
> 
> This works fine for me on 2.4.17 with a SERIAL console.  Could this
> be related to some differences (new features) in the VGA console?
> I am totally ignorant of how the consoles work.

I tried it under uml on 2.4.19pre9aa2 and it worked fine there too.

Andrea

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Andi Kleen]

Mike Kravetz <kravetz@us.ibm.com> writes:

> This works fine for me on 2.4.17 with a SERIAL console.  Could this
> be related to some differences (new features) in the VGA console?
> I am totally ignorant of how the consoles work.

One possibility is that something relies on schedule_task() - keventd
doesn't run with realtime priority and can be starved.

Seems to be the case indeed: 

/usr/src/linux/drivers/char% grep schedule_task *.c
console.c:      schedule_task(&console_callback_tq);
...

the console switch does.

Fixing it would require boosting keventd's priority either globally 
or temporarily. E.g. if the original reporter could put this
(untested/uncompiled) at the beginning of kernel/context.c:context_thread():

    current->policy = SCHED_RR;
    current->rt_priority = 99;

it could fix his problem.

-Andi

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Mike Kravetz]

On Sat, Jun 01, 2002 at 07:05:47PM +0200, Andi Kleen wrote:
> Mike Kravetz <kravetz@us.ibm.com> writes:
> 
> > This works fine for me on 2.4.17 with a SERIAL console.  Could this
> > be related to some differences (new features) in the VGA console?
> > I am totally ignorant of how the consoles work.
> 
> One possibility is that something relies on schedule_task() - keventd
> doesn't run with realtime priority and can be starved.
> 
> Seems to be the case indeed: 
> 
> /usr/src/linux/drivers/char% grep schedule_task *.c
> console.c:      schedule_task(&console_callback_tq);
> ...
> 
> the console switch does.

Thanks Andi!

Part of the 'problem' is the following in the 'sched_setscheduler'
man page.

"      As  a  non-blocking  end-less  loop in a process scheduled
       under SCHED_FIFO or SCHED_RR will block all processes with
       lower priority forever, a software developer should always
       keep available on the console a shell  scheduled  under  a
       higher  static  priority than the tested application. This
       will allow an emergency kill of tested real-time  applica­
       tions  that  do  not  block  or  terminate as expected. As
       SCHED_FIFO and SCHED_RR processes can preempt  other  pro­
       cesses  forever,  only root processes are allowed to acti­
       vate these policies under Linux.
"

Seems that this tells people to leave a high priority real-
time shell running on the console.  However, if one can not
get to the console, then there is no point in leaving a high
priority shell running there.  Part of the problem may be
in the definition of 'console'.  Different console implementations
behave differently.

Is this something we should 'fix'?  I would envision a 'solution'
for each console implementation.  OR we could remove the above
from the man page. :)

Comments?
-- 
Mike

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Andrew Morton]

Mike Kravetz wrote:
> 
> ...
> Seems that this tells people to leave a high priority real-
> time shell running on the console.  However, if one can not
> get to the console, then there is no point in leaving a high
> priority shell running there.  Part of the problem may be
> in the definition of 'console'.  Different console implementations
> behave differently.
> 
> Is this something we should 'fix'?  I would envision a 'solution'
> for each console implementation.  OR we could remove the above
> from the man page. :)
> 

keventd is a "process context bottom half handler".  It's designed
for use by interrupt handlers for handing off awkward, occasional
things which need process context.  For example, device hotplugging,
which was the original reason for its introduction.

So it makes sense to give keventd SCHED_RR policy and maximum
priority.  Which should fix this problem as well, yes?

keventd is also being (ab)used for performing disk I/O.
You know who you are ;)  But even given that, I don't expect
that elevating its policy&priority will cause any problems.

-

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Robert Love]

On Mon, 2002-06-03 at 13:08, Andrew Morton wrote:

> keventd is a "process context bottom half handler".  It's designed
> for use by interrupt handlers for handing off awkward, occasional
> things which need process context.  For example, device hotplugging,
> which was the original reason for its introduction.
> 
> So it makes sense to give keventd SCHED_RR policy and maximum
> priority.  Which should fix this problem as well, yes?

Next to ditching keventd, this is probably the best thing we can do.

I wonder how much code _really_ needs it - that is, what really needs to
be running in process-context?  Obviously device hotplug probably does. 
But for things like that, what about spawning (temporarily) a kernel
thread?

	Robert Love

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Andrew Morton]

Robert Love wrote:
> 
> On Mon, 2002-06-03 at 13:08, Andrew Morton wrote:
> 
> > keventd is a "process context bottom half handler".  It's designed
> > for use by interrupt handlers for handing off awkward, occasional
> > things which need process context.  For example, device hotplugging,
> > which was the original reason for its introduction.
> >
> > So it makes sense to give keventd SCHED_RR policy and maximum
> > priority.  Which should fix this problem as well, yes?
> 
> Next to ditching keventd, this is probably the best thing we can do.

I think the design is OK.  It's for "misc stuff".  There's only
a single instance, it's only lightly used.

> I wonder how much code _really_ needs it - that is, what really needs to
> be running in process-context?

Pretty much every use of keventd make sense as-is, IMO.

>  Obviously device hotplug probably does.
> But for things like that, what about spawning (temporarily) a kernel
> thread?

We need process context for starting a thread...

It's just an 8k stack.  I believe that keventd is OK, as
long as people don't go nuts when using it.   It may make
some sense to overload ksoftirqd to provide keventd functionality.
Except ksoftirqd runs at super-low priority, which is exactly
what keventd doesn't want.

-

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Andrea Arcangeli]

On Mon, Jun 03, 2002 at 01:08:17PM -0700, Andrew Morton wrote:
> Mike Kravetz wrote:
> > 
> > ...
> > Seems that this tells people to leave a high priority real-
> > time shell running on the console.  However, if one can not
> > get to the console, then there is no point in leaving a high
> > priority shell running there.  Part of the problem may be
> > in the definition of 'console'.  Different console implementations
> > behave differently.
> > 
> > Is this something we should 'fix'?  I would envision a 'solution'
> > for each console implementation.  OR we could remove the above
> > from the man page. :)
> > 
> 
> keventd is a "process context bottom half handler".  It's designed
> for use by interrupt handlers for handing off awkward, occasional
> things which need process context.  For example, device hotplugging,
> which was the original reason for its introduction.
> 
> So it makes sense to give keventd SCHED_RR policy and maximum
> priority.  Which should fix this problem as well, yes?
> 
> keventd is also being (ab)used for performing disk I/O.
> You know who you are ;)  But even given that, I don't expect
> that elevating its policy&priority will cause any problems.

that's correct, infact that's almost required, kupdate also doesn't have
RT prio and inodes are flushed either by kupdate or keventd.

Andrea

Re: realtime scheduling problems with 2.4 linux kernel >= 2.4.10

[Andi Kleen]

> Is this something we should 'fix'?  I would envision a 'solution'
> for each console implementation.  OR we could remove the above
> from the man page. :)

There is only a single console implementation AFAIK.
I think it should be fixed, probably by the way I outlined in my previous
mail (but no feedback yet so it seems the problem isn't that important for
anybody ;)

-Andi