[PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Steven Rostedt]

According to the comments in include/linux/sched.h

/*
* Priority of a process goes from 0..MAX_PRIO-1, valid RT
* priority is 0..MAX_RT_PRIO-1, and SCHED_NORMAL tasks are
* in the range MAX_RT_PRIO..MAX_PRIO-1. Priority values
* are inverted: lower p->prio value means higher priority.
*
* The MAX_USER_RT_PRIO value allows the actual maximum
* RT priority to be separate from the value exported to
* user-space.  This allows kernel threads to set their
* priority to a value higher than any user task. Note:
* MAX_RT_PRIO must not be smaller than MAX_USER_RT_PRIO.
*/

This makes it look like the priority goes as follows:

prio: 0 .. MAX_RT_PRIO .. MAX_USER_RT_PRIO .. MAX_PRIO

where 0 is of highest priority

but in reality we have:

prio: 0 .. MAX_USER_RT_PRIO .. MAX_RT_PRIO .. MAX_PRIO

The comments say that MAX_RT_PRIO must not be smaller than
MAX_USER_RT_PRIO, but if it is bigger (thinking bigger means greater
than) then the system will crash on a SMP machine.

Here's how it works.  The migration_thread sets the priority of its
thread to MAX_RT_PRIO-1 via:

__setscheduler(p, SCHED_FIFO, MAX_RT_PRIO-1);

Now looking at __setscheduler

static void __setscheduler(struct task_struct *p, int policy, int prio)
{
        BUG_ON(p->array);
        p->policy = policy;
        p->rt_priority = prio;
        if (policy != SCHED_NORMAL)
                p->prio = MAX_USER_RT_PRIO-1 - p->rt_priority;
        else
                p->prio = p->static_prio;
}

If we have MAX_USER_RT_PRIO = 99 and MAX_RT_PRIO = 100 then we would get

  p->prio = 99-1 - 100-1 = -1;

This would be very bad when it comes time to schedule.  Not to mention
that kstop_machine uses MAX_RT_PRIO and then calls
sys_sched_setscheduler, which would fail if MAX_RT_PRIO >
MAX_USER_RT_PRIO. Below is a patch that makes MAX_RT_PRIO work if it is
greater than MAX_USER_RT_PRIO on a SMP machine.  The p->mm is to allow
kstop_machine to work and any other kernel threads.

I tested the patch on an SMP machine where MAX_RT_PRIO = 100 and
MAX_USER_RT_PRIO = 99. Without the patch, the system crashes with a
reboot.

Funny, back in July 2002, this was noticed by an Anton Wilson and he was
just lost in the noise!
http://seclists.org/lists/linux-kernel/2002/Jul/1695.html


-- Steve

diff -u linux-2.6.12-rc5.orig/kernel/sched.c linux-2.6.12-rc5/kernel/sched.c
--- linux-2.6.12-rc5.orig/kernel/sched.c	2005-06-06 22:37:15.000000000 -0400
+++ linux-2.6.12-rc5/kernel/sched.c	2005-06-06 21:58:39.000000000 -0400
@@ -3347,7 +3347,7 @@
 	p->policy = policy;
 	p->rt_priority = prio;
 	if (policy != SCHED_NORMAL)
-		p->prio = MAX_USER_RT_PRIO-1 - p->rt_priority;
+		p->prio = MAX_RT_PRIO-1 - p->rt_priority;
 	else
 		p->prio = p->static_prio;
 }
@@ -3379,7 +3379,8 @@
 	 * 1..MAX_USER_RT_PRIO-1, valid priority for SCHED_NORMAL is 0.
 	 */
 	if (param->sched_priority < 0 ||
-	    param->sched_priority > MAX_USER_RT_PRIO-1)
+	    (p->mm &&  param->sched_priority > MAX_USER_RT_PRIO-1) ||
+	    (!p->mm && param->sched_priority > MAX_RT_PRIO-1))
 		return -EINVAL;
 	if ((policy == SCHED_NORMAL) != (param->sched_priority == 0))
 		return -EINVAL;

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Steven Rostedt]

On Mon, 2005-06-06 at 22:46 -0400, Steven Rostedt wrote:

> This makes it look like the priority goes as follows:
> 
> prio: 0 .. MAX_RT_PRIO .. MAX_USER_RT_PRIO .. MAX_PRIO
> 
> where 0 is of highest priority

I'm correcting my own post :-)

What we really want is:

prio: 0 .. [MAX_RT_PRIO - MAX_USER_RT_PRIO] .. MAX_RT_PRIO .. MAX_PRIO

                                                    |---- nice -------|
                         |------ user RT prio ------|
      |------------ kernel RT prio -----------------|

Remember, 0 is of highest priority.


-- Steve

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> I tested the patch on an SMP machine where MAX_RT_PRIO = 100 and 
> MAX_USER_RT_PRIO = 99. Without the patch, the system crashes with a 
> reboot.

Acked-by: Ingo Molnar <mingo@elte.hu>

	Ingo

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Steven Rostedt]

On Tue, 2005-06-07 at 07:33 +0200, Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > I tested the patch on an SMP machine where MAX_RT_PRIO = 100 and 
> > MAX_USER_RT_PRIO = 99. Without the patch, the system crashes with a 
> > reboot.
> 
> Acked-by: Ingo Molnar <mingo@elte.hu>

If this patch does go in, then xpc_activating in
arch/ia64/sn/kernel/xpc_main.c (from rc6) also needs to use MAX_RT_PRIO
instead of MAX_USER_RT_PRIO. Unless it is OK that it runs lower in
priority than other kernel threads.

-- Steve

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Dean Nelson]

On Tue, Jun 07, 2005 at 07:25:04AM -0400, Steven Rostedt wrote:
> On Tue, 2005-06-07 at 07:33 +0200, Ingo Molnar wrote:
> > * Steven Rostedt <rostedt@goodmis.org> wrote:
> > 
> > > I tested the patch on an SMP machine where MAX_RT_PRIO = 100 and 
> > > MAX_USER_RT_PRIO = 99. Without the patch, the system crashes with a 
> > > reboot.
> > 
> > Acked-by: Ingo Molnar <mingo@elte.hu>
> 
> If this patch does go in, then xpc_activating in
> arch/ia64/sn/kernel/xpc_main.c (from rc6) also needs to use MAX_RT_PRIO
> instead of MAX_USER_RT_PRIO. Unless it is OK that it runs lower in
> priority than other kernel threads.

You are correct xpc_activating() needs to be changed to use MAX_RT_PRIO.
So please do add that change to your patch.

Thanks,
Dean

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Steven Rostedt]

On Tue, 2005-06-07 at 10:48 -0500, Dean Nelson wrote:
> You are correct xpc_activating() needs to be changed to use MAX_RT_PRIO.
> So please do add that change to your patch.

I haven't tested this patch, I just used the previous patch (which I did
test) and added your change.

-- Steve

--- linux-2.6.12-rc6/kernel/sched.c.orig	2005-06-07 13:22:33.000000000 -0400
+++ linux-2.6.12-rc6/kernel/sched.c	2005-06-07 13:22:37.000000000 -0400
@@ -3347,7 +3347,7 @@
 	p->policy = policy;
 	p->rt_priority = prio;
 	if (policy != SCHED_NORMAL)
-		p->prio = MAX_USER_RT_PRIO-1 - p->rt_priority;
+		p->prio = MAX_RT_PRIO-1 - p->rt_priority;
 	else
 		p->prio = p->static_prio;
 }
@@ -3379,7 +3379,8 @@
 	 * 1..MAX_USER_RT_PRIO-1, valid priority for SCHED_NORMAL is 0.
 	 */
 	if (param->sched_priority < 0 ||
-	    param->sched_priority > MAX_USER_RT_PRIO-1)
+	    (p->mm &&  param->sched_priority > MAX_USER_RT_PRIO-1) ||
+	    (!p->mm && param->sched_priority > MAX_RT_PRIO-1))
 		return -EINVAL;
 	if ((policy == SCHED_NORMAL) != (param->sched_priority == 0))
 		return -EINVAL;
--- linux-2.6.12-rc6/arch/ia64/sn/kernel/xpc_main.c.orig	2005-06-07 13:23:26.000000000 -0400
+++ linux-2.6.12-rc6/arch/ia64/sn/kernel/xpc_main.c	2005-06-07 13:23:43.000000000 -0400
@@ -420,7 +420,7 @@
 	partid_t partid = (u64) __partid;
 	struct xpc_partition *part = &xpc_partitions[partid];
 	unsigned long irq_flags;
-	struct sched_param param = { sched_priority: MAX_USER_RT_PRIO - 1 };
+	struct sched_param param = { sched_priority: MAX_RT_PRIO - 1 };
 	int ret;
 
 

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Dean Nelson]

On Tue, Jun 07, 2005 at 01:31:59PM -0400, Steven Rostedt wrote:
> On Tue, 2005-06-07 at 10:48 -0500, Dean Nelson wrote:
> > You are correct xpc_activating() needs to be changed to use MAX_RT_PRIO.
> > So please do add that change to your patch.
> 
> I haven't tested this patch, I just used the previous patch (which I did
> test) and added your change.

I just built and tested a kernel and xp/xpc/xpnet modules with your patch
applied. It ran fine. The priorities of the xpc kthreads were correct.

Looks good to me.

Thanks,
Dean

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Steven Rostedt]

On Tue, 2005-06-07 at 14:10 -0500, Dean Nelson wrote:

> I just built and tested a kernel and xp/xpc/xpnet modules with your patch
> applied. It ran fine. The priorities of the xpc kthreads were correct.
> 
> Looks good to me.

Dean, 

If you can do me a favor, the way you really want to test this is by
changing MAX_USER_RT_PRIO to 99 and MAX_RT_PRIO to 
(MAX_USER_RT_PRIO+1).  This will make sure that the patch is working.
Your kernel thread should still run at priority 99. 

Check it with:  ps -eo pid,rtprio,comm

And grep for your thread name.

Thanks,

-- Steve

Re: [PATCH] MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!

[Dean Nelson]

On Tue, Jun 07, 2005 at 03:23:02PM -0400, Steven Rostedt wrote:
> On Tue, 2005-06-07 at 14:10 -0500, Dean Nelson wrote:
> 
> > I just built and tested a kernel and xp/xpc/xpnet modules with your patch
> > applied. It ran fine. The priorities of the xpc kthreads were correct.
> > 
> > Looks good to me.
> 
> Dean, 
> 
> If you can do me a favor, the way you really want to test this is by
> changing MAX_USER_RT_PRIO to 99 and MAX_RT_PRIO to 
> (MAX_USER_RT_PRIO+1).  This will make sure that the patch is working.
> Your kernel thread should still run at priority 99. 
> 
> Check it with:  ps -eo pid,rtprio,comm
> 
> And grep for your thread name.

Just did as you asked and things seem fine. Ran on an SGI altix.
(The first process shown below shouldn't have a priority of 99,
just the others.)

cranberry5:~ # ps -eo pid,rtprio,comm | grep xpc
13325      - xpc_hb
13327     99 xpc08
13467     99 xpc06
13469     99 xpc06c1
13501     99 xpc06c1
13502     99 xpc06c1
cranberry5:~ #