CFS patch (v6) -- dynamic RT priorities?

CFS patch (v6) -- dynamic RT priorities?

[Chris Rigg]

Hello,

I have a system with 2.6.20.7 patched with the v6 CFS patch. I am having 
issues (I believe) with fairness in regards to my real-time tasks. 
First, let me describe my setup:

* I use the kernel preemption config option.
* I am using an Intel Xeon with Hyperthreading enabled (and using the 
SMP config option)
* I have 2 kernel modules -- one that is a device driver (receives 
interrupts from a device) but also spawns a thread for processing 
messages from kernel module 2. Kernel module 2 controls the hardware and 
receives messages from the driver via a message ring. It also puts 
messages into a separate ring to control the hardware through the 
hardware. I have 4 different threads spawned when I insmod this module. 
The 4 threads (call them control threads) all do the same thing (i.e. 
there is a thread associated with each port on the device). The threads 
will get messages from the driver on behalf of their port and send 
control messages back to their port through the driver's message ring, 
etc. This may sound complicated but really all I'm doing is abstracting 
out the details of controlling the hardware from the control threads and 
provide a simple interface (the message rings) for controlling the 
hardware and receiving events from it.
* I have given the 4 control threads and the 1 device driver "receive 
message" thread MAX_RT_PRIO - 1 priority (with a call to 
sched_setscheduler(...).
* As I stated earlier, I use a set of kfifo message rings to communicate 
between a control thread and the driver. When the control thread puts a 
message into the ring for the driver thread, he calls "up()" on a 
semaphore that the driver exported. When the driver puts a message into 
the ring for the control thread, he too calls "up()" on a different 
semaphore exported by the control thread.
* I expect all 5 threads to be the highest priority tasks in the system 
and ALWAYS be scheduled if they are ready (provided that another one of 
my 5 RT prio threads isn't already running or is further ahead in the 
queue).

Ok, so far, I have described my basic setup. Hopefully I'm not doing 
anything out of the ordinary here.

Now, for the problem....

When I have 3 of the 4 control threads running at a very high frequency 
of interrupts (and hence a high volume/high frequency of "up()"s on the 
sync semaphore when the driver puts a message into the control thread's 
ring), it appears that the priority of the 3 are somehow getting 
"elevated" by the scheduler? (Even though I set them to all have equal 
priority of MAX_RT_PRIO.) Is there a dynamic priority factoring in that 
I'm not aware of? Is the scheduler looking at the frequency of 'up()'s 
for that task and sees that they have been scheduled a lot and somehow 
gives them a higher priority over my 4th control thread that isn't 
getting as high a frequency of 'up()'s or something?? My notion of a 
real-time, fair, scheduler would be to have a queue for highest priority 
threads and when a thread is ready to run, it is put into the queue and 
serviced when its at the front of the queue. What it seems like I'm 
seeing is the 3 control threads that get a high frequency of 'ups()' on 
their semaphore get moved to the FRONT of the ready queue? The bottom 
line for me is that I end up missing protocol deadlines for my 4th 
control thread even though the driver is servicing interrupts and 
subsequently putting 'event' message into the ring for the control 
thread in plenty of time. But for some reason, the scheduler doesn't get 
around to scheduling it until its scheduled the other 3 MANY, MANY times.

Does this seem right? Is there away I can disable dynamic priority 
calculation (if that is what's going on here) for my RT threads so they 
are all treated equally at ALL times?

Or is there a "problem" with using semaphores for signaling between 
kernel threads like I'm doing? This technique is preferred in VXWorks 
but maybe I shouldn't be using it in Linux?

Anyway, any help you could provide would be MUCH appreciated.

Thanks in advance
Chris

Re: CFS patch (v6) -- dynamic RT priorities?

[Willy Tarreau]

On Mon, Sep 17, 2007 at 02:33:28PM -0600, Chris Rigg wrote:
> Hello,
> 
> I have a system with 2.6.20.7 patched with the v6 CFS patch. I am having 
> issues (I believe) with fairness in regards to my real-time tasks. 
> First, let me describe my setup:

Chris,

CFSv6 is *very* old. It was not that bad, but looking back, it has
improved a lot since. You should upgrade to v20.x first, and it's
likely that most of your problems will vanish.

Regards,
willy

Re: CFS patch (v6) -- dynamic RT priorities?

[Ingo Molnar]

* Willy Tarreau <w@1wt.eu> wrote:

> On Mon, Sep 17, 2007 at 02:33:28PM -0600, Chris Rigg wrote:
> > Hello,
> > 
> > I have a system with 2.6.20.7 patched with the v6 CFS patch. I am having 
> > issues (I believe) with fairness in regards to my real-time tasks. 
> > First, let me describe my setup:
> 
> Chris,
> 
> CFSv6 is *very* old. It was not that bad, but looking back, it has 
> improved a lot since. You should upgrade to v20.x first, and it's 
> likely that most of your problems will vanish.

yeah. I remember we fixed something in the RT task balancing area too, 
so chances are that this bug got fixed. The latest backported CFS 
version can be picked up from:

    http://people.redhat.com/mingo/cfs-scheduler/

	Ingo

MAX_RT_PRIO - 1 Highest prio?

[Chris Rigg]

Hello,

First, I'm assuming that if I want my task to have the HIGHEST priority 
in the system (i.e. preempt any other task whenever it is put into the 
ready queue (assuming I have preemption turned on/configured)), I use 
sched_setscheduler (...) and use the sched_priority in sched_param for 
MAX_RT_PRIO -1. Is this correct?

Second, assuming that MAX_RT_PRIO-1 is the highest, would it be bad on 
an SMP/Hyperthreading system (that's using the migration thread 
balancing in 2.6.20.7) to set a task's priority to MAX_RT_PRIO -1 given 
the fact that the migration threads are already set to MAX_RT_PRIO -1? 
Should I be setting my task's prio to MAX_RT_PRIO-2 to not interfere 
with the load balancing?

Thanks,
Chris

Re: MAX_RT_PRIO - 1 Highest prio?

[Steven Rostedt]

On Wed, Sep 19, 2007 at 05:20:27PM -0600, Chris Rigg wrote:
> Hello,

Hi Chris,

>
> First, I'm assuming that if I want my task to have the HIGHEST priority in 
> the system (i.e. preempt any other task whenever it is put into the ready 
> queue (assuming I have preemption turned on/configured)), I use 
> sched_setscheduler (...) and use the sched_priority in sched_param for 
> MAX_RT_PRIO -1. Is this correct?

Actually it's probably best to use sched_get_priority_max for the max
prio.

>
> Second, assuming that MAX_RT_PRIO-1 is the highest, would it be bad on an 
> SMP/Hyperthreading system (that's using the migration thread balancing in 
> 2.6.20.7) to set a task's priority to MAX_RT_PRIO -1 given the fact that 
> the migration threads are already set to MAX_RT_PRIO -1? Should I be 
> setting my task's prio to MAX_RT_PRIO-2 to not interfere with the load 
> balancing?

The migrate task of a given CPU isn't the only one that will take
tasks off its CPU to push them to others (although it does do that).
But there's other load balancing work going on in the scheduler
(looking at the 2.6.20 sched.c).

Although it would be interesting to see what the result would be if you
had N+1 tasks running on N CPUs all doing busy loops, and make one of
the tasks with the prio of MAX_RT_PRIO-1, and see if we have one task
that is starved and never schedules. But I'm sure this should be fixed
(if it was ever broken) with the latest scheduling work that's being
done in the most recent kernels.

-- Steve