I have been experimenting with realtime scheduling and have come across some 
unexpected results.  When running the attached testcase it's apparent that 
lower priority SCHED_FIFO threads are running while higher priority 
SCHED_FIFO threads sit runnable on another CPU's run_queue.

First, what are the expectations for the realtime scheduler?  Are we trying 
for "system wide strict realtime priority scheduling", should we consider the 
above scenario a bug?

From what I could gather reading sched.c (2.6.16-rt11), the scheduler will 
increment rt_overload anytime a run_queue has 2 RT tasks on it - so 
rt_overload is >0 anytime NR_CPUS+1 RT tasks are runnable on the system.  
schedule() calls pull_rt_tasks() anytime it runs and sees that rt_overload is 
> 0 and pulls the highest priority runnable rt task off each run_queue and 
places them on its CPU's run_queue.  I also see that we call 
smp_send_reschedule() (or similar) when an RT task is waking up or being 
bumped off the run_queue.  So clearly we are trying to prevent the above 
scenario from happening.

I'd appreciate your thoughts on this.  Any comments on the test case are also 
welcome.  As I understand it, sched_yield() will place a SCHED_FIFO task back 
on the active array, and if it is still the highest prio RT task it will get 
rescheduled immediately (which it likely is since a higher priority task 
should have preempted it when it became runnable).

(the testcase "fails" more often on 4way and bigger machines, with around 
2xCPUS threads per team. './sched_football 8 10' for example.)

Thanks,

--Darren