futex requeue_pi breakage

[Darren Hart <dvhart@linux.intel.com>]

Steven,

As we've discussed, the following changes break the futex_requeue_pi 
testcase I use to validate the futex_requeue_pi feature introduced in 
2.6.32(ish).

	commit 43fa5460fe60dea5c610490a1d263415419c60f6
	    sched: Try not to migrate higher priority RT tasks

	commit b3bc211cfe7d5fe94b310480d78e00bea96fbf2a
	    sched: Give CPU bound RT tasks preference

The source to this test is here:

http://git.kernel.org/?p=linux/kernel/git/dvhart/futextest.git;a=summary
branch: futex-lock
file: futextest/functional/futex_requeue_pi.c

The futex_requeue_pi test main loop runs as sched_other while all the 
child threads are spawned as sched_fifo 1. This scenario results in the 
parent thread spinning in futex_wait_queue_me (most likely on the hash 
bucket spinlock) while attempting a pthread_join() on the rt child 
threads. However, if I run with chrt, forcing the parent thread to fifo 
1, it completes reliably:

$ sudo chrt -f 1 ./futex_requeue_pi
futex_requeue_pi: Test requeue functionality
         Arguments: broadcast=0 locked=0 owner=0 timeout=0ns
Result:  PASS

Whereas as sched_other:
$ sudo ./futex_requeue_pi
futex_requeue_pi: Test requeue functionality
         Arguments: broadcast=0 locked=0 owner=0 timeout=0ns
<hang>

$ ps -eLo tid,rtprio,wchan:20,comm | grep futex
  1987      - futex_wait_queue_me  auditd
  1988      - futex_wait_queue_me  audispd
  5533      - futex_wait_queue_me  rsyslogd
  5547      - futex_wait_queue_me  futex_requeue_p
  5549      1 futex_wait_queue_me  futex_requeue_p
  5550      1 futex_wait_queue_me  futex_requeue_p
  5551      1 futex_wait_queue_me  futex_requeue_p
  5552      1 futex_wait_queue_me  futex_requeue_p
  5553      1 futex_wait_queue_me  futex_requeue_p
  5554      1 futex_wait_queue_me  futex_requeue_p
  5555      1 futex_wait_queue_me  futex_requeue_p
  5556      1 futex_wait_queue_me  futex_requeue_p
  5557      1 futex_wait_queue_me  futex_requeue_p
  5558      1 ?                    futex_requeue_p
  5559      1 ?                    futex_requeue_p

I'm still thinking through how your changes would impact this, and if 
there could be some assumption in the futex code that fails with those 
changes. I wanted to get the information out there in case it triggers 
any insight from you.

This suggests that somehow an rt task blocked while holding a spinlock 
(without issuing a warning to the console) or that somehow a SCHED_OTHER 
task preempted an RT task which is not getting scheduled on another CPU. 
Neither of these seem likely.

Thanks,

-- 
Darren Hart
Embedded Linux Kernel