Re: [PATCH] sched: fix race in schedule

[Hiroshi Shimamoto <h-shimamoto@ct.jp.nec.com>]

Peter Zijlstra wrote:
> On Mon, 2008-03-10 at 19:12 -0700, Hiroshi Shimamoto wrote:
>> Peter Zijlstra wrote:
>>> On Mon, 2008-03-10 at 13:01 -0700, Hiroshi Shimamoto wrote:
>>>
>>>> thanks, your patch looks nice to me.
>>>> I had focused setprio, on_rq=0 and running=1 situation, it makes me to
>>>> fix these functions.
>>>> But one point, I've just noticed. I'm not sure on same situation against
>>>> sched_rt. I think the pre_schedule() of rt has chance to drop rq lock.
>>>> Is it OK?
>>> Ah, you are quite right, that'll teach me to rush out a patch just
>>> because dinner is ready :-). 
>>>
>>> How about we submit the following patch for mainline and CC -stable to
>>> fix .23 and .24:
>>>
>> Unfortunately, I encountered similar panic with this patch on -rt.
>> I'll look into this, again. I might have missed something...
>>
>> Unable to handle kernel NULL pointer dereference at 0000000000000128 RIP:
>>  [<ffffffff802297f5>] pick_next_task_fair+0x2d/0x42
> 
> :-(
> 
> OK, so that means I'm not getting it.
> 
> So what does your patch do that mine doesn't?
> 
> It removes the dependency of running (=task_current()) from on_rq
> (p->se.on_rq).
> 
> So how can a current task not be on the runqueue?
> 
> Only sched.c:dequeue_task() and sched_fair.c:account_entity_dequeue()
> set on_rq to 0, the only one changing rq->curr is schedule().
> 
> So the only scheme I can come up with is that we did dequeue p (on_rq ==
> 0), but we didn't yet schedule so rq->curr == p.
> 
> Is this how you ended up with your previuos analysis that it must be due
> to a hole introduced by double_lock_balance()?
> 
> Because now we can seemingly call deactivate_task() and put_prev_task()
> in non-atomic fashion, but by placing the put_prev_task() before the
> load balance calls we should avoid doing that.
> 
> So what else is going on... /me puzzled

thanks for taking time.
I've tested it with debug tracer, it took several hours to half day to
reproduce it on my box. And I got the possible scenario.

Before begin, I can tell that se->on_rq is changed at enqueue_task() or
dequeue_task() in sched.c.

Here is the flow to panic which I got;
   CPU0                               CPU1
    |                              schedule()
    |                              ->deactivate_task()
    |                              -->dequeue_task()
    |                                  * on_rq=0
    |                              ->put_prev_task_fair()
    |                              ->idle_balance()
    |                              -->load_balance_newidle()
(a wakeup function)                    |
    |                              --->double_lock_balance()
    *get lock                          *rel lock
    * wake up target is CPU1's curr    |
->enqueue_task()                       |
    * on_rq=1                          |
->rt_mutex_setprio()                   |
    * on_rq=1, ruuning=1               |
-->dequeue_task()!!                    |
-->put_prev_task_fair()!!              |
    * sched_class is changed           |
-->set_curr_task_fair()!!              |
-->enqueue_task()!!                    |
    *rel lock                          *get lock again
    :                                  |
                                       :
                                   ->pick_next_task_fair()
                                       => panic

The difference from the previous scenario is;
some one enqueues CPU1,s current task before setprio,
it makes on_rq=1 and it causes set_curr_task_fair() called.
It means that the cfs_rq state of the current task is set to
before put_prev_task_fair(), I think.

Does this scenario help to solve this issue?
And I only test it on -rt because I can reproduce it on -rt only.

thanks,
Hiroshi Shimamoto