Re: [PATCH] tick/broadcast-hrtimer : Fix suspicious RCU usage in idle loop

[Preeti U Murthy <preeti@linux.vnet.ibm.com>]

On 03/16/2015 08:26 PM, Peter Zijlstra wrote:
> On Thu, Mar 05, 2015 at 10:06:30AM +0530, Preeti U Murthy wrote:
>>
>> On 03/02/2015 08:23 PM, Peter Zijlstra wrote:
>>> On Thu, Feb 26, 2015 at 08:52:02AM +0530, Preeti U Murthy wrote:
>>>> The hrtimer mode of broadcast queues hrtimers in the idle entry
>>>> path so as to wakeup cpus in deep idle states. 
>>>
>>> Callgraph please...
>>
>> cpuidle_idle_call()
>> |____ clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, ....))
>>      |_____tick_broadcast_set_event()
>>            |____clockevents_program_event()
>>                 |____bc_set_next()
>>>
>>>> hrtimer_{start/cancel}
>>>> functions call into tracing which uses RCU. But it is not legal to call
>>>> into RCU in cpuidle because it is one of the quiescent states. Hence
>>>> protect this region with RCU_NONIDLE which informs RCU that the cpu
>>>> is momentarily non-idle.
>>>
>>> It it not clear to me that every user of bc_set_next() is from IDLE.
>>> From what I can tell it ends up being clockevents_program_event() and
>>> that is called quite a lot.
>>
>> bc_set_next() is called from at places:
>> 1. Idle entry : It is called when a cpu in its idle entry path finds the
>> need to reset the broadcast hrtimer.
>> 2. CPU offline operations : When the cpu on which the broadcast hrtimer
>> is being queued goes offline.
>>
>> So you see that almost all the time, it is called in idle entry path.
> 
> How about:
> 
> 	hrtimer_reprogram()
> 	  tick_program_event()
> 	    clockevents_program_event()
> 	      ->set_next_ktime()
> 
> That is called from !idle loads of times. I guess I'm not seeing what
> avoids &ce_broadcast_hrtimer from being the 'normal' clock event.

It is a normal clock event. In the above context, this hrtimer is being
moved from CPUx to the CPU executing that code. Hence it needs to be
enqueued onto the new CPU. Any hrtimer enqueue calls into tracing.
A hrtimer_reprogram() alone will not suffice.
Moreover hrtimer_reprogram() cannot be called directly, can it? nor is
it safe. Or am I missing your point ?

> 
> Sure; it might be that for power you only end up with that broadcast
> crap enabled on idle/hotplug, but is this always so?

The hrtimer broadcast framework gets invoked only during idle. This is
platform agnostic.

Regards
Preeti U Murthy
>