PREEMPT_RT with Full Tickless

PREEMPT_RT with Full Tickless

[GeHao Kang]

Hi Thomas, Peter, and Jason,

When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu never stops.

Whenever the nohz_full cpu wants to stop its tick, it would use
get_next_timer_interrupt() to get the time of next timer and check
whether this time  is more than one jiffy after current time.
However, get_next_timer_interrupt() always returns now+1 in PREEMPT_RT
and this cpu has no chance of stopping its tick.  Please reference
kernel/time/timer.c: get_next_timer_interrupt().

#ifdef CONFIG_PREEMPT_RT_FULL
        /*
         * On PREEMPT_RT we cannot sleep here. As a result we can't take
         * the base lock to check when the next timer is pending and so
         * we assume the next jiffy.
         */
        return now + 1;
#endif

Does this mean it is a restriction on the current development of PREEMPT_RT?

Thanks,
- Kang

Re: PREEMPT_RT with Full Tickless

[Mike Galbraith]

On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
> Hi Thomas, Peter, and Jason,
> 
> When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
> in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu
> never stops.

Because the workarounds that let it work have been dropped.  I recall
Thomas mentioning that, but can't find the message in my rubble pile.

	-Mike 

Re: PREEMPT_RT with Full Tickless

[Sebastian Andrzej Siewior]

* Mike Galbraith | 2015-12-11 15:36:17 [+0100]:

>On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
>> When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
>> in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu
>> never stops.
>
>Because the workarounds that let it work have been dropped.  I recall
>Thomas mentioning that, but can't find the message in my rubble pile.

Right. This was the case on the first v4.0 or v4.1 release. The longterm
plan is to bring it back once the timers have been re-done so we don't
need any hacks.

>	-Mike 

Sebastian

Re: PREEMPT_RT with Full Tickless

[Gary Robertson]

On Fri, Dec 11, 2015 at 11:50 AM, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> * Mike Galbraith | 2015-12-11 15:36:17 [+0100]:
>
>>On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
>>> When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
>>> in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu
>>> never stops.
>>


Also - since in RT the tick timer interrupt results in a schedulable
thread running to handle the software interrupt work stimulated by the
tick - upon exit from the interrupt handler there are always at least
2 runnable threads in the CPU running the tick if there was a runnable
task interrupted by the tick event.  Therefore the scheduler is still
needed on that CPU, the CPU is ineligible to enter full tickless mode,
and another tick timer event is scheduled.  Below is an ftrace snippet
from an isolation test of an Linaro Networking Group kernel with the
RT patch applied.  It is a v4.1.10 kernel running on a dual-core ARM
Cortex A15 armv7a platform and compiled with
CONFIG_NO_HZ_FULL_ALL=y... so CPU 1 is designated for full tickless
operation when possible.  The cpuhog process is affined to the
isolated CPU and runs a tight loop with no system calls so it is
always runnable.  The snippet below shows two tick cycles but this
condition persists indefinitely.

# tracer: function
#
# entries-in-buffer/entries-written: 34219/132331   #P:2
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            |/  _-----=> need-resched_lazy
#                            || / _---=> hardirq/softirq
#                            ||| / _--=> preempt-depth
#                            |||| /_--=> preempt-lazy-depth
#                            |||||  _-=> migrate-disable
#                            ||||| /    delay
#           TASK-PID   CPU#  ||||||    TIMESTAMP  FUNCTION
#              | |       |   ||||||       |         |
          cpuhog-1800  [001] dn..1..  1103.211867: tick_stop:
success=no msg=more than 1 task in runqueue

          cpuhog-1800  [001] dn..3..  1103.211870: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=12083 [ns] vruntime=90040150190 [ns]
          cpuhog-1800  [001] d...3..  1103.211872: sched_switch:
prev_comm=cpuhog prev_pid=1800 prev_prio=120 prev_state=R ==>
next_comm=ksoftirqd/1 next_pid=18 next_prio=98
     ksoftirqd/1-18    [001] ....111  1103.211880: softirq_entry:
vec=1 [action=TIMER]
     ksoftirqd/1-18    [001] ....111  1103.211883: softirq_exit: vec=1
[action=TIMER]
     ksoftirqd/1-18    [001] d...3..  1103.211889: sched_stat_wait:
comm=cpuhog pid=1800 delay=37834 [ns]
     ksoftirqd/1-18    [001] d...3..  1103.211891: sched_switch:
prev_comm=ksoftirqd/1 prev_pid=18 prev_prio=98 prev_state=S ==>
next_comm=cpuhog next_pid=1800 next_prio=120
          cpuhog-1800  [001] d..h1..  1103.216830: irq_handler_entry:
irq=49 name=arch_timer
          cpuhog-1800  [001] d..h2..  1103.216832: hrtimer_cancel:
hrtimer=ee7a5c98
          cpuhog-1800  [001] d..h1..  1103.216834:
hrtimer_expire_entry: hrtimer=ee7a5c98 function=tick_sched_timer
now=1103195005858
          cpuhog-1800  [001] d..h2..  1103.216836: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=4949166 [ns] vruntime=90045099356 [ns]
          cpuhog-1800  [001] d..h1..  1103.216839: softirq_raise:
vec=1 [action=TIMER]
          cpuhog-1800  [001] d..h1..  1103.216841:
hrtimer_expire_exit: hrtimer=ee7a5c98
          cpuhog-1800  [001] d..h2..  1103.216843: hrtimer_start:
hrtimer=ee7a5c98 function=tick_sched_timer expires=1103200000000
softexpires=1103200000000
          cpuhog-1800  [001] d..h1..  1103.216845: irq_handler_exit:
irq=49 ret=handled
          cpuhog-1800  [001] d...3..  1103.216849:
tick_nohz_full_kick_cpu <-enqueue_top_rt_rq
          cpuhog-1800  [001] dn..3..  1103.216853: sched_wakeup:
comm=ksoftirqd/1 pid=18 prio=98 success=1 target_cpu=001
          cpuhog-1800  [001] dn..1..  1103.216855: tick_nohz_irq_exit <-irq_exit
          cpuhog-1800  [001] dn..1..  1103.216856: tick_stop:
success=no msg=more than 1 task in runqueue

          cpuhog-1800  [001] dn..1..  1103.216862: tick_nohz_irq_exit <-irq_exit
          cpuhog-1800  [001] dn..1..  1103.216863: tick_stop:
success=no msg=more than 1 task in runqueue

          cpuhog-1800  [001] dn..3..  1103.216866: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=12209 [ns] vruntime=90045111565 [ns]
          cpuhog-1800  [001] d...3..  1103.216868: sched_switch:
prev_comm=cpuhog prev_pid=1800 prev_prio=120 prev_state=R ==>
next_comm=ksoftirqd/1 next_pid=18 next_prio=98
     ksoftirqd/1-18    [001] ....111  1103.216876: softirq_entry:
vec=1 [action=TIMER]
     ksoftirqd/1-18    [001] ....111  1103.216879: softirq_exit: vec=1
[action=TIMER]
     ksoftirqd/1-18    [001] d...3..  1103.216885: sched_stat_wait:
comm=cpuhog pid=1800 delay=34791 [ns]
     ksoftirqd/1-18    [001] d...3..  1103.216887: sched_switch:
prev_comm=ksoftirqd/1 prev_pid=18 prev_prio=98 prev_state=S ==>
next_comm=cpuhog next_pid=1800 next_prio=120
          cpuhog-1800  [001] d..h1..  1103.221830: irq_handler_entry:
irq=49 name=arch_timer
          cpuhog-1800  [001] d..h2..  1103.221832: hrtimer_cancel:
hrtimer=ee7a5c98
          cpuhog-1800  [001] d..h1..  1103.221834:
hrtimer_expire_entry: hrtimer=ee7a5c98 function=tick_sched_timer
now=1103200005650
          cpuhog-1800  [001] d..h2..  1103.221836: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=4952959 [ns] vruntime=90050064524 [ns]
          cpuhog-1800  [001] d..h1..  1103.221839: softirq_raise:
vec=1 [action=TIMER]
          cpuhog-1800  [001] d..h1..  1103.221841:
hrtimer_expire_exit: hrtimer=ee7a5c98
          cpuhog-1800  [001] d..h2..  1103.221843: hrtimer_start:
hrtimer=ee7a5c98 function=tick_sched_timer expires=1103205000000
softexpires=1103205000000
          cpuhog-1800  [001] d..h1..  1103.221845: irq_handler_exit:
irq=49 ret=handled
          cpuhog-1800  [001] d...3..  1103.221849:
tick_nohz_full_kick_cpu <-enqueue_top_rt_rq
          cpuhog-1800  [001] dn..3..  1103.221852: sched_wakeup:
comm=ksoftirqd/1 pid=18 prio=98 success=1 target_cpu=001
          cpuhog-1800  [001] dn..1..  1103.221855: tick_nohz_irq_exit <-irq_exit
          cpuhog-1800  [001] dn..1..  1103.221856: tick_stop:
success=no msg=more than 1 task in runqueue

          cpuhog-1800  [001] dn..1..  1103.221862: tick_nohz_irq_exit <-irq_exit
          cpuhog-1800  [001] dn..1..  1103.221863: tick_stop:
success=no msg=more than 1 task in runqueue

Re: PREEMPT_RT with Full Tickless

[Mike Galbraith]

On Sat, 2015-12-12 at 14:11 -0600, Gary Robertson wrote:
> On Fri, Dec 11, 2015 at 11:50 AM, Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
> > * Mike Galbraith | 2015-12-11 15:36:17 [+0100]:
> > 
> > > On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
> > > > When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
> > > > in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full
> > > > cpu
> > > > never stops.
> > > 
> 
> 
> Also - since in RT the tick timer interrupt results in a schedulable
> thread running to handle the software interrupt work stimulated by
> the
> tick - upon exit from the interrupt handler there are always at least
> 2 runnable threads in the CPU running the tick if there was a
> runnable
> task interrupted by the tick event.

Yup, handling that was one of the hacks that was dropped.  If you look
at what all was dropped in one 'let's con NOHZ_FULL into working'
patch, you'd think twice about submitting it for review.

	-Mike