Timer tick not disabled with NO_HZ_FULL

Timer tick not disabled with NO_HZ_FULL

[Francis Giraldeau]

It seems that the timer tick occurs with a PREEMPT_RT kernel even if configured
with NO_HZ_FULL. On a real-time system where some cores are configured with
isolcpu, the tick increases the execution jitter.

# tracer: nop
#
# entries-in-buffer/entries-written: 20219/20219   #P:8
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            |/  _-----=> need-resched_lazy
#                            || / _---=> hardirq/softirq
#                            ||| / _--=> preempt-depth
#                            |||| /_--=> preempt-lazy-depth
#                            |||||  _-=> migrate-disable
#                            ||||| /    delay
#           TASK-PID   CPU#  ||||||    TIMESTAMP  FUNCTION
#              | |       |   ||||||       |         |
          <idle>-0     [002] d..h2..    41.314675:
hrtimer_expire_entry: hrtimer=ffff88013fc8f860
function=tick_sched_timer now=41136043338
          <idle>-0     [002] d..h2..    41.318675:
hrtimer_expire_entry: hrtimer=ffff88013fc8f860
function=tick_sched_timer now=41140043943
          <idle>-0     [002] d..h2..    41.322675:
hrtimer_expire_entry: hrtimer=ffff88013fc8f860
function=tick_sched_timer now=41144043138

I think the issue comes from commit 8aede461ab35 in kernel/time/timer.c:

    u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
    {
    [...]
    #ifdef CONFIG_PREEMPT_RT_FULL

        /*
        * On PREEMPT_RT we cannot sleep here. As a result we can't take
        * the base lock to check when the next timer is pending and so
        * we assume the next jiffy.
        */
        return basem + TICK_NSEC;
    #endif
    [...]

This commit is quite old, maybe a solution exists to the problem of base timer
locking. Would it be possible for instance to use RCU instead here to support
NO_HZ_FULL with a PREEMPT_RT kernel?

Thanks,

Francis

Re: Timer tick not disabled with NO_HZ_FULL

[Sebastian Andrzej Siewior]

* Francis Giraldeau | 2016-07-29 16:56:15 [-0400]:

>It seems that the timer tick occurs with a PREEMPT_RT kernel even if configured
>with NO_HZ_FULL. On a real-time system where some cores are configured with
correct.

…
>This commit is quite old, maybe a solution exists to the problem of base timer
>locking. Would it be possible for instance to use RCU instead here to support
>NO_HZ_FULL with a PREEMPT_RT kernel?

Unfortunately RCU can't fix everything in its own magical way. But there
is hope! Since v4.6.4-rt8 we have NO_HZ_FULL working thanks to the timer
wheel rework.

>Thanks,
>
>Francis

Sebastian

The timer softirq on the RT kernel

[yunhong jiang]

Hi,Mike & Steven
	On https://lkml.org/lkml/2015/3/24/1178, the patch of "timers:
do not raise softirq unconditionally" is reverted. Thanks for Steven's
detailed comments, it's quite clear explained.
	I remember Mike has a patch trying to fix it but late
abandoned. Do you still have any plan to work on this? Otherwise, I
will have a try.
	
Thanks
--jyh

Re: The timer softirq on the RT kernel

[Mike Galbraith]

On Thu, 2016-08-04 at 17:36 -0700, yunhong jiang wrote:
> Hi,Mike & Steven
> 	On https://lkml.org/lkml/2015/3/24/1178, the patch of "timers:
> do not raise softirq unconditionally" is reverted. Thanks for
> Steven's
> detailed comments, it's quite clear explained.
> 	I remember Mike has a patch trying to fix it but late
> abandoned. Do you still have any plan to work on this? Otherwise, I
> will have a try.

Sebastian converted the lock to raw in -rt, so nohz_full now works
without any extra hackery.

	-Mike

Re: The timer softirq on the RT kernel

[yunhong jiang]

On Fri, 05 Aug 2016 04:36:23 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Thu, 2016-08-04 at 17:36 -0700, yunhong jiang wrote:
> > Hi,Mike & Steven
> > 	On https://lkml.org/lkml/2015/3/24/1178, the patch of
> > "timers: do not raise softirq unconditionally" is reverted. Thanks
> > for Steven's
> > detailed comments, it's quite clear explained.
> > 	I remember Mike has a patch trying to fix it but late
> > abandoned. Do you still have any plan to work on this? Otherwise, I
> > will have a try.
> 
> Sebastian converted the lock to raw in -rt, so nohz_full now works
> without any extra hackery.

Hi, Mike, thanks for reply and the update. I still noticed timer softirq
on my isolated CPUs, possibly something wrong on my environment. I will
re-check it.

Thanks
--jyh

> 
> 	-Mike