PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

[Ramesh Thomas]

Normally CONFIG_NO_HZ_FULL (full dynticks) feature blocks timer interrupts 
when a single task is running in an isolated CPU.

This behavior changes when CONFIG_PREEMPT_RT_FULL is also enabled. I notice
timer ticks are blocked the very first time after boot and a single task is
running. However, when the task is stopped and run for the second time, the
timer ticks are not blocked anymore.

Disabling CONFIG_PREEMPT_RT_FULL brings back the expected behavior of
CONFIG_NO_HZ_FULL even if the task is stopped and rerun many times, as long 
as it is the single task running.

Following are the system and kernel configurations used in my tests:

Kernel version: 4.18.0-rc8, branch linux-4.18.y-rt
Set CONFIG_NO_HZ_FULL=y

CPU isolaion is done using following kernel parameters:
isolcpus=nohz,domain,1-3 nohz_full=1-3 rcu_nocbs=1-3 irqaffinity=0

Workload is run in CPU 3, with SCHED_FIFO priority 99 as follows:
taskset -c 3 chrt -f 99 ./jitter

Give RT process 100% time for testing purpose:
echo -1 > /proc/sys/kernel/sched_rt_runtime_us

Following are the list of processes in CPU 3 in the 3 cases:

Case #1 with CONFIG_PREEMPT_RT_FULL=n
S [cpuhp/3]
S [migration/3]
S [rcuc/3]
S [ksoftirqd/3]
I [kworker/3:0-mm_]
I [kworker/3:0H]
R [kworker/3:1-mm_]
R ./jitter

Case #2 with CONFIG_PREEMPT_RT_FULL=y (First run after boot)
S [cpuhp/3]
S [migration/3]
S [posixcputmr/3]
S [rcuc/3]
S [ktimersoftd/3]
S [ksoftirqd/3]
I [kworker/3:0-mm_]
I [kworker/3:0H]
R [irq/125-nvme0q4]
R [kworker/3:1-mm_]
R ./jitter

Case #3 with CONFIG_PREEMPT_RT_FULL=y (Second run after boot)
S [cpuhp/3]
S [migration/3]
S [posixcputmr/3]
S [rcuc/3]
R [ktimersoftd/3]
S [ksoftirqd/3]
I [kworker/3:0-mm_]
I [kworker/3:0H]
S [irq/125-nvme0q4]
R [kworker/3:1-mm_]
R ./jitter

In Case #3, /proc/interupts show timer interrupts occuring on CPU 3 while it
is stopped in the other cases. ktimersoftd is in runnable state in Case #3

Case #1 with PREEMPT_RT_FULL disabled has fewer processes and looks clean.

We need both PREEMPT_RT_FULL and NO_HZ_FULL for use cases where a single RT
task in a core can run with bare metal performance without timer interrupts.

Is this a known issue and is it being looked at by anyone?

If it is an issue, I would be glad to help in any way to get these 2 very
important features compatible with each other.

Thanks,
Ramesh

Re: PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

[Sebastian Andrzej Siewior]

On 2018-08-26 20:39:22 [-0700], Ramesh Thomas wrote:
> Case #2 with CONFIG_PREEMPT_RT_FULL=y (First run after boot)
> S [cpuhp/3]
> S [migration/3]
> S [posixcputmr/3]
> S [rcuc/3]
> S [ktimersoftd/3]
> S [ksoftirqd/3]
> I [kworker/3:0-mm_]
> I [kworker/3:0H]
> R [irq/125-nvme0q4]
> R [kworker/3:1-mm_]
> R ./jitter

irq/125 shouldn't be there, right?

> Case #3 with CONFIG_PREEMPT_RT_FULL=y (Second run after boot)
> S [cpuhp/3]
> S [migration/3]
> S [posixcputmr/3]
> S [rcuc/3]
> R [ktimersoftd/3]
> S [ksoftirqd/3]
> I [kworker/3:0-mm_]
> I [kworker/3:0H]
> S [irq/125-nvme0q4]
> R [kworker/3:1-mm_]
> R ./jitter
> 
> In Case #3, /proc/interupts show timer interrupts occuring on CPU 3 while it
> is stopped in the other cases. ktimersoftd is in runnable state in Case #3

can you trace down who or what is arming the timer on CPU3?

> Is this a known issue and is it being looked at by anyone?

now that I know of. Do you happen to know if this is a regression
compared to v4.14-RT?

> If it is an issue, I would be glad to help in any way to get these 2 very
> important features compatible with each other.

So if the ktimersoftd runs and you see the interrupt counter
incrementing for CPU3 then it would be interesting to figure out why
there is an armed timer on the second invocation (and none on the first
one).

> Thanks,
> Ramesh

Sebastian

Re: PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

[Ramesh Thomas]

On 2018-08-30 at 16:18:56 +0200, Sebastian Andrzej Siewior wrote:
> On 2018-08-26 20:39:22 [-0700], Ramesh Thomas wrote:
> > Case #2 with CONFIG_PREEMPT_RT_FULL=y (First run after boot)
> > S [cpuhp/3]
> > S [migration/3]
> > S [posixcputmr/3]
> > S [rcuc/3]
> > S [ktimersoftd/3]
> > S [ksoftirqd/3]
> > I [kworker/3:0-mm_]
> > I [kworker/3:0H]
> > R [irq/125-nvme0q4]
> > R [kworker/3:1-mm_]
> > R ./jitter
> 
> irq/125 shouldn't be there, right?
> 
Yes. Also posixcputmr and ktimersoftd are not seen if PREEMPT_RT_FULL is not 
enabled. They don't seem to be running when the timer interrupts occur. But 
they being there by itself indicates something different is happening.
		     Sched    RT_Prio   Cpu_Time
S [posixcputmr/3]      FF      99 	00:00:00
R [ktimersoftd/3]      FF       1 	00:00:00
S [irq/125-nvme0q4]    FF      50 	00:00:00
R ./jitter             FF      99 	00:01:53

> > Case #3 with CONFIG_PREEMPT_RT_FULL=y (Second run after boot)
> > S [cpuhp/3]
> > S [migration/3]
> > S [posixcputmr/3]
> > S [rcuc/3]
> > R [ktimersoftd/3]
> > S [ksoftirqd/3]
> > I [kworker/3:0-mm_]
> > I [kworker/3:0H]
> > S [irq/125-nvme0q4]
> > R [kworker/3:1-mm_]
> > R ./jitter
> > 
> > In Case #3, /proc/interupts show timer interrupts occuring on CPU 3 while it
> > is stopped in the other cases. ktimersoftd is in runnable state in Case #3
> 
> can you trace down who or what is arming the timer on CPU3?
> 

Ok, I will take a look.

> > Is this a known issue and is it being looked at by anyone?
> 
> now that I know of. Do you happen to know if this is a regression
> compared to v4.14-RT?
> 

I see the issue in 4.14.63 RT as well. There are slight differences in 
behavior due to changes that went in 4.17, but the main issue is seen there 
also.

> > If it is an issue, I would be glad to help in any way to get these 2 very
> > important features compatible with each other.
> 
> So if the ktimersoftd runs and you see the interrupt counter
> incrementing for CPU3 then it would be interesting to figure out why
> there is an armed timer on the second invocation (and none on the first
> one).
> 

In 4.18.5 RT, the issue does not always happen in the second invocation.  
Sometimes it works as expected, but the issue will show up after a few 
tries.

Looks like there are 2 issues when PREEMPT_RT_FULL is enabled
1. Some additional processes are pinned to isolated cores.
2. Timer is armed even though only a single high priority task is running.


> > Thanks,
> > Ramesh
> 
> Sebastian

Re: PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

[Sebastian Andrzej Siewior]

On 2018-08-31 01:18:20 [-0700], Ramesh Thomas wrote:
> Yes. Also posixcputmr and ktimersoftd are not seen if PREEMPT_RT_FULL is not 
> enabled. They don't seem to be running when the timer interrupts occur. But 
> they being there by itself indicates something different is happening.

both threads are created on RT and should be idle unless used.

> I see the issue in 4.14.63 RT as well. There are slight differences in 
> behavior due to changes that went in 4.17, but the main issue is seen there 
> also.

okay, so it is not a regression. Thanks for confirming.

> > > If it is an issue, I would be glad to help in any way to get these 2 very
> > > important features compatible with each other.
> > 
> > So if the ktimersoftd runs and you see the interrupt counter
> > incrementing for CPU3 then it would be interesting to figure out why
> > there is an armed timer on the second invocation (and none on the first
> > one).
> > 
> 
> In 4.18.5 RT, the issue does not always happen in the second invocation.  
> Sometimes it works as expected, but the issue will show up after a few 
> tries.
> 
> Looks like there are 2 issues when PREEMPT_RT_FULL is enabled
> 1. Some additional processes are pinned to isolated cores.

as explained, above that is expected.

> 2. Timer is armed even though only a single high priority task is running.

something must have armed that timer. The clock_nanosleep() API won't
use those new threads but for instance timer_create() / timer_settime()
will.

> > > Thanks,
> > > Ramesh

Sebastian

Re: PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

[Julia Cartwright]

On Fri, Aug 31, 2018 at 06:31:52PM +0200, Sebastian Andrzej Siewior wrote:
> On 2018-08-31 01:18:20 [-0700], Ramesh Thomas wrote:
[..]
> > 2. Timer is armed even though only a single high priority task is running.
> 
> something must have armed that timer. The clock_nanosleep() API won't
> use those new threads but for instance timer_create() / timer_settime()
> will.

Did we actually ever solve the "ktimersoftd uselessly woken on every
tick" problem?

   Julia

Re: PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

[Sebastian Andrzej Siewior]

On 2018-08-31 18:30:19 [-0500], Julia Cartwright wrote:
> On Fri, Aug 31, 2018 at 06:31:52PM +0200, Sebastian Andrzej Siewior wrote:
> > On 2018-08-31 01:18:20 [-0700], Ramesh Thomas wrote:
> [..]
> > > 2. Timer is armed even though only a single high priority task is running.
> > 
> > something must have armed that timer. The clock_nanosleep() API won't
> > use those new threads but for instance timer_create() / timer_settime()
> > will.
> 
> Did we actually ever solve the "ktimersoftd uselessly woken on every
> tick" problem?

I think so, yes. According to the report the problem does not happen the
first invocation. However the "ktimersoftd uselessly woken on every
tick" problem occurred "always".

>    Julia

Sebastian