[EVL] Unstable with kernel command line nohz=on nohz_full=2-3

[EVL] Unstable with kernel command line nohz=on nohz_full=2-3

[Hannes Diethelm]

Hello

Due to I am switching back and forwards between PREEMT_RT and Xenomai4, I just left
the kernel command line with the best PREEMT_RT performance in for Xenomai4. This is:

isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll nohz=on nohz_full=2-3 intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=off

I had strange behavior when running an EVL task on CPU 2 or 3. Without network, it was fine. But as soon as I run my application with network enabled,
it just freeze the threads from time to time. It looks like evl_sleep_until() just never wakes up sometimes. The reason might be the active network but also
somewhere else in the code. Without network, there are also no 10'000 ns sleeps.

The code is:
evl_read_clock(EVL_CLOCK_MONOTONIC, &ts); -> ts: 1682 s 249982420 ns
rtapi_timespec_advance(ts, ts, ns); -> ts: 1682 s 249992420 ns
evl_sleep_until(EVL_CLOCK_MONOTONIC, &ts); -> Never returns

As soon as I remove "nohz=on nohz_full=2-3", everything works fine. Might be evl relies on scheduling-clock ticks?

After that resolved, I changed the command line to:
isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=off evl.oobcpus=3
so the Ethernet and also the main task run all on CPU3.

After setting also the IRQ affinity for the Ethernet card to CPU3, the max latency is now quite good, max jitter 4us for the task entry, 15us at the end.

I just report this anyway just in case it is a bug or others have the same issue.

Regards
Hannes

Re: [EVL] Unstable with kernel command line nohz=on nohz_full=2-3

[Philippe Gerum]

Hannes Diethelm <hannes.diethelm@gmail.com> writes:

> Hello
>
> Due to I am switching back and forwards between PREEMT_RT and Xenomai4, I just left
> the kernel command line with the best PREEMT_RT performance in for Xenomai4. This is:
>
> isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll nohz=on nohz_full=2-3 intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=off
>
> I had strange behavior when running an EVL task on CPU 2 or 3. Without network, it was fine. But as soon as I run my application with network enabled,
> it just freeze the threads from time to time. It looks like evl_sleep_until() just never wakes up sometimes. The reason might be the active network but also
> somewhere else in the code. Without network, there are also no 10'000 ns sleeps.
>
> The code is:
> evl_read_clock(EVL_CLOCK_MONOTONIC, &ts); -> ts: 1682 s 249982420 ns
> rtapi_timespec_advance(ts, ts, ns); -> ts: 1682 s 249992420 ns
> evl_sleep_until(EVL_CLOCK_MONOTONIC, &ts); -> Never returns
>
> As soon as I remove "nohz=on nohz_full=2-3", everything works fine. Might be evl relies on scheduling-clock ticks?
>

No, it can't and doesn't, that would inject in-band generated latency
into the oob timeline.

> After that resolved, I changed the command line to:
> isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=off evl.oobcpus=3
> so the Ethernet and also the main task run all on CPU3.
>
> After setting also the IRQ affinity for the Ethernet card to CPU3, the max latency is now quite good, max jitter 4us for the task entry, 15us at the end.
>
> I just report this anyway just in case it is a bug or others have the same issue.
>

Thanks for the heads up. I'll have a look.

-- 
Philippe.