Re: Regression on rt kernel while using POSIX timers

["Patel, Vedang" <vedang.patel@intel.com>]

On Thu, 2017-02-16 at 02:34 +0000, Patel, Vedang wrote:
> On Wed, 2017-02-15 at 20:05 -0600, Julia Cartwright wrote:
> > 
> > On Wed, Feb 15, 2017 at 05:54:47PM +0100, bigeasy@linutronix.de
> > wrote:
> > > 
> > > 
> > > On 2017-02-13 18:48:33 [+0000], Patel, Vedang wrote:
> > > > 
> > > > 
> > > > I am getting very similar results even if I change the priority
> > > > of
> > > > ktimersoftd to 99. Are there any recent rt patches which might
> > > > have
> > > > changed the behaviour of POSIX timers? 
> > > I had this running
> > > 
> > > > 
> > > > 
> > > > ~# cyclictest -t1 -p 80 -i 500 -l 100000
> > > > # /dev/cpu_dma_latency set to 0us
> > > > policy: fifo: loadavg: 627.48 661.98 543.65 406/2254
> > > > 18149          
> > > > 
> > > > T: 0 (16148) P:80 I:500 C: 100000 Min:      5 Act:   11
> > > > Avg:   11
> > > > Max: 23
> > > on a AMD-A10 box and the latest v4.9-RT. This does not look that
> > > bad.
> > > From tracing it doesn't look too good either. The -m option
> > > should
> > > be
> > > your friend. Since the cyclictest isn't pinned to a CPU and I
> > > have
> > > four
> > > of them, the scheduler decides to migrate cyclictest on each wake
> > > up.
> > > yay. cyclictest itself gets woken up via a signal from the
> > > ktimersoftirq.
> > Perhaps a separate off-shoot topic, but has there been significant
> > changes in v4.9 upstream (or v4.9-rt) which impact how often
> > migration
> > is performed?  We've seen what appears to be much more aggressive
> > migration of SCHED_FAIR tasks, (aggressive in the sense that more
> > threads are migrated more often), which appears to cause contention
> > on
> > rq-locks impacting latencies in the wake_up paths.
> > 
> > We're still working to further characterize at this
> > point....hopefully
> > more information forthcoming.
> I tried conducting similar experiments on v4.4.47-rt59. But, the
> results are same as v4.9 kernel.
> 
> My machine also has 8 cores and migration might cause regressions.
> Let
> me try pinning the processes to a single CPU and see if that is
> making
> any difference. 
> 
So, I tried pinning the process to a single CPU. But, turns out that
the latency in case of Pinned CPU is worse than the latency when CPU is
not pinned. I tried 2 different ways for pinning the cyclictest task to
a CPU -- -a argument in cyclictest and taskset command to set a CPU. I
got the same results for both the methods. I also tried different CPUs
thinking one of the CPUs might have something else pinned. But, the
result remained the same.

Also, I checked the number of cpu migrations and context switches using
perf stat. Following are the results for both the cases:

cyclictest not pinned to CPU: the number of CPU migrations and context
switches are equal to the number of loops I am running for cyclictest.
Here, there are a lot of context switches with the ktimersoftd process.

cyclictest pinned to a CPU: there are very few CPU migrations. But, the
number of context switches are approximately 3 times the number of
loops. Most of the context switches are with the swapper (idle)
process.

In both the cases, the number of page faults are pretty low (~65).

I also tried similar experiments with mainline kernel (v4.4.47). The
number of CPU migrations were pretty low (single digits) for both the
cases described above. Also, the number of context switches were equal
to the number of loops provided as an argument to cyclictest.

Thanks,
Vedang Patel
Software Engineer
Intel Corporation

> More information to follow soon...
> 
> Thanks for all the inputs, 
> Vedang Patel
> Software Engineer
> Intel Corporation
> > 
> >    Julia\x04�{.n�+�������+%��lzwm��b�맲��r��zX��\x1a�ǫ�)���w*\x1fjg���\x1e�����ݢ
> > j/���z�ޖ��2�ޙ���&�)ߡ�a��\x7f��\x1e�G���h�\x0f�j:+v���w�٥