[Xenomai] I find a way to reduce latency, but I don't know why.

[Xenomai] I find a way to reduce latency, but I don't know why.

[=?gb18030?B?yum09NfT?=]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb18030", Size: 476 bytes --]

The x86 machine have two CPUs, and I need periodic task on CPU1.
In order to reduce latency, a busy task will run on CPU2.

cpu1_task
{
  rt_task_wait_period;
  real_work;
  rt_sem_v(&busySEM);
}

cpu2_task
{
  rt_sem_p(&busySEM);
  rt_timer_spin(spin_ns);
}

And spin_ns will be long enough: spin_ns + real_work_ns > periodic cycle.
Yes, it means rt_code is always running on CPU1 or CPU2.

In doing this, the cpu1_task will get low latency and drift, but why ?

thanks, zhou

Re: [Xenomai] I find a way to reduce latency, but I don't know why.

[Gilles Chanteperdrix]

On Tue, Mar 17, 2015 at 09:41:05PM +0800, 书呆子 wrote:
> The x86 machine have two CPUs, and I need periodic task on CPU1.
> In order to reduce latency, a busy task will run on CPU2.
> 
> cpu1_task
> {
>   rt_task_wait_period;
>   real_work;
>   rt_sem_v(&busySEM);
> }
> 
> cpu2_task
> {
>   rt_sem_p(&busySEM);
>   rt_timer_spin(spin_ns);
> }
> 
> And spin_ns will be long enough: spin_ns + real_work_ns > periodic cycle.
> Yes, it means rt_code is always running on CPU1 or CPU2.
> 
> In doing this, the cpu1_task will get low latency and drift, but why ?

Two possibilities:
- task2 is cache friendly, so if the cache is shared between the two
cpus, it prevents Linux from evicting xenomai and I-pipe code from
the cache, we know that cache has a big influence on latency;
- task2 prevents the cpu from entering idle mode, and so potentially
prevents any low power optimization from happening, low power modes
are generally not latency friendly.

Note however that measurements on an idle system do not really
matter much for finding the worst case latency. Real measurements
should last for hours and with the machine put under load. I bet
with measurements done this way, the differences you observe with
the tests you do would be negligible compared to the worst case
latency. And if there is a real difference, then the I-pipe tracer
would help you find from where it comes.

-- 
					    Gilles.

Re: [Xenomai] I find a way to reduce latency, but I don't know why.

[Gilles Chanteperdrix]

On Wed, Mar 18, 2015 at 09:31:33PM +0800, Zhoupeng wrote:
> > On Tue, Mar 17, 2015 at 09:41:05PM +0800, 书呆子 wrote:
> > > The x86 machine have two CPUs, and I need periodic task on CPU1.
> > > In order to reduce latency, a busy task will run on CPU2.
> > > 
> > > cpu1_task
> > > {
> > >   rt_task_wait_period;
> > >   real_work;
> > >   rt_sem_v(&busySEM);
> > > }
> > > 
> > > cpu2_task
> > > {
> > >   rt_sem_p(&busySEM);
> > >   rt_timer_spin(spin_ns);
> > > }
> > > 
> > > And spin_ns will be long enough: spin_ns + real_work_ns > periodic cycle.
> > > Yes, it means rt_code is always running on CPU1 or CPU2.
> > > 
> > > In doing this, the cpu1_task will get low latency and drift, but why ?
> 
> > Two possibilities:
> > - task2 is cache friendly, so if the cache is shared between the two
> > cpus, it prevents Linux from evicting xenomai and I-pipe code from
> > the cache, we know that cache has a big influence on latency;
> 
> I agree this is the best guess, the I5 CPUs do share L3 cache between them. 
> 
> I also use rt_timer_read() wrap real_work() to measure time consumption: 
> without task2 real_work() expend 20 to 36us, with task2 real_work() 
> expend 20 to 22us to compelte. 
> real_work() only contain float-point arithmetic, the time stability is
> improved obviously.
> 
> If this is cache problem, is task2 the right way to keep code in cache ?
> 
> > - task2 prevents the cpu from entering idle mode, and so potentially
> > prevents any low power optimization from happening, low power modes
> > are generally not latency friendly.
> 
> CONFIG_CPU_FREQ and CONFIG_CPU_IDLE is already disabled, smi workaround
> also enabled.
> I have notice C1E mode is disabled in Xenomai 3, but not in Xenomai 2,
> but C1E is triggered by mwait, I don't think this is the problem.

Linux idle task uses mwait, and nothing prevents the chip maker to
do any optimization he wants when hlt is called. On geode
processors, with some configuration, hlt causes the tsc to be
stopped for instance.

-- 
					    Gilles.

Re: [Xenomai] I find a way to reduce latency, but I don't know why.

[Zhoupeng]

On Wed, Mar 18, 2015 09:39 PM, Gilles wrote:
> > > Two possibilities:
> > > - task2 is cache friendly, so if the cache is shared between the two
> > > cpus, it prevents Linux from evicting xenomai and I-pipe code from
> > > the cache, we know that cache has a big influence on latency;
> > 
> > I agree this is the best guess, the I5 CPUs do share L3 cache between them. 
> > 
> > I also use rt_timer_read() wrap real_work() to measure time consumption: 
> > without task2 real_work() expend 20 to 36us, with task2 real_work() 
> > expend 20 to 22us to compelte. 
> > real_work() only contain float-point arithmetic, the time stability is
> > improved obviously.
> > 
> > If this is cache problem, is task2 the right way to keep code in cache ?
> > 
> > > - task2 prevents the cpu from entering idle mode, and so potentially
> > > prevents any low power optimization from happening, low power modes
> > > are generally not latency friendly.
> > 
> > CONFIG_CPU_FREQ and CONFIG_CPU_IDLE is already disabled, smi workaround
> > also enabled.
> > I have notice C1E mode is disabled in Xenomai 3, but not in Xenomai 2,
> > but C1E is triggered by mwait, I don't think this is the problem.
> 
> Linux idle task uses mwait, and nothing prevents the chip maker to
> do any optimization he wants when hlt is called. On geode
> processors, with some configuration, hlt causes the tsc to be
> stopped for instance.

Yes, this is due to power optimization.
After add "idle=poll" to boot parameter, latency and time comsumption
is just like task2 running.

Just like you said, "idle=poll" makes latency more smaller; but after long
time test, the worst case is still there. 

thanks, zhou

Re: [Xenomai] I find a way to reduce latency, but I don't know why.

[=?gb18030?B?WmhvdXBlbmc=?=]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb18030", Size: 2127 bytes --]

> On Tue, Mar 17, 2015 at 09:41:05PM +0800, Êé´ô×Ó wrote:
> > The x86 machine have two CPUs, and I need periodic task on CPU1.
> > In order to reduce latency, a busy task will run on CPU2.
> > 
> > cpu1_task
> > {
> >   rt_task_wait_period;
> >   real_work;
> >   rt_sem_v(&busySEM);
> > }
> > 
> > cpu2_task
> > {
> >   rt_sem_p(&busySEM);
> >   rt_timer_spin(spin_ns);
> > }
> > 
> > And spin_ns will be long enough: spin_ns + real_work_ns > periodic cycle.
> > Yes, it means rt_code is always running on CPU1 or CPU2.
> > 
> > In doing this, the cpu1_task will get low latency and drift, but why ?

> Two possibilities:
> - task2 is cache friendly, so if the cache is shared between the two
> cpus, it prevents Linux from evicting xenomai and I-pipe code from
> the cache, we know that cache has a big influence on latency;

I agree this is the best guess, the I5 CPUs do share L3 cache between them. 

I also use rt_timer_read() wrap real_work() to measure time consumption: 
without task2 real_work() expend 20 to 36us, with task2 real_work() 
expend 20 to 22us to compelte. 
real_work() only contain float-point arithmetic, the time stability is
improved obviously.

If this is cache problem, is task2 the right way to keep code in cache ?

> - task2 prevents the cpu from entering idle mode, and so potentially
> prevents any low power optimization from happening, low power modes
> are generally not latency friendly.

CONFIG_CPU_FREQ and CONFIG_CPU_IDLE is already disabled, smi workaround
also enabled.
I have notice C1E mode is disabled in Xenomai 3, but not in Xenomai 2,
but C1E is triggered by mwait, I don't think this is the problem.

> Note however that measurements on an idle system do not really
> matter much for finding the worst case latency. Real measurements
> should last for hours and with the machine put under load. I bet
> with measurements done this way, the differences you observe with
> the tests you do would be negligible compared to the worst case
> latency. And if there is a real difference, then the I-pipe tracer
> would help you find from where it comes.

OK, I will try I-pipe tracer.