Re: Re: Disabling lapic timer for a certain core

Re: Re: Disabling lapic timer for a certain core

[M. Koehrer]

Hi Luis,

thanks for the reply.
It is an Intel Core2Quad CPU - one CPU with 4 cores,  
this is a SMP system...
Concerning the SMI issue:
I have globally disabled SMI with the LPC registers...
The main board is a server board from Supermicro, 
no USB enabled, VGA text mode only. No X running.

The total amount of time we have for a cycle is 40us,
most often the application manages to run within 25-30us, 
however there are some jitters 
that increases the run time to be 45us which is too slow.
I have done a kernel hack to be able to return directly from
the interrupt routine of the "Local Timer Interrupt" for core 3.
This helps, however it is really an ugly hack and I am looking
for a smarter way to do that. I have done this just to identify
the cause of the jitter.

The issue is here, that the interrupt routine of the kernel
takes too long here.
It would be fine to have the timer interrupt called more 
often and to process with each call only a subset of the 
jobs to be done...
This would reduce the time the CPU the user mode 
is interrupted by the timer routine.

The idea of running this application in an endless loop is to avoid
to use timers (including the interrupt latency caused by them).
By pure polling, no interrupt is required.

Regards

Mathias

> | Hi all!
> | 
> | I am running the RT_PREEMPT kernel 2.6.31.2-rt13 on a Intel Quad Core
> CPU.
> | I start my kernel with isolcpus=1-7 option to force all processes to run
> on CPU core 0 only.
> | Now we have the need for a very fast loop. Within this loop some accesses
> from/to a PCIe I/O board
> | (mapped in user space) and some additional computation has to be done.
> | For this, I use a real time thread, run this on CPU core 3 and let it run
> in an endless polling loop.
> | So far everything works fine.
> | This thread is the only running user mode thread on CPU core 3.
> | However, we measure some run time jitters when accessing the I/O board in
> the range of up to 
> | 15 microseconds which are not tolerable by the application.
> | I see that on all cores the "Local Timer Interrupt" occurs 100 times a
> | second (of course this is the timer frequency select in the kernel
> configuration).
> 
> Are CPU0 and CPU3 on the same socket? Are you using a SMP or a NUMA box? I
> would suggest running your application in a CPU on a different socket just
> to ensure you are not having any cache issues.
> 
> Have you tried running hwlat_detect or smi-test? Your 15us threshold is
> pretty
> tight and could easilly be affected by SMI spikes (if present in your
> system).
> 
> Luis
> 


-- 
Mathias Koehrer
mathias_koehrer@arcor.de


Tolle Dekolletés oder scharfe Tatoos? Vote jetzt ... oder mach selbst mit und zeige Deine Schokoladenseite
bei Topp oder Hopp von Arcor: http://www.arcor.de/rd/footer.toh
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disabling lapic timer for a certain core

[Thomas Gleixner]

On Thu, 4 Mar 2010, M. Koehrer wrote:
> Hi Luis,

Please do _NOT_ top post. Please reply inline and in context.

> thanks for the reply.
> It is an Intel Core2Quad CPU - one CPU with 4 cores,  
> this is a SMP system...
> Concerning the SMI issue:
> I have globally disabled SMI with the LPC registers...

I hope you know what you are doing. Disabling SMIs can be dangerous in
various aspects:

  - thermal protection might be disabled by it
  - SMIs which fix chip(set) erratas are not longer working

You should at least confirm with your hardware vendor whether it's
safe to do so and which side effects you have to take into account.

> The main board is a server board from Supermicro, 
> no USB enabled, VGA text mode only. No X running.
> 
> The total amount of time we have for a cycle is 40us,

FYI, 40us is in the range where the hardware induced latencies can
bite you already badly. You run on a machine with shared L2 caches, so
your other core(s) might evict your code / data and you run into cache
misses which might take a while due to DMAs running etc...

These machines are designed for throughput, not for deterministic
behaviour in the single digit micro seconds range.

> The issue is here, that the interrupt routine of the kernel
> takes too long here.
> It would be fine to have the timer interrupt called more 
> often and to process with each call only a subset of the 
> jobs to be done...
> This would reduce the time the CPU the user mode 
> is interrupted by the timer routine.

Err, by splitting the work you introduce even more overhead. That's
the wrong approach. The first question is which timers are running on
that CPU as you have isolated it.

/proc/timer_stats and /proc/timer_list and the event tracer might help
you to identify them.

In theory it's possible to remove the timer interrupt from such an
isolated core completely, but there needs to be some work done vs. the
scheduler, accounting, RCU etc. There are people looking into this,
but we have no patches yet.

> The idea of running this application in an endless loop is to avoid
> to use timers (including the interrupt latency caused by them).
> By pure polling, no interrupt is required.

What kind of application is that ? Data acquistion or closed loop
processsing ?

Thanks,

	tglx

Re: Re: Disabling lapic timer for a certain core

[M. Koehrer]

Hi Thomas, 

thank you very much for the reply.
> Please do _NOT_ top post. Please reply inline and in context.
Sorry for that...

> > The issue is here, that the interrupt routine of the kernel
> > takes too long here.
> > It would be fine to have the timer interrupt called more 
> > often and to process with each call only a subset of the 
> > jobs to be done...
> > This would reduce the time the CPU the user mode 
> > is interrupted by the timer routine.
> 
> Err, by splitting the work you introduce even more overhead. That's
> the wrong approach. The first question is which timers are running on
> that CPU as you have isolated it.
You are right. The total overhead is of course larger.
However, the overhead that could appear within a single of our 40us loops
would be smaller. It is fine for me to have a 5us add on with each loop.
But it is not allowed to have a 15us add on with every 1000th loop...

> 
> In theory it's possible to remove the timer interrupt from such an
> isolated core completely, but there needs to be some work done vs. the
> scheduler, accounting, RCU etc. There are people looking into this,
> but we have no patches yet.
I have checked the LAPIC addresses via the MSRs.
All LAPIC addresses for all CPU cores are the same. 
I assume they share the very same configuration, thus a minimum step 
would be to make a copy of this configuration data and to let CPU core 3
point to this copy. This would allow to disable the timer.

 
> 
> What kind of application is that ? Data acquistion or closed loop
> processsing ?
I am running a close loop application.

Thank you very much.

Regards

Mathias

-- 
Mathias Koehrer
mathias_koehrer@arcor.de


Tolle Dekolletés oder scharfe Tatoos? Vote jetzt ... oder mach selbst mit und zeige Deine Schokoladenseite
bei Topp oder Hopp von Arcor: http://www.arcor.de/rd/footer.toh
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: Disabling lapic timer for a certain core

[Thomas Gleixner]

On Fri, 5 Mar 2010, M. Koehrer wrote:
> > In theory it's possible to remove the timer interrupt from such an
> > isolated core completely, but there needs to be some work done vs. the
> > scheduler, accounting, RCU etc. There are people looking into this,
> > but we have no patches yet.
> I have checked the LAPIC addresses via the MSRs.
> All LAPIC addresses for all CPU cores are the same. 
> I assume they share the very same configuration, thus a minimum step 
> would be to make a copy of this configuration data and to let CPU core 3
> point to this copy. This would allow to disable the timer.

Really ? If it would be enough to disable the timer interrupt and not
let it fire, it would have been done years ago.

Did you even try to read what I said above ?

> >                 ...., but there needs to be some work done vs. the
> > scheduler, accounting, RCU etc.

Linux was not designed that way and it requires a non trivial amount
of work to get this sorted out:

> > There are people looking into this, but we have no patches yet.

Thanks,

	tglx