[Adeos-main] 260 uSec latency and no idea why

[Adeos-main] 260 uSec latency and no idea why

[Dirk Roloff]

Hello List,

i have a interrupt latency of ~ 260 µSec and i do not know why.
I am using kernel 2.4.28 + adeos r10
Have a Celeron 2 GHz. Single CPU with IO-APIC (But tested this also with noapic boot parameter)
APM is disabled (The IDLE calls could cause some latency)
Running in Textmode - so no X is started.
Having a IDE-Device which dose not use DMA.
Any more Infos needed ?

The results are 4 to 8 µSec latency in the Realtime Domain normaly but somtimes i see ~260 - 270 µSek.
The only thing i found in the kernel tree is the prink() funktion which uses some spinlock which disables irq's.
Could this be the source. Or has anybody a idea where to search deeper ?


How i mesure the delay: (This is only a test, because i have latency problems with an ugly device driver)

1.) Connect pin 9 to 10 of my ParPort. So if i set Pin 10 up it aserts a IRQ 7 on Pin 9
2.) Write a module to messure the latency.

Short description:

- creating a proc-fs entry 
- create a relatime domain with higher Priority then Linux.
- in the domain main adeos_virtualize_irq(7, &rt_latency_isr,rt_latency_ack, IPIPE_PASS_MASK | IPIPE_HANDLE_MASK );

The proc-fs read function i will 
{
    interruptcount=0;

    adeos_hw_local_irq_save( flags );
    do_gettimeofday(&tv[0]);
    outb(0x80,SPPDATAPORT); /* assert the interrupt signal */
    adeos_hw_local_irq_restore( flags );

    while( !interruptcount )
    {
            latency_sleep(1);
                        if(signal_pending(current)) break;
    }

   calc timediff between tv[0] tv[1] tv[2] and tv[3] and give the reslut back
}

in rt_latency_ack, 
{
	do_gettimeofday(&tv[1]);
	return 0;
}

in rt_latency_isr the realtime isr
{
	do_gettimeofday(&tv[2]);
}
in the Linux isr
{
  do_gettimeofday(&tv[3]);
  outb(0x00,SPPDATAPORT); /* deassert the interrupt signal */
  interruptcount++;
}

Could anybody help ?

Thanks 
Dirk Roloff

Re: [Adeos-main] 260 uSec latency and no idea why

[Paolo Mantegazza]

Dirk Roloff wrote:
> Hello List,
> 
> i have a interrupt latency of ~ 260 µSec and i do not know why.
> I am using kernel 2.4.28 + adeos r10
> Have a Celeron 2 GHz. Single CPU with IO-APIC (But tested this also with noapic boot parameter)
> APM is disabled (The IDLE calls could cause some latency)
> Running in Textmode - so no X is started.
> Having a IDE-Device which dose not use DMA.
> Any more Infos needed ?
> 
> The results are 4 to 8 µSec latency in the Realtime Domain normaly but somtimes i see ~260 - 270 µSek.
> The only thing i found in the kernel tree is the prink() funktion which uses some spinlock which disables irq's.
> Could this be the source. Or has anybody a idea where to search deeper ?
> 
> 
> How i mesure the delay: (This is only a test, because i have latency problems with an ugly device driver)
> 
> 1.) Connect pin 9 to 10 of my ParPort. So if i set Pin 10 up it aserts a IRQ 7 on Pin 9
> 2.) Write a module to messure the latency.
> 
> Short description:
> 
> - creating a proc-fs entry 
> - create a relatime domain with higher Priority then Linux.
> - in the domain main adeos_virtualize_irq(7, &rt_latency_isr,rt_latency_ack, IPIPE_PASS_MASK | IPIPE_HANDLE_MASK );
> 
> The proc-fs read function i will 
> {
>     interruptcount=0;
> 
>     adeos_hw_local_irq_save( flags );
>     do_gettimeofday(&tv[0]);
>     outb(0x80,SPPDATAPORT); /* assert the interrupt signal */
>     adeos_hw_local_irq_restore( flags );
> 
>     while( !interruptcount )
>     {
>             latency_sleep(1);
>                         if(signal_pending(current)) break;
>     }
> 
>    calc timediff between tv[0] tv[1] tv[2] and tv[3] and give the reslut back
> }
> 
> in rt_latency_ack, 
> {
> 	do_gettimeofday(&tv[1]);
> 	return 0;
> }
> 
> in rt_latency_isr the realtime isr
> {
> 	do_gettimeofday(&tv[2]);
> }
> in the Linux isr
> {
>   do_gettimeofday(&tv[3]);
>   outb(0x00,SPPDATAPORT); /* deassert the interrupt signal */
>   interruptcount++;
> }
> 

Have you tried using rt_get_cpu_time_ns to avoid mixing up with Linux stuff?
Have you checkd the interrupt level acknolwdge jitter/latency using the 
calibration tool available in RTAI?
Can you change your code so that it can be used anywhere? In such a case 
it would be simple to run it on a safe machine (e.g. mine) and see if it 
can be repeated, or is machine specific (yours).

Paolo.

Re: [Adeos-main] 260 uSec latency and no idea why

[Dirk Roloff]

Am Donnerstag, 11. März 2004 20:04 schrieb Paolo Mantegazza:

Hi Paolo,

> Dirk Roloff wrote:
> Have you tried using rt_get_cpu_time_ns to avoid mixing up with Linux
> stuff? 


No i don't because i am not using rtai. I "just" use an adeos patched kernel.
But i changed the do_gettimeofday() by a rdtsc call. I dont want to know the time,
just the time elapsed. So i dont need the time calculation done by do_gettimeofday()
and i avoid mixing stuff. But the results are the same. :-(

>Have you checkd the interrupt level acknolwdge jitter/latency using
> the calibration tool available in RTAI?

again - no rtai

> Can you change your code so that it can be used anywhere? In such a case
> it would be simple to run it on a safe machine (e.g. mine) and see if it
> can be repeated, or is machine specific (yours).

Yes but it will not help a lot. I can't see such a latency on other machines.
f.e. on my Box here at home 7-17 MicroSec in the adeos domain.
And i try stressing the Box- but its a complete different h/ware.

So it must be the celeron box or some kernel/compiler settings on it.

But what is able to delay the delivery of an interrupt if there is no one 
calling cli or masking irq's (Both is prohibited by adeos isn't it ?)

One thing would be waking a suspended cpu - but i disabled APM.
Someone told me Intel SpeedStep switching the mode will take hundrets of MicroSecs
but i didn't find what this is yet. Have to google some more.

Dirk.

Re: [Adeos-main] 260 uSec latency and no idea why

[Der Herr Hofrat]

> Am Donnerstag, 11. März 2004 20:04 schrieb Paolo Mantegazza:
> 
> Hi Paolo,
> 
> > Dirk Roloff wrote:
> > Have you tried using rt_get_cpu_time_ns to avoid mixing up with Linux
> > stuff? 
> 
> 
> No i don't because i am not using rtai. I "just" use an adeos patched kernel.
> But i changed the do_gettimeofday() by a rdtsc call. I dont want to know the time,
> just the time elapsed. So i dont need the time calculation done by do_gettimeofday()
> and i avoid mixing stuff. But the results are the same. :-(
> 
> >Have you checkd the interrupt level acknolwdge jitter/latency using
> > the calibration tool available in RTAI?
> 
> again - no rtai
> 
> > Can you change your code so that it can be used anywhere? In such a case
> > it would be simple to run it on a safe machine (e.g. mine) and see if it
> > can be repeated, or is machine specific (yours).
> 
> Yes but it will not help a lot. I can't see such a latency on other machines.
> f.e. on my Box here at home 7-17 MicroSec in the adeos domain.
> And i try stressing the Box- but its a complete different h/ware.
> 
> So it must be the celeron box or some kernel/compiler settings on it.
> 
> But what is able to delay the delivery of an interrupt if there is no one 
> calling cli or masking irq's (Both is prohibited by adeos isn't it ?)

It is a software layer - no hardware prhibiting this - so if
you have a X-server doing cli/sti in the driver or a binary driver that
does somthing like that - they adeos can't stop it.

> 
> One thing would be waking a suspended cpu - but i disabled APM.
> Someone told me Intel SpeedStep switching the mode will take hundrets of MicroSecs
> but i didn't find what this is yet. Have to google some more.
>a
could you run the test code in the adeos cvs repository
and send me the numbers ?

hofrat 

Re: [Adeos-main] 260 uSec latency and no idea why

[Dirk Roloff]

> > So it must be the celeron box or some kernel/compiler settings on it.
> >
> > But what is able to delay the delivery of an interrupt if there is no one
> > calling cli or masking irq's (Both is prohibited by adeos isn't it ?)
>
> It is a software layer - no hardware prhibiting this - so if
> you have a X-server doing cli/sti in the driver or a binary driver that
> does somthing like that - they adeos can't stop it.

There is no X on it. Think of a Box that is very slim.
Its a Satelitereceiver (My problem is the driver for the receiver card - the tests i'am doing are just
to figure out where i get the latencys)
Ok here a list of whats in there:

h/ware:
CPU: Celeron 4 GHz
256 MB MainMemory
A Intel D865GBF MainBoard:
	Intel 856G Chipset
	Intel® Extreme Graphics 2 controller (non AGP)
	10/100 Ethernet onBoard (using e100.o module)
	onBoard Audio, USB etc... not user no drivers loaded or compiled in the kernel
	Two Serial ATA IDE interfaces (ICH5)
ReceiverCard - but driver not loaded while this testings.
A FlashDisk on the IDE-Controler. It is _not_ using DMA !!!!

s/ware:
self compiled Kernel 2.4.24-adeos r10
(All options not needed are disabled - make is smal)
root patition mounted readonly
a ramdisk of 64 MB

>ps fax
  PID  Uid     VmSize Stat Command
    1 root        520 S   init
    2 root            SW  [keventd]
    3 root            SWN [ksoftirqd_CPU0]
    4 root            SW  [kswapd]
    5 root            SW  [bdflush]
    6 root            SW  [kupdated]
   39 root        664 S   /bin/sh
   40 root        524 S   init
   92 root        588 R   ps fax

init is a busybox binary (hmmm ... ok i will check this for cli calls)

Thats it. Any suspects found?

>
> > One thing would be waking a suspended cpu - but i disabled APM.
> > Someone told me Intel SpeedStep switching the mode will take hundrets of
> > MicroSecs but i didn't find what this is yet. Have to google some more.
> >a
>
> could you run the test code in the adeos cvs repository
> and send me the numbers ?

Long mail is on the way to you.
Result:
irq_jitter shows:
Out of bounds 222 us
Out of bounds -223 us

Thanks 
Dirk

Re: [Adeos-main] 260 uSec latency and no idea why

[Paolo Mantegazza]

Der Herr Hofrat wrote:

> 
> It is a software layer - no hardware prhibiting this - so if
> you have a X-server doing cli/sti in the driver or a binary driver that
> does somthing like that - they adeos can't stop it.
> 

This is true also. But it is really annoying to verify it. In suspicion 
of such an event I tried to disassamble a few drivers and then look for 
cli/sti. It turned out there were in asm but then they where not there 
in . In fact it was the disassambler that misunderstood FA/FB in 
disassembling a switch table. Then I concluded that it is not a simple 
matter of objdumping and seeing so gave up.

A further example: I've a nice new ATHLON machine that in alpha mode 
shows very low latencies under high Linux load, but as I'm in X they 
skyrockets to msecs, and that happens with both RTAI on ADEOS and on its 
old legacy RTHAL.

Yours is the first case I step on such a problem with Celerons.

Paolo.

Re: [Adeos-main] 260 uSec latency and no idea why

[Gilles Chanteperdrix]

Paolo Mantegazza wrote:
 > Der Herr Hofrat wrote:
 > Yours is the first case I step on such a problem with Celerons.

What about IDE controllers bus mastering DMA ? Can't this be the cause
of big bus locks ?

-- 


					    Gilles Chanteperdrix.

Re: [Adeos-main] 260 uSec latency and no idea why

[Der Herr Hofrat]

> Paolo Mantegazza wrote:
>  > Der Herr Hofrat wrote:
>  > Yours is the first case I step on such a problem with Celerons.
> 
> What about IDE controllers bus mastering DMA ? Can't this be the cause
> of big bus locks ?
>
260 us sounds a bit much - this would have to show up 
on many systems - large caches and bus mastering DMA are an issue
but I never found more than 80us (flush_all) and DMA has been below
that.

hofrat 

Re: [Adeos-main] 260 uSec latency and no idea why

[Dirk Roloff]

Am Freitag, 12. März 2004 10:52 schrieb Gilles Chanteperdrix:
> Paolo Mantegazza wrote:
>  > Der Herr Hofrat wrote:
>  > Yours is the first case I step on such a problem with Celerons.
>
> What about IDE controllers bus mastering DMA ? Can't this be the cause
> of big bus locks ?

Not if i understand the pci latency timer right.
its 32 PCI-Cycles in my case - this time after the grant a DMA master had to give away the bus.
So the maximum latency is number of posible dma-masters * latency timer.
calculating this on a 33 MHz bus is under 1 microsecond.

I am unsure for how long the CPU can hold the Bus.

And my IDE controler is not using DMA. 
And on the other Hand what has the ParPort to do with teh PCI bus ?

Dirk

Re: [Adeos-main] 260 uSec latency and no idea why

[Paolo Mantegazza]

Dirk Roloff wrote:
> Am Donnerstag, 11. März 2004 20:04 schrieb Paolo Mantegazza:
> 
> Hi Paolo,
> 
> 
>>Dirk Roloff wrote:
>>Have you tried using rt_get_cpu_time_ns to avoid mixing up with Linux
>>stuff? 
> 
> 
> 
> No i don't because i am not using rtai. I "just" use an adeos patched kernel.
> But i changed the do_gettimeofday() by a rdtsc call. I dont want to know the time,
> just the time elapsed. So i dont need the time calculation done by do_gettimeofday()
> and i avoid mixing stuff. But the results are the same. :-(
> 
> 
>>Have you checkd the interrupt level acknolwdge jitter/latency using
>>the calibration tool available in RTAI?
> 
> 
> again - no rtai
> 
> 
>>Can you change your code so that it can be used anywhere? In such a case
>>it would be simple to run it on a safe machine (e.g. mine) and see if it
>>can be repeated, or is machine specific (yours).
> 
> 
> Yes but it will not help a lot. I can't see such a latency on other machines.
> f.e. on my Box here at home 7-17 MicroSec in the adeos domain.
> And i try stressing the Box- but its a complete different h/ware.
> 
> So it must be the celeron box or some kernel/compiler settings on it.
> 
> But what is able to delay the delivery of an interrupt if there is no one 
> calling cli or masking irq's (Both is prohibited by adeos isn't it ?)
> 
> One thing would be waking a suspended cpu - but i disabled APM.
> Someone told me Intel SpeedStep switching the mode will take hundrets of MicroSecs
> but i didn't find what this is yet. Have to google some more.
> 

On some of the newest PC chipsets there seems to be a lot of 
possibilities to programm chipsets so that real time bocomes unfeasable. 
The locking of the bus may come from smart cards and a few other things. 
You are not using RTAI but in RTAI there is a README with a way to 
reprogram INTEL chipsets to avoid one of those problems.

Paolo.

Re: [Adeos-main] 260 uSec latency and no idea why

[Dirk Roloff]

> On some of the newest PC chipsets there seems to be a lot of
> possibilities to programm chipsets so that real time bocomes unfeasable.
> The locking of the bus may come from smart cards and a few other things.
> You are not using RTAI but in RTAI there is a README with a way to
> reprogram INTEL chipsets to avoid one of those problems.

Paolo, after reading the READMI.SMI i think we are getting closer.
intel ICH5 is used here. 
Any updates on INTEL 82801EB I/O Controller Hub (ICH5)

Dirk

Re: [Adeos-main] 260 uSec latency and no idea why

[Vitor Angelo]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]


Hi Dirk,

# Any updates on INTEL 82801EB I/O Controller Hub (ICH5)
# 
# Dirk

could you please test the attached code? You should do a

echo enable > /proc/smi/global

and a

echo disable > /proc/smi/global

to test if the "global SMI bit" changes anything for you.

The other entries under /proc/smi are not important, unless you want to
test changes in BIOS or ACPI, doing a "cat /proc/smi/...". The "global
SMI bit" should override everything.

Regards,

Vitor.


[-- Attachment #2: smi.tar.bz2 --]
[-- Type: application/octet-stream, Size: 2760 bytes --]

Re: [Adeos-main] 260 uSec latency and no idea why

[Dirk Roloff]

Hi Vitor,

many thanks - your code works fine. And the jitters are gone after disableing the
SMI.

Also many thanks to all the other helping me to find out whats going on.
That i had never find on my own.

Dirk


Am Montag, 15. März 2004 09:53 schrieb Vitor Angelo:
> Hi Dirk,
>
> # Any updates on INTEL 82801EB I/O Controller Hub (ICH5)
> #
> # Dirk
>
> could you please test the attached code? You should do a
>
> echo enable > /proc/smi/global
>
> and a
>
> echo disable > /proc/smi/global
>
> to test if the "global SMI bit" changes anything for you.
>
> The other entries under /proc/smi are not important, unless you want to
> test changes in BIOS or ACPI, doing a "cat /proc/smi/...". The "global
> SMI bit" should override everything.
>
> Regards,
>
> Vitor.