* [Xenomai-help] AT91SAM9260 latency
@ 2008-01-23 6:52 Juan Antonio Garcia Redondo
2008-01-23 10:04 ` Gilles Chanteperdrix
0 siblings, 1 reply; 27+ messages in thread
From: Juan Antonio Garcia Redondo @ 2008-01-23 6:52 UTC (permalink / raw)
To: xenomai; +Cc: jagarcia
Hi list,
These are my latency results over a custom board based on AT91SAM9260.
Environment:
kernel 2.6.20 + xenomai 2.4.0
Load: dd if=/dev/zero of=/dev/null plus an external ping -f to
stress the ethernet.
The results:
running: ./run -- -p 500 -T 60 -t0 # latency
== Sampling period: 500 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...
RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
RTD| 45.747| 62.177| 78.286| 0| 45.747| 78.286
.............................................................................
RTD| 39.948| 61.211| 78.286| 0| 39.304| 88.273
---|------------|------------|------------|--------|-------------------------
RTS| 39.304| 61.855| 88.273| 0| 00:01:00/00:01:00
Thu Mar 30 01:51:36 UTC 2000
running: ./run -- -p 500 -T 60 -t1 # latency
== Sampling period: 500 us
== Test mode: in-kernel periodic task
== All results in microseconds
warming up...
RTT| 00:00:01 (in-kernel periodic task, 500 us period, priority 99)
RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
RTD| 19.007| 31.731| 43.813| 0| 19.007| 43.813
.............................................................................
---|------------|------------|------------|--------|-------------------------
RTS| 13.501| 31.126| 71.188| 0| 00:01:00/00:01:00
Thu Mar 30 01:52:52 UTC 2000
running: ./run -- -p 500 -T 60 -t2 # latency
*
*
* Type ^C to stop this application.
*
*
== Sampling period: 500 us
== Test mode: in-kernel timer handler
== All results in microseconds
warming up...
RTT| 00:00:01 (in-kernel timer handler, 500 us period, priority 99)
RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
.............................................................................
---|------------|------------|------------|--------|-------------------------
RTS| -0.977| 8.521| 28.337| 0| 00:01:00/00:01:00
Thu Mar 30 01:54:07 UTC 2000
running: ./run -- -T 120 # switchtest
== Testing FPU check routines...
== FPU check routines: unimplemented, skipping FPU switches tests.
== Threads: sleeper-0 rtk-1 rtk-2 rtup-3 rtup-4 rtus-5 rtus-6 rtuo-7 rtuo-8
RTT| 00:00:01
RTH|ctx switches|-------total
RTD| 756| 756
..............................
RTD| 723| 87717
RTD| 762| 88479
RTD| 750| 89229
RTD| 42| 89271
Thu Mar 30 01:56:22 UTC 2000
running: ./run -- -p500 -h # switchbench
== Sampling period: 500 us
== Do not interrupt this program
RTH| lat min| lat avg| lat max| lost
RTD| 47.358| 52.512| 68.298| 0
---|---range-|---samples
HSD| 47 - 48 | 1
HSD| 48 - 49 | 5
HSD| 49 - 50 | 6
HSD| 50 - 51 | 2065
HSD| 51 - 52 | 22440
HSD| 52 - 53 | 41582
HSD| 53 - 54 | 19749
HSD| 54 - 55 | 11696
HSD| 55 - 56 | 2131
HSD| 56 - 57 | 1
HSD| 60 - 61 | 1
HSD| 62 - 63 | 1
HSD| 63 - 64 | 21
HSD| 64 - 65 | 95
HSD| 65 - 66 | 107
HSD| 66 - 67 | 63
HSD| 67 - 68 | 29
HSD| 68 - 69 | 3
HSS| 99996| 52.271| 1.278
Thu Mar 30 01:57:25 UTC 2000
running: ./run -- -p 10 -n -l 1000 # cyclictest
T: 0 ( 0) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
^[[3A1.30 1.01 0.63 2/33 2107
...........................................................................................
T: 0 ( 2107) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
^[[3A1.30 1.01 0.63 2/33 2107
T: 0 ( 2107) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
^[[3A1.30 1.01 0.63 2/33 2107
T: 0 ( 2107) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
^[[3A1.30 1.01 0.63 2/33 2107
T: 0 ( 2107) P:10 I: 1000 C: 19 Min: 45 Act: 45 Avg: 69 Max: 78
^[[3A1.30 1.01 0.63 2/33 2107
...........................................................................................
T: 0 ( 2107) P:10 I: 1000 C: 999 Min: 39 Act: 48 Avg: 65 Max: 84
I see everything OK except for the first samples of cyclictests. Any comments ?
Gilles, I've read your comments on adeos list about the performance
problems which you have to solve before upgrading to 2.6.23. Any news ?
Could you tell me if the above numbers shows the perfomance problem or
if I have to do another kind of test to reproduce it ?
Regards,
Juan Antonio
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-23 6:52 [Xenomai-help] AT91SAM9260 latency Juan Antonio Garcia Redondo
@ 2008-01-23 10:04 ` Gilles Chanteperdrix
2008-01-23 13:15 ` Gilles Chanteperdrix
0 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-23 10:04 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
<juan-antonio.garcia@domain.hid> wrote:
> Hi list,
>
> These are my latency results over a custom board based on AT91SAM9260.
>
> Environment:
> kernel 2.6.20 + xenomai 2.4.0
> Load: dd if=/dev/zero of=/dev/null plus an external ping -f to
> stress the ethernet.
>
> The results:
> running: ./run -- -p 500 -T 60 -t0 # latency
>
> == Sampling period: 500 us
> == Test mode: periodic user-mode task
> == All results in microseconds
> warming up...
> RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
> RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
> RTD| 45.747| 62.177| 78.286| 0| 45.747| 78.286
> .............................................................................
> RTD| 39.948| 61.211| 78.286| 0| 39.304| 88.273
> ---|------------|------------|------------|--------|-------------------------
> RTS| 39.304| 61.855| 88.273| 0| 00:01:00/00:01:00
>
> Thu Mar 30 01:51:36 UTC 2000
> running: ./run -- -p 500 -T 60 -t1 # latency
> == Sampling period: 500 us
> == Test mode: in-kernel periodic task
> == All results in microseconds
> warming up...
> RTT| 00:00:01 (in-kernel periodic task, 500 us period, priority 99)
> RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
> RTD| 19.007| 31.731| 43.813| 0| 19.007| 43.813
> .............................................................................
> ---|------------|------------|------------|--------|-------------------------
> RTS| 13.501| 31.126| 71.188| 0| 00:01:00/00:01:00
>
> Thu Mar 30 01:52:52 UTC 2000
> running: ./run -- -p 500 -T 60 -t2 # latency
> *
> *
> * Type ^C to stop this application.
> *
> *
> == Sampling period: 500 us
> == Test mode: in-kernel timer handler
> == All results in microseconds
> warming up...
> RTT| 00:00:01 (in-kernel timer handler, 500 us period, priority 99)
> RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
> .............................................................................
> ---|------------|------------|------------|--------|-------------------------
> RTS| -0.977| 8.521| 28.337| 0| 00:01:00/00:01:00
>
> Thu Mar 30 01:54:07 UTC 2000
> running: ./run -- -T 120 # switchtest
> == Testing FPU check routines...
> == FPU check routines: unimplemented, skipping FPU switches tests.
> == Threads: sleeper-0 rtk-1 rtk-2 rtup-3 rtup-4 rtus-5 rtus-6 rtuo-7 rtuo-8
> RTT| 00:00:01
> RTH|ctx switches|-------total
> RTD| 756| 756
> ..............................
> RTD| 723| 87717
> RTD| 762| 88479
> RTD| 750| 89229
> RTD| 42| 89271
>
> Thu Mar 30 01:56:22 UTC 2000
> running: ./run -- -p500 -h # switchbench
> == Sampling period: 500 us
> == Do not interrupt this program
> RTH| lat min| lat avg| lat max| lost
> RTD| 47.358| 52.512| 68.298| 0
> ---|---range-|---samples
> HSD| 47 - 48 | 1
> HSD| 48 - 49 | 5
> HSD| 49 - 50 | 6
> HSD| 50 - 51 | 2065
> HSD| 51 - 52 | 22440
> HSD| 52 - 53 | 41582
> HSD| 53 - 54 | 19749
> HSD| 54 - 55 | 11696
> HSD| 55 - 56 | 2131
> HSD| 56 - 57 | 1
> HSD| 60 - 61 | 1
> HSD| 62 - 63 | 1
> HSD| 63 - 64 | 21
> HSD| 64 - 65 | 95
> HSD| 65 - 66 | 107
> HSD| 66 - 67 | 63
> HSD| 67 - 68 | 29
> HSD| 68 - 69 | 3
> HSS| 99996| 52.271| 1.278
>
> Thu Mar 30 01:57:25 UTC 2000
> running: ./run -- -p 10 -n -l 1000 # cyclictest
>
> T: 0 ( 0) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
> [3A1.30 1.01 0.63 2/33 2107
>
> ...........................................................................................
>
> T: 0 ( 2107) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
> [3A1.30 1.01 0.63 2/33 2107
>
> T: 0 ( 2107) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
> [3A1.30 1.01 0.63 2/33 2107
>
> T: 0 ( 2107) P:10 I: 1000 C: 0 Min: 1000000 Act: 0 Avg: 0 Max:-1000000
> [3A1.30 1.01 0.63 2/33 2107
>
> T: 0 ( 2107) P:10 I: 1000 C: 19 Min: 45 Act: 45 Avg: 69 Max: 78
> [3A1.30 1.01 0.63 2/33 2107
>
> ...........................................................................................
>
> T: 0 ( 2107) P:10 I: 1000 C: 999 Min: 39 Act: 48 Avg: 65 Max: 84
>
> I see everything OK except for the first samples of cyclictests. Any comments ?
The load you apply does not load the cache, which is a source of
jitter. You should run the cache calibrator (I do not find the cache
calibrator URL, but it is somewhere in Xenomai distribution or wiki).
>
> Gilles, I've read your comments on adeos list about the performance
> problems which you have to solve before upgrading to 2.6.23. Any news ?
> Could you tell me if the above numbers shows the perfomance problem or
> if I have to do another kind of test to reproduce it ?
I gave news about this issue on xenomai-core mailing list.
Unfortunately, there is no spurious masking section, the big masking
section is the one which takes place around Xenomai user-space context
switches. So, we can only improve interrupt latency, the only way to
improve scheduling latency is to avoid user-space real-time.
>
> Regards,
> Juan Antonio
>
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
>
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-23 10:04 ` Gilles Chanteperdrix
@ 2008-01-23 13:15 ` Gilles Chanteperdrix
2008-01-24 9:41 ` Juan Antonio Garcia Redondo
0 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-23 13:15 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
<gilles.chanteperdrix@xenomai.org> wrote:
> On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
>
> <juan-antonio.garcia@domain.hid> wrote:
> > I see everything OK except for the first samples of cyclictests. Any comments ?
>
> The load you apply does not load the cache, which is a source of
> jitter. You should run the cache calibrator (I do not find the cache
> calibrator URL, but it is somewhere in Xenomai distribution or wiki).
It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-23 13:15 ` Gilles Chanteperdrix
@ 2008-01-24 9:41 ` Juan Antonio Garcia Redondo
2008-01-24 10:02 ` Gilles Chanteperdrix
2008-02-10 20:31 ` [Xenomai-core] " Gilles Chanteperdrix
0 siblings, 2 replies; 27+ messages in thread
From: Juan Antonio Garcia Redondo @ 2008-01-24 9:41 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
On 23/01/08 14:15, Gilles Chanteperdrix wrote:
> On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org> wrote:
> > On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
> >
> > <juan-antonio.garcia@domain.hid> wrote:
> > > I see everything OK except for the first samples of cyclictests. Any comments ?
> >
> > The load you apply does not load the cache, which is a source of
> > jitter. You should run the cache calibrator (I do not find the cache
> > calibrator URL, but it is somewhere in Xenomai distribution or wiki).
>
> It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
>
> --
> Gilles Chanteperdrix
Thanks Gilles, I've done more tests using the cache calibrator from
http://www.cwi.nl/~manegold/Calibrator. The latency numbers are very
similar althought I've found an strange behaviour related to telnet
sessions.
Environment:
o Tests running from console over atmel serial port.
o A telnet session over on-chip ethernet.
o System without load.
./latency -p 500 -t0
== All results in microseconds
warming up...
RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
RTH|-RTH----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat
worst
RTD| 49.613| 52.190| 62.822| 0| 49.613| 62.822
RTD| 42.203| 52.512| 66.365| 0| 42.203| 66.365
Now If hit a key on the telnet session :
RTD| 36.726| 57.989| 109.536| 0| 31.572| 109.536 <-------- Here I've hit the key.
RTD| 36.404| 51.868| 69.587| 0| 31.572| 109.536
RTD| 35.760| 51.868| 73.775| 0| 31.572| 109.536
Now, I launch an script which executes four instances of cache
calibrator.
RTD| 45.103| 57.667| 75.708| 0| 32.538| 122.422
RTD| 45.425| 57.023| 76.030| 0| 32.538| 122.422
RTD| 46.069| 57.023| 75.708| 0| 32.538| 122.422
Now, I can hit a key on the telnet session without effects over latency
numbers:
RTD| 44.136| 57.989| 75.386| 0| 27.384| 128.221
RTD| 46.713| 57.345| 76.353| 0| 27.384| 128.221
RTD| 44.780| 57.345| 76.675| 0| 27.384| 128.221
RTD| 43.492| 56.701| 76.997| 0| 27.384| 128.221
Now I stop the calibrator process and launch 'ping -f -s2048 192.168.2.82' from an external
machine.
RTD| 40.270| 68.621| 90.850| 0| 27.384| 128.221
RTD| 36.082| 68.621| 88.273| 0| 27.384| 128.221
RTD| 40.592| 67.976| 91.494| 0| 27.384| 128.221
RTD| 41.237| 68.298| 89.239| 0| 27.384| 128.221
Now If hit a key on the telnet session :
RTD| 42.203| 67.976| 88.273| 0| 27.384| 128.221
RTD| 32.216| 93.427| 128.543| 0| 27.384| 128.543 <---------- Here I've hit the key.
RTD| 42.203| 68.298| 87.628| 0| 27.384| 128.543
And again the calibrator execution results on eliminate the strange
behaviour whith the telnet session.
Any clues ?
BTW, if finally the bad numbers on ARM are user-context switches related,
are you considering the ipipe upgrading to 2.6.23 ?
Regards,
Juan Antonio
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-24 9:41 ` Juan Antonio Garcia Redondo
@ 2008-01-24 10:02 ` Gilles Chanteperdrix
2008-01-25 10:04 ` Juan Antonio Garcia Redondo
2008-02-10 20:31 ` [Xenomai-core] " Gilles Chanteperdrix
1 sibling, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-24 10:02 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
On Jan 24, 2008 10:41 AM, Juan Antonio Garcia Redondo
<juan-antonio.garcia@domain.hid> wrote:
>
> On 23/01/08 14:15, Gilles Chanteperdrix wrote:
> > On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
> > <gilles.chanteperdrix@xenomai.org> wrote:
> > > On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
> > >
> > > <juan-antonio.garcia@domain.hid> wrote:
> > > > I see everything OK except for the first samples of cyclictests. Any comments ?
> > >
> > > The load you apply does not load the cache, which is a source of
> > > jitter. You should run the cache calibrator (I do not find the cache
> > > calibrator URL, but it is somewhere in Xenomai distribution or wiki).
> >
> > It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
> >
> > --
> > Gilles Chanteperdrix
>
> Thanks Gilles, I've done more tests using the cache calibrator from
> http://www.cwi.nl/~manegold/Calibrator. The latency numbers are very
> similar althought I've found an strange behaviour related to telnet
> sessions.
Are you kidding ? In the first results you posted, the latency was
around 80us whereas now, you get a latency around 130us. And from I
read, you did not run the tests for long period. If you want reliable
results, you should let the test run, under load, for hours.
>
> Environment:
> o Tests running from console over atmel serial port.
> o A telnet session over on-chip ethernet.
> o System without load.
>
> ./latency -p 500 -t0
> == All results in microseconds
> warming up...
> RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
> RTH|-RTH----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat
> worst
> RTD| 49.613| 52.190| 62.822| 0| 49.613| 62.822
> RTD| 42.203| 52.512| 66.365| 0| 42.203| 66.365
>
>
> Now If hit a key on the telnet session :
>
> RTD| 36.726| 57.989| 109.536| 0| 31.572| 109.536 <-------- Here I've hit the key.
> RTD| 36.404| 51.868| 69.587| 0| 31.572| 109.536
> RTD| 35.760| 51.868| 73.775| 0| 31.572| 109.536
>
> Now, I launch an script which executes four instances of cache
> calibrator.
>
> RTD| 45.103| 57.667| 75.708| 0| 32.538| 122.422
> RTD| 45.425| 57.023| 76.030| 0| 32.538| 122.422
> RTD| 46.069| 57.023| 75.708| 0| 32.538| 122.422
>
> Now, I can hit a key on the telnet session without effects over latency
> numbers:
>
> RTD| 44.136| 57.989| 75.386| 0| 27.384| 128.221
> RTD| 46.713| 57.345| 76.353| 0| 27.384| 128.221
> RTD| 44.780| 57.345| 76.675| 0| 27.384| 128.221
> RTD| 43.492| 56.701| 76.997| 0| 27.384| 128.221
>
> Now I stop the calibrator process and launch 'ping -f -s2048 192.168.2.82' from an external
> machine.
>
> RTD| 40.270| 68.621| 90.850| 0| 27.384| 128.221
> RTD| 36.082| 68.621| 88.273| 0| 27.384| 128.221
> RTD| 40.592| 67.976| 91.494| 0| 27.384| 128.221
> RTD| 41.237| 68.298| 89.239| 0| 27.384| 128.221
>
>
> Now If hit a key on the telnet session :
>
> RTD| 42.203| 67.976| 88.273| 0| 27.384| 128.221
> RTD| 32.216| 93.427| 128.543| 0| 27.384| 128.543 <---------- Here I've hit the key.
> RTD| 42.203| 68.298| 87.628| 0| 27.384| 128.543
>
> And again the calibrator execution results on eliminate the strange
> behaviour whith the telnet session.
>
> Any clues ?
No mystery: hitting a key on a telnet session causes an interrupt
masking section of 110us, you see it as the maximum if you never
observed longer masking sections, but it is not the maximum if you
observed longer masking sections.
>
> BTW, if finally the bad numbers on ARM are user-context switches related,
> are you considering the ipipe upgrading to 2.6.23 ?
No comment. I have already answered this question.
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-24 10:02 ` Gilles Chanteperdrix
@ 2008-01-25 10:04 ` Juan Antonio Garcia Redondo
2008-01-25 17:00 ` Gilles Chanteperdrix
0 siblings, 1 reply; 27+ messages in thread
From: Juan Antonio Garcia Redondo @ 2008-01-25 10:04 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
On 24/01/08 11:02, Gilles Chanteperdrix wrote:
> On Jan 24, 2008 10:41 AM, Juan Antonio Garcia Redondo
> <juan-antonio.garcia@domain.hid> wrote:
> >
> > On 23/01/08 14:15, Gilles Chanteperdrix wrote:
> > > On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
> > > <gilles.chanteperdrix@xenomai.org> wrote:
> > > > On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
> > > >
> > > > <juan-antonio.garcia@domain.hid> wrote:
> > > > > I see everything OK except for the first samples of cyclictests. Any comments ?
> > > >
> > > > The load you apply does not load the cache, which is a source of
> > > > jitter. You should run the cache calibrator (I do not find the cache
> > > > calibrator URL, but it is somewhere in Xenomai distribution or wiki).
> > >
> > > It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
> > >
> > > --
> > > Gilles Chanteperdrix
> >
> > Thanks Gilles, I've done more tests using the cache calibrator from
> > http://www.cwi.nl/~manegold/Calibrator. The latency numbers are very
> > similar althought I've found an strange behaviour related to telnet
> > sessions.
>
> Are you kidding ? In the first results you posted, the latency was
> around 80us whereas now, you get a latency around 130us. And from I
> read, you did not run the tests for long period. If you want reliable
> results, you should let the test run, under load, for hours.
Well, after several tests, (one of them 4 hours long), I can't see
latencies above 100 us. Anyway I'll do more tests this weekend. The
latency around 130us occurs with telnet activity.
>
> >
> > Environment:
> > o Tests running from console over atmel serial port.
> > o A telnet session over on-chip ethernet.
> > o System without load.
> >
> > ./latency -p 500 -t0
> > == All results in microseconds
> > warming up...
> > RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
> > RTH|-RTH----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat
> > worst
> > RTD| 49.613| 52.190| 62.822| 0| 49.613| 62.822
> > RTD| 42.203| 52.512| 66.365| 0| 42.203| 66.365
> >
> >
> > Now If hit a key on the telnet session :
> >
> > RTD| 36.726| 57.989| 109.536| 0| 31.572| 109.536 <-------- Here I've hit the key.
> > RTD| 36.404| 51.868| 69.587| 0| 31.572| 109.536
> > RTD| 35.760| 51.868| 73.775| 0| 31.572| 109.536
> >
> > Now, I launch an script which executes four instances of cache
> > calibrator.
> >
> > RTD| 45.103| 57.667| 75.708| 0| 32.538| 122.422
> > RTD| 45.425| 57.023| 76.030| 0| 32.538| 122.422
> > RTD| 46.069| 57.023| 75.708| 0| 32.538| 122.422
> >
> > Now, I can hit a key on the telnet session without effects over latency
> > numbers:
> >
> > RTD| 44.136| 57.989| 75.386| 0| 27.384| 128.221
> > RTD| 46.713| 57.345| 76.353| 0| 27.384| 128.221
> > RTD| 44.780| 57.345| 76.675| 0| 27.384| 128.221
> > RTD| 43.492| 56.701| 76.997| 0| 27.384| 128.221
> >
> > Now I stop the calibrator process and launch 'ping -f -s2048 192.168.2.82' from an external
> > machine.
> >
> > RTD| 40.270| 68.621| 90.850| 0| 27.384| 128.221
> > RTD| 36.082| 68.621| 88.273| 0| 27.384| 128.221
> > RTD| 40.592| 67.976| 91.494| 0| 27.384| 128.221
> > RTD| 41.237| 68.298| 89.239| 0| 27.384| 128.221
> >
> >
> > Now If hit a key on the telnet session :
> >
> > RTD| 42.203| 67.976| 88.273| 0| 27.384| 128.221
> > RTD| 32.216| 93.427| 128.543| 0| 27.384| 128.543 <---------- Here I've hit the key.
> > RTD| 42.203| 68.298| 87.628| 0| 27.384| 128.543
> >
> > And again the calibrator execution results on eliminate the strange
> > behaviour whith the telnet session.
> >
> > Any clues ?
>
> No mystery: hitting a key on a telnet session causes an interrupt
> masking section of 110us, you see it as the maximum if you never
> observed longer masking sections, but it is not the maximum if you
> observed longer masking sections.
OK, but why the masking section on linux side affects to xenomai side ?
Another thing I don't understand is why when the system has load (above
I'm talking about calibrator but the same occurs with dd if=/dev/zero
of=/dev/null), the effect seems to dissapear.
>
> >
> > BTW, if finally the bad numbers on ARM are user-context switches related,
> > are you considering the ipipe upgrading to 2.6.23 ?
>
> No comment. I have already answered this question.
Sorry, I missed the last entries on "High latencies on ARM" from
xenomai-core list.
Regards,
Juan Antonio
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-25 10:04 ` Juan Antonio Garcia Redondo
@ 2008-01-25 17:00 ` Gilles Chanteperdrix
2008-01-28 8:51 ` Juan Antonio Garcia Redondo
0 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-25 17:00 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
On Jan 25, 2008 11:04 AM, Juan Antonio Garcia Redondo
<juan-antonio.garcia@domain.hid> wrote:
> On 24/01/08 11:02, Gilles Chanteperdrix wrote:
> Well, after several tests, (one of them 4 hours long), I can't see
> latencies above 100 us. Anyway I'll do more tests this weekend. The
> latency around 130us occurs with telnet activity.
This contradicts what you told us in previous posts. You told us that
the maximum latency went up to 130us when starting the calibrator.
> > No mystery: hitting a key on a telnet session causes an interrupt
> > masking section of 110us, you see it as the maximum if you never
> > observed longer masking sections, but it is not the maximum if you
> > observed longer masking sections.
>
> OK, but why the masking section on linux side affects to xenomai side ?
> Another thing I don't understand is why when the system has load (above
> I'm talking about calibrator but the same occurs with dd if=/dev/zero
> of=/dev/null), the effect seems to dissapear.
It is probably not a masking section on linux side but rather a
masking section on I-pipe side. Anyway, the effect does not disappear:
it means that the cache effects cause larger latencies than the
ethernet interrupt, but maybe I did not understand what you explained.
The results you obtain with no load are simply irrelevant.
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-25 17:00 ` Gilles Chanteperdrix
@ 2008-01-28 8:51 ` Juan Antonio Garcia Redondo
2008-01-28 9:21 ` Juan Antonio Garcia Redondo
2008-01-28 13:19 ` Gilles Chanteperdrix
0 siblings, 2 replies; 27+ messages in thread
From: Juan Antonio Garcia Redondo @ 2008-01-28 8:51 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
On 25/01/08 18:00, Gilles Chanteperdrix wrote:
> On Jan 25, 2008 11:04 AM, Juan Antonio Garcia Redondo
> <juan-antonio.garcia@domain.hid> wrote:
> > On 24/01/08 11:02, Gilles Chanteperdrix wrote:
> > Well, after several tests, (one of them 4 hours long), I can't see
> > latencies above 100 us. Anyway I'll do more tests this weekend. The
> > latency around 130us occurs with telnet activity.
>
> This contradicts what you told us in previous posts. You told us that
> the maximum latency went up to 130us when starting the calibrator.
I guess that there has been a misunderstanding. In the mail
https://mail.gna.org/public/xenomai-help/2008-01/msg00108.html I
wanted to point to the fact that the telnet activity had a direct and
reproducible efect on lat_max field. So my test was:
1) From a minicom terminal run 'latency -t0 -p500'
2) From a telnet terminal launch calibrator, kill calibrator etc.
So, the lat_worst numbers I showed in that email has no value. I
launched the calibrator process during several minutes, and, while I
was doing the tests I saw the telnet behaviour and I thought that could be
interesting to report it.
I've done more tests this weekend. I've switched to at91sam9260_ek
development plattform because I can hold it doing xenomai tests without affect
my current work.
My numbers:
o Test xenomai-2.4.0 + 5 * (dd if=/dev/sda of=/dev/null) + 5 * (calibrator 180
4M calibra) + ping -f -s2048
./latency -p 500 -t0
RTD| 47.358| 67.976| 93.105| 0| 28.672| 122.422
RTD| 47.680| 68.298| 91.817| 0| 28.672| 122.422
RTD| 49.291| 68.298| 91.494| 0| 28.672| 122.422
RTD| 45.747| 68.621| 90.528| 0| 28.672| 122.422
RTD| 47.358| 68.621| 90.206| 0| 28.672| 122.422
RTD| 47.358| 68.621| 91.172| 0| 28.672| 122.422
RTD| 37.048| 68.621| 92.139| 0| 28.672| 122.422
RTT| 07:35:22 (periodic user-mode task, 500 us period, priority 99)
>
>
> > > No mystery: hitting a key on a telnet session causes an interrupt
> > > masking section of 110us, you see it as the maximum if you never
> > > observed longer masking sections, but it is not the maximum if you
> > > observed longer masking sections.
> >
> > OK, but why the masking section on linux side affects to xenomai side ?
> > Another thing I don't understand is why when the system has load (above
> > I'm talking about calibrator but the same occurs with dd if=/dev/zero
> > of=/dev/null), the effect seems to dissapear.
>
> It is probably not a masking section on linux side but rather a
> masking section on I-pipe side. Anyway, the effect does not disappear:
> it means that the cache effects cause larger latencies than the
> ethernet interrupt, but maybe I did not understand what you explained.
> The results you obtain with no load are simply irrelevant.
I'll try to explain it better:
o Without load I run ./latency -t0 -p500.
RTD| 33.182| 53.479| 67.976| 0| 31.250| 77.319
RTD| 43.170| 53.479| 67.654| 0| 31.250| 77.319
RTD| 41.881| 53.479| 67.332| 0| 31.250| 77.319
RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
o Each time I press a key (over a telnet session) I can see the lat_max field increase on 40 to 50 us aprox.
RTD| 33.505| 53.479| 71.842| 0| 26.739| 77.319
RTD| 40.592| 62.177| 123.067| 0| 26.739| 123.067
-------
\_________: Key pressed
RTD| 50.579| 53.479| 73.775| 0| 26.739| 123.067
o Stop the latency test.
o run dd if=/dev/zero of=/dev/null
o run ./latency -t0 -p 500
RTD| 44.780| 55.734| 89.884| 0| 36.082| 93.105
RTD| 45.425| 55.412| 89.561| 0| 36.082| 93.105
RTD| 44.458| 55.734| 90.206| 0| 36.082| 93.105
RTD| 45.103| 55.412| 90.206| 0| 36.082| 93.105
RTD| 45.425| 55.734| 88.273| 0| 36.082| 93.105
RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
o I can't see any effect on lat_max field when I press a key on the
telnet session.
RTD| 44.136| 55.734| 92.461| 0| 36.082| 94.394
-------
\_________: Key pressed
RTD| 43.814| 55.734| 90.528| 0| 36.082| 94.394
-------
\_________: Key pressed
RTD| 45.103| 55.412| 89.239| 0| 36.082| 94.394
-------
\_________: Key pressed
RTT| 00:03:31 (periodic user-mode task, 500 us period, priority 99)
As you can see the lat_max numbers are under the 100 us while I can go
up 120 us easily whith the former test.
Regards,
Juan Antonio
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 8:51 ` Juan Antonio Garcia Redondo
@ 2008-01-28 9:21 ` Juan Antonio Garcia Redondo
2008-01-28 13:19 ` Gilles Chanteperdrix
1 sibling, 0 replies; 27+ messages in thread
From: Juan Antonio Garcia Redondo @ 2008-01-28 9:21 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
On 28/01/08 09:51, Juan Antonio Garcia Redondo wrote:
> On 25/01/08 18:00, Gilles Chanteperdrix wrote:
> > On Jan 25, 2008 11:04 AM, Juan Antonio Garcia Redondo
> > <juan-antonio.garcia@domain.hid> wrote:
> > > On 24/01/08 11:02, Gilles Chanteperdrix wrote:
> > > Well, after several tests, (one of them 4 hours long), I can't see
> > > latencies above 100 us. Anyway I'll do more tests this weekend. The
> > > latency around 130us occurs with telnet activity.
> >
> > This contradicts what you told us in previous posts. You told us that
> > the maximum latency went up to 130us when starting the calibrator.
>
> I guess that there has been a misunderstanding. In the mail
> https://mail.gna.org/public/xenomai-help/2008-01/msg00108.html I
> wanted to point to the fact that the telnet activity had a direct and
> reproducible efect on lat_max field. So my test was:
> 1) From a minicom terminal run 'latency -t0 -p500'
> 2) From a telnet terminal launch calibrator, kill calibrator etc.
>
> So, the lat_worst numbers I showed in that email has no value. I
> launched the calibrator process during several minutes, and, while I
> was doing the tests I saw the telnet behaviour and I thought that could be
> interesting to report it.
>
> I've done more tests this weekend. I've switched to at91sam9260_ek
> development plattform because I can hold it doing xenomai tests without affect
> my current work.
>
> My numbers:
>
> o Test xenomai-2.4.0 + 5 * (dd if=/dev/sda of=/dev/null) + 5 * (calibrator 180
> 4M calibra) + ping -f -s2048
>
> ./latency -p 500 -t0
>
> RTD| 47.358| 67.976| 93.105| 0| 28.672| 122.422
> RTD| 47.680| 68.298| 91.817| 0| 28.672| 122.422
> RTD| 49.291| 68.298| 91.494| 0| 28.672| 122.422
> RTD| 45.747| 68.621| 90.528| 0| 28.672| 122.422
> RTD| 47.358| 68.621| 90.206| 0| 28.672| 122.422
> RTD| 47.358| 68.621| 91.172| 0| 28.672| 122.422
> RTD| 37.048| 68.621| 92.139| 0| 28.672| 122.422
> RTT| 07:35:22 (periodic user-mode task, 500 us period, priority 99)
>
> >
> >
> > > > No mystery: hitting a key on a telnet session causes an interrupt
> > > > masking section of 110us, you see it as the maximum if you never
> > > > observed longer masking sections, but it is not the maximum if you
> > > > observed longer masking sections.
> > >
> > > OK, but why the masking section on linux side affects to xenomai side ?
> > > Another thing I don't understand is why when the system has load (above
> > > I'm talking about calibrator but the same occurs with dd if=/dev/zero
> > > of=/dev/null), the effect seems to dissapear.
> >
> > It is probably not a masking section on linux side but rather a
> > masking section on I-pipe side. Anyway, the effect does not disappear:
> > it means that the cache effects cause larger latencies than the
> > ethernet interrupt, but maybe I did not understand what you explained.
> > The results you obtain with no load are simply irrelevant.
>
> I'll try to explain it better:
>
> o Without load I run ./latency -t0 -p500.
> RTD| 33.182| 53.479| 67.976| 0| 31.250| 77.319
> RTD| 43.170| 53.479| 67.654| 0| 31.250| 77.319
> RTD| 41.881| 53.479| 67.332| 0| 31.250| 77.319
> RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
>
> o Each time I press a key (over a telnet session) I can see the lat_max field increase on 40 to 50 us aprox.
> RTD| 33.505| 53.479| 71.842| 0| 26.739| 77.319
> RTD| 40.592| 62.177| 123.067| 0| 26.739| 123.067
> -------
> \_________: Key pressed
> RTD| 50.579| 53.479| 73.775| 0| 26.739| 123.067
>
>
> o Stop the latency test.
> o run dd if=/dev/zero of=/dev/null
> o run ./latency -t0 -p 500
> RTD| 44.780| 55.734| 89.884| 0| 36.082| 93.105
> RTD| 45.425| 55.412| 89.561| 0| 36.082| 93.105
> RTD| 44.458| 55.734| 90.206| 0| 36.082| 93.105
> RTD| 45.103| 55.412| 90.206| 0| 36.082| 93.105
> RTD| 45.425| 55.734| 88.273| 0| 36.082| 93.105
> RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
>
> o I can't see any effect on lat_max field when I press a key on the
> telnet session.
>
> RTD| 44.136| 55.734| 92.461| 0| 36.082| 94.394
> -------
> \_________: Key pressed
> RTD| 43.814| 55.734| 90.528| 0| 36.082| 94.394
> -------
> \_________: Key pressed
> RTD| 45.103| 55.412| 89.239| 0| 36.082| 94.394
> -------
> \_________: Key pressed
> RTT| 00:03:31 (periodic user-mode task, 500 us period, priority 99)
>
> As you can see the lat_max numbers are under the 100 us while I can go
> up 120 us easily whith the former test.
>
Just to say that the test were done with xenomai-2.4.1 (ipipe 1.8-03).
Regards,
Juan Antonio
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 8:51 ` Juan Antonio Garcia Redondo
2008-01-28 9:21 ` Juan Antonio Garcia Redondo
@ 2008-01-28 13:19 ` Gilles Chanteperdrix
2008-01-28 13:34 ` Jan Kiszka
2008-01-29 8:09 ` Juan Antonio Garcia Redondo
1 sibling, 2 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-28 13:19 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
On Jan 28, 2008 9:51 AM, Juan Antonio Garcia Redondo
<juan-antonio.garcia@domain.hid> wrote:
> On 25/01/08 18:00, Gilles Chanteperdrix wrote:
> > On Jan 25, 2008 11:04 AM, Juan Antonio Garcia Redondo
> > <juan-antonio.garcia@domain.hid> wrote:
> > > On 24/01/08 11:02, Gilles Chanteperdrix wrote:
> > > Well, after several tests, (one of them 4 hours long), I can't see
> > > latencies above 100 us. Anyway I'll do more tests this weekend. The
> > > latency around 130us occurs with telnet activity.
> >
> > This contradicts what you told us in previous posts. You told us that
> > the maximum latency went up to 130us when starting the calibrator.
>
> I guess that there has been a misunderstanding. In the mail
> https://mail.gna.org/public/xenomai-help/2008-01/msg00108.html I
> wanted to point to the fact that the telnet activity had a direct and
> reproducible efect on lat_max field. So my test was:
> 1) From a minicom terminal run 'latency -t0 -p500'
> 2) From a telnet terminal launch calibrator, kill calibrator etc.
>
> So, the lat_worst numbers I showed in that email has no value. I
> launched the calibrator process during several minutes, and, while I
> was doing the tests I saw the telnet behaviour and I thought that could be
> interesting to report it.
>
> I've done more tests this weekend. I've switched to at91sam9260_ek
> development plattform because I can hold it doing xenomai tests without affect
> my current work.
>
> My numbers:
>
> o Test xenomai-2.4.0 + 5 * (dd if=/dev/sda of=/dev/null) + 5 * (calibrator 180
> 4M calibra) + ping -f -s2048
>
> ./latency -p 500 -t0
>
> RTD| 47.358| 67.976| 93.105| 0| 28.672| 122.422
> RTD| 47.680| 68.298| 91.817| 0| 28.672| 122.422
> RTD| 49.291| 68.298| 91.494| 0| 28.672| 122.422
> RTD| 45.747| 68.621| 90.528| 0| 28.672| 122.422
> RTD| 47.358| 68.621| 90.206| 0| 28.672| 122.422
> RTD| 47.358| 68.621| 91.172| 0| 28.672| 122.422
> RTD| 37.048| 68.621| 92.139| 0| 28.672| 122.422
> RTT| 07:35:22 (periodic user-mode task, 500 us period, priority 99)
Ok, so worst case latency is around 130us, as I thought I understood.
>
> >
> >
> > > > No mystery: hitting a key on a telnet session causes an interrupt
> > > > masking section of 110us, you see it as the maximum if you never
> > > > observed longer masking sections, but it is not the maximum if you
> > > > observed longer masking sections.
> > >
> > > OK, but why the masking section on linux side affects to xenomai side ?
> > > Another thing I don't understand is why when the system has load (above
> > > I'm talking about calibrator but the same occurs with dd if=/dev/zero
> > > of=/dev/null), the effect seems to dissapear.
> >
> > It is probably not a masking section on linux side but rather a
> > masking section on I-pipe side. Anyway, the effect does not disappear:
> > it means that the cache effects cause larger latencies than the
> > ethernet interrupt, but maybe I did not understand what you explained.
> > The results you obtain with no load are simply irrelevant.
>
> I'll try to explain it better:
>
> o Without load I run ./latency -t0 -p500.
> RTD| 33.182| 53.479| 67.976| 0| 31.250| 77.319
> RTD| 43.170| 53.479| 67.654| 0| 31.250| 77.319
> RTD| 41.881| 53.479| 67.332| 0| 31.250| 77.319
> RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
>
> o Each time I press a key (over a telnet session) I can see the lat_max field increase on 40 to 50 us aprox.
> RTD| 33.505| 53.479| 71.842| 0| 26.739| 77.319
> RTD| 40.592| 62.177| 123.067| 0| 26.739| 123.067
> -------
> \_________: Key pressed
> RTD| 50.579| 53.479| 73.775| 0| 26.739| 123.067
This is where you are wrong:
- first, let me repeat it: test made without load are irrelevant;
- second, an event has no relative effect on max latency, its effect
is absolute: pressing a key over a telnet session causes, for unknown
reason, a masking section of around 130us, which happens to also be
the worst case latency that we measured properly with a loaded system.
Now, if you want to know why you get such a masking section, you are
free to investigate.
>
>
> o Stop the latency test.
> o run dd if=/dev/zero of=/dev/null
> o run ./latency -t0 -p 500
> RTD| 44.780| 55.734| 89.884| 0| 36.082| 93.105
> RTD| 45.425| 55.412| 89.561| 0| 36.082| 93.105
> RTD| 44.458| 55.734| 90.206| 0| 36.082| 93.105
> RTD| 45.103| 55.412| 90.206| 0| 36.082| 93.105
> RTD| 45.425| 55.734| 88.273| 0| 36.082| 93.105
> RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
>
> o I can't see any effect on lat_max field when I press a key on the
> telnet session.
>
> RTD| 44.136| 55.734| 92.461| 0| 36.082| 94.394
> -------
> \_________: Key pressed
> RTD| 43.814| 55.734| 90.528| 0| 36.082| 94.394
> -------
> \_________: Key pressed
> RTD| 45.103| 55.412| 89.239| 0| 36.082| 94.394
> -------
> \_________: Key pressed
> RTT| 00:03:31 (periodic user-mode task, 500 us period, priority 99)
>
> As you can see the lat_max numbers are under the 100 us while I can go
> up 120 us easily whith the former test.
The worst case latency of your system is above 120us, so, the results
you get when not running the calibrator are not really significant.
The behaviour you get may be due, for instance, to the fact that the
processor goes into some sleep mode when idle and to a wake-up
latency; if you run some load, there is no wake-up latency. It is hard
to say anything. In order to investigate, I would instrument the
kernel to trace the irq masking sections.
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 13:19 ` Gilles Chanteperdrix
@ 2008-01-28 13:34 ` Jan Kiszka
2008-01-28 13:35 ` Gilles Chanteperdrix
2008-01-29 8:09 ` Juan Antonio Garcia Redondo
1 sibling, 1 reply; 27+ messages in thread
From: Jan Kiszka @ 2008-01-28 13:34 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
Gilles Chanteperdrix wrote:
> ...
> The behaviour you get may be due, for instance, to the fact that the
> processor goes into some sleep mode when idle and to a wake-up
> latency; if you run some load, there is no wake-up latency. It is hard
> to say anything. In order to investigate, I would instrument the
> kernel to trace the irq masking sections.
Don't we have the I-pipe tracer on this platform?
Jan
--
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 13:34 ` Jan Kiszka
@ 2008-01-28 13:35 ` Gilles Chanteperdrix
2008-01-28 13:46 ` Jan Kiszka
0 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-28 13:35 UTC (permalink / raw)
To: Jan Kiszka; +Cc: jagarcia, xenomai
On Jan 28, 2008 2:34 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> Gilles Chanteperdrix wrote:
> > ...
> > The behaviour you get may be due, for instance, to the fact that the
> > processor goes into some sleep mode when idle and to a wake-up
> > latency; if you run some load, there is no wake-up latency. It is hard
> > to say anything. In order to investigate, I would instrument the
> > kernel to trace the irq masking sections.
>
> Don't we have the I-pipe tracer on this platform?
Unfortunately, the tracer on arm has too much overhead to give
meaningful results.
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 13:35 ` Gilles Chanteperdrix
@ 2008-01-28 13:46 ` Jan Kiszka
2008-01-28 13:51 ` Gilles Chanteperdrix
0 siblings, 1 reply; 27+ messages in thread
From: Jan Kiszka @ 2008-01-28 13:46 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
Gilles Chanteperdrix wrote:
> On Jan 28, 2008 2:34 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>> Gilles Chanteperdrix wrote:
>>> ...
>>> The behaviour you get may be due, for instance, to the fact that the
>>> processor goes into some sleep mode when idle and to a wake-up
>>> latency; if you run some load, there is no wake-up latency. It is hard
>>> to say anything. In order to investigate, I would instrument the
>>> kernel to trace the irq masking sections.
>> Don't we have the I-pipe tracer on this platform?
>
> Unfortunately, the tracer on arm has too much overhead to give
> meaningful results.
Hmm. Unless the system becomes unusable while running the tracer, I
don't see the show-stopper yet. We are looking for an increase of the
latency when triggering a certain event. That should remain measurable
even if the base latency is far higher than usually.
Do you know what makes the tracer so slow? My "lowest-end" experience
with it are based on a 133 MHz Pentium where the tracer doubled the
worst-case latencies, but the results remained useful for spotting long
paths.
Jan
--
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 13:46 ` Jan Kiszka
@ 2008-01-28 13:51 ` Gilles Chanteperdrix
2008-01-28 14:10 ` Jan Kiszka
0 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-28 13:51 UTC (permalink / raw)
To: Jan Kiszka; +Cc: jagarcia, xenomai
On Jan 28, 2008 2:46 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> Gilles Chanteperdrix wrote:
> > On Jan 28, 2008 2:34 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> >> Gilles Chanteperdrix wrote:
> >>> ...
> >>> The behaviour you get may be due, for instance, to the fact that the
> >>> processor goes into some sleep mode when idle and to a wake-up
> >>> latency; if you run some load, there is no wake-up latency. It is hard
> >>> to say anything. In order to investigate, I would instrument the
> >>> kernel to trace the irq masking sections.
> >> Don't we have the I-pipe tracer on this platform?
> >
> > Unfortunately, the tracer on arm has too much overhead to give
> > meaningful results.
>
> Hmm. Unless the system becomes unusable while running the tracer, I
> don't see the show-stopper yet. We are looking for an increase of the
> latency when triggering a certain event. That should remain measurable
> even if the base latency is far higher than usually.
>
> Do you know what makes the tracer so slow?
>
I suspected the fact that we disabled and enabled hardware irqs in
ipipe_get_tsc, but I reimplemented ipipe_get_tsc to not shut
interrupts off and the overhead of the tracer remains high.
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 13:51 ` Gilles Chanteperdrix
@ 2008-01-28 14:10 ` Jan Kiszka
0 siblings, 0 replies; 27+ messages in thread
From: Jan Kiszka @ 2008-01-28 14:10 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
Gilles Chanteperdrix wrote:
> On Jan 28, 2008 2:46 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>> Gilles Chanteperdrix wrote:
>>> On Jan 28, 2008 2:34 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> ...
>>>>> The behaviour you get may be due, for instance, to the fact that the
>>>>> processor goes into some sleep mode when idle and to a wake-up
>>>>> latency; if you run some load, there is no wake-up latency. It is hard
>>>>> to say anything. In order to investigate, I would instrument the
>>>>> kernel to trace the irq masking sections.
>>>> Don't we have the I-pipe tracer on this platform?
>>> Unfortunately, the tracer on arm has too much overhead to give
>>> meaningful results.
>> Hmm. Unless the system becomes unusable while running the tracer, I
>> don't see the show-stopper yet. We are looking for an increase of the
>> latency when triggering a certain event. That should remain measurable
>> even if the base latency is far higher than usually.
>>
>> Do you know what makes the tracer so slow?
>>
>
> I suspected the fact that we disabled and enabled hardware irqs in
> ipipe_get_tsc, but I reimplemented ipipe_get_tsc to not shut
> interrupts off and the overhead of the tracer remains high.
The tracer itself is built upon atomicity through irq disabling, so the
issue is broader - if only playing with irq flags is the problem.
Are there any trivial functions needlessly instrumented that cause too
much overhead? Maybe spending some more "notrace" would help then.
Jan
--
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-28 13:19 ` Gilles Chanteperdrix
2008-01-28 13:34 ` Jan Kiszka
@ 2008-01-29 8:09 ` Juan Antonio Garcia Redondo
2008-01-29 8:35 ` Gilles Chanteperdrix
2008-01-29 17:19 ` Gilles Chanteperdrix
1 sibling, 2 replies; 27+ messages in thread
From: Juan Antonio Garcia Redondo @ 2008-01-29 8:09 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
On 28/01/08 14:19, Gilles Chanteperdrix wrote:
> > >
> > >
> > > > > No mystery: hitting a key on a telnet session causes an interrupt
> > > > > masking section of 110us, you see it as the maximum if you never
> > > > > observed longer masking sections, but it is not the maximum if you
> > > > > observed longer masking sections.
> > > >
> > > > OK, but why the masking section on linux side affects to xenomai side ?
> > > > Another thing I don't understand is why when the system has load (above
> > > > I'm talking about calibrator but the same occurs with dd if=/dev/zero
> > > > of=/dev/null), the effect seems to dissapear.
> > >
> > > It is probably not a masking section on linux side but rather a
> > > masking section on I-pipe side. Anyway, the effect does not disappear:
> > > it means that the cache effects cause larger latencies than the
> > > ethernet interrupt, but maybe I did not understand what you explained.
> > > The results you obtain with no load are simply irrelevant.
> >
> > I'll try to explain it better:
> >
> > o Without load I run ./latency -t0 -p500.
> > RTD| 33.182| 53.479| 67.976| 0| 31.250| 77.319
> > RTD| 43.170| 53.479| 67.654| 0| 31.250| 77.319
> > RTD| 41.881| 53.479| 67.332| 0| 31.250| 77.319
> > RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
> >
> > o Each time I press a key (over a telnet session) I can see the lat_max field increase on 40 to 50 us aprox.
> > RTD| 33.505| 53.479| 71.842| 0| 26.739| 77.319
> > RTD| 40.592| 62.177| 123.067| 0| 26.739| 123.067
> > -------
> > \_________: Key pressed
> > RTD| 50.579| 53.479| 73.775| 0| 26.739| 123.067
>
> This is where you are wrong:
> - first, let me repeat it: test made without load are irrelevant;
I can't agree with you. When we stress a system with load is, as far as
I know, because usually, the large latencies don't appear on a quiet
system. Here we have a case where a large latency (the lat_worst number
I've gotten after more than 7 hours with the system fully loaded is even
less than this) appears on a quiet system and directly related to an
external event.
> - second, an event has no relative effect on max latency, its effect
> is absolute: pressing a key over a telnet session causes, for unknown
> reason, a masking section of around 130us, which happens to also be
> the worst case latency that we measured properly with a loaded system.
I pointed out the lat_max field because, if you keep the latency test
running and hit a key on a telnet session, you can easily see how the
lat_max increase each time you hit the key, while the lat_worst increase
depends on the former history.
>
> Now, if you want to know why you get such a masking section, you are
> free to investigate.
I'll try to do it.
Regards,
Juan Antonio
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-29 8:09 ` Juan Antonio Garcia Redondo
@ 2008-01-29 8:35 ` Gilles Chanteperdrix
2008-01-29 17:19 ` Gilles Chanteperdrix
1 sibling, 0 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-29 8:35 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
On Jan 29, 2008 9:09 AM, Juan Antonio Garcia Redondo
<juan-antonio.garcia@domain.hid> wrote:
>
> On 28/01/08 14:19, Gilles Chanteperdrix wrote:
> > > >
> > > >
> > > > > > No mystery: hitting a key on a telnet session causes an interrupt
> > > > > > masking section of 110us, you see it as the maximum if you never
> > > > > > observed longer masking sections, but it is not the maximum if you
> > > > > > observed longer masking sections.
> > > > >
> > > > > OK, but why the masking section on linux side affects to xenomai side ?
> > > > > Another thing I don't understand is why when the system has load (above
> > > > > I'm talking about calibrator but the same occurs with dd if=/dev/zero
> > > > > of=/dev/null), the effect seems to dissapear.
> > > >
> > > > It is probably not a masking section on linux side but rather a
> > > > masking section on I-pipe side. Anyway, the effect does not disappear:
> > > > it means that the cache effects cause larger latencies than the
> > > > ethernet interrupt, but maybe I did not understand what you explained.
> > > > The results you obtain with no load are simply irrelevant.
> > >
> > > I'll try to explain it better:
> > >
> > > o Without load I run ./latency -t0 -p500.
> > > RTD| 33.182| 53.479| 67.976| 0| 31.250| 77.319
> > > RTD| 43.170| 53.479| 67.654| 0| 31.250| 77.319
> > > RTD| 41.881| 53.479| 67.332| 0| 31.250| 77.319
> > > RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
> > >
> > > o Each time I press a key (over a telnet session) I can see the lat_max field increase on 40 to 50 us aprox.
> > > RTD| 33.505| 53.479| 71.842| 0| 26.739| 77.319
> > > RTD| 40.592| 62.177| 123.067| 0| 26.739| 123.067
> > > -------
> > > \_________: Key pressed
> > > RTD| 50.579| 53.479| 73.775| 0| 26.739| 123.067
> >
> > This is where you are wrong:
> > - first, let me repeat it: test made without load are irrelevant;
> I can't agree with you. When we stress a system with load is, as far as
> I know, because usually, the large latencies don't appear on a quiet
> system.
Wrong: they are less likely to appear, but may appear as well.
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-29 8:09 ` Juan Antonio Garcia Redondo
2008-01-29 8:35 ` Gilles Chanteperdrix
@ 2008-01-29 17:19 ` Gilles Chanteperdrix
2008-01-30 9:03 ` Juan Antonio Garcia Redondo
1 sibling, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-01-29 17:19 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
On Jan 29, 2008 9:09 AM, Juan Antonio Garcia Redondo
<juan-antonio.garcia@domain.hid> wrote:
>
> On 28/01/08 14:19, Gilles Chanteperdrix wrote:
> > > >
> > > >
> > > > > > No mystery: hitting a key on a telnet session causes an interrupt
> > > > > > masking section of 110us, you see it as the maximum if you never
> > > > > > observed longer masking sections, but it is not the maximum if you
> > > > > > observed longer masking sections.
> > > > >
> > > > > OK, but why the masking section on linux side affects to xenomai side ?
> > > > > Another thing I don't understand is why when the system has load (above
> > > > > I'm talking about calibrator but the same occurs with dd if=/dev/zero
> > > > > of=/dev/null), the effect seems to dissapear.
> > > >
> > > > It is probably not a masking section on linux side but rather a
> > > > masking section on I-pipe side. Anyway, the effect does not disappear:
> > > > it means that the cache effects cause larger latencies than the
> > > > ethernet interrupt, but maybe I did not understand what you explained.
> > > > The results you obtain with no load are simply irrelevant.
> > >
> > > I'll try to explain it better:
> > >
> > > o Without load I run ./latency -t0 -p500.
> > > RTD| 33.182| 53.479| 67.976| 0| 31.250| 77.319
> > > RTD| 43.170| 53.479| 67.654| 0| 31.250| 77.319
> > > RTD| 41.881| 53.479| 67.332| 0| 31.250| 77.319
> > > RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
> > >
> > > o Each time I press a key (over a telnet session) I can see the lat_max field increase on 40 to 50 us aprox.
> > > RTD| 33.505| 53.479| 71.842| 0| 26.739| 77.319
> > > RTD| 40.592| 62.177| 123.067| 0| 26.739| 123.067
> > > -------
> > > \_________: Key pressed
> > > RTD| 50.579| 53.479| 73.775| 0| 26.739| 123.067
> >
> > This is where you are wrong:
> > - first, let me repeat it: test made without load are irrelevant;
> I can't agree with you. When we stress a system with load is, as far as
> I know, because usually, the large latencies don't appear on a quiet
> system. Here we have a case where a large latency (the lat_worst number
> I've gotten after more than 7 hours with the system fully loaded is even
> less than this) appears on a quiet system and directly related to an
> external event.
>
> > - second, an event has no relative effect on max latency, its effect
> > is absolute: pressing a key over a telnet session causes, for unknown
> > reason, a masking section of around 130us, which happens to also be
> > the worst case latency that we measured properly with a loaded system.
> I pointed out the lat_max field because, if you keep the latency test
> running and hit a key on a telnet session, you can easily see how the
> lat_max increase each time you hit the key, while the lat_worst increase
> depends on the former history.
>
> >
> > Now, if you want to know why you get such a masking section, you are
> > free to investigate.
>
> I'll try to do it.
Does the network driver use the AT91 PDC ? If yes, and if it is
possible to disable it, could you try disabling it ?
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-help] AT91SAM9260 latency
2008-01-29 17:19 ` Gilles Chanteperdrix
@ 2008-01-30 9:03 ` Juan Antonio Garcia Redondo
0 siblings, 0 replies; 27+ messages in thread
From: Juan Antonio Garcia Redondo @ 2008-01-30 9:03 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
On 29/01/08 18:19, Gilles Chanteperdrix wrote:
> On Jan 29, 2008 9:09 AM, Juan Antonio Garcia Redondo
> <juan-antonio.garcia@domain.hid> wrote:
> >
> > On 28/01/08 14:19, Gilles Chanteperdrix wrote:
> > > > >
> > > > >
> > > > > > > No mystery: hitting a key on a telnet session causes an interrupt
> > > > > > > masking section of 110us, you see it as the maximum if you never
> > > > > > > observed longer masking sections, but it is not the maximum if you
> > > > > > > observed longer masking sections.
> > > > > >
> > > > > > OK, but why the masking section on linux side affects to xenomai side ?
> > > > > > Another thing I don't understand is why when the system has load (above
> > > > > > I'm talking about calibrator but the same occurs with dd if=/dev/zero
> > > > > > of=/dev/null), the effect seems to dissapear.
> > > > >
> > > > > It is probably not a masking section on linux side but rather a
> > > > > masking section on I-pipe side. Anyway, the effect does not disappear:
> > > > > it means that the cache effects cause larger latencies than the
> > > > > ethernet interrupt, but maybe I did not understand what you explained.
> > > > > The results you obtain with no load are simply irrelevant.
> > > >
> > > > I'll try to explain it better:
> > > >
> > > > o Without load I run ./latency -t0 -p500.
> > > > RTD| 33.182| 53.479| 67.976| 0| 31.250| 77.319
> > > > RTD| 43.170| 53.479| 67.654| 0| 31.250| 77.319
> > > > RTD| 41.881| 53.479| 67.332| 0| 31.250| 77.319
> > > > RTT| 00:02:07 (periodic user-mode task, 500 us period, priority 99)
> > > >
> > > > o Each time I press a key (over a telnet session) I can see the lat_max field increase on 40 to 50 us aprox.
> > > > RTD| 33.505| 53.479| 71.842| 0| 26.739| 77.319
> > > > RTD| 40.592| 62.177| 123.067| 0| 26.739| 123.067
> > > > -------
> > > > \_________: Key pressed
> > > > RTD| 50.579| 53.479| 73.775| 0| 26.739| 123.067
> > >
> > > This is where you are wrong:
> > > - first, let me repeat it: test made without load are irrelevant;
> > I can't agree with you. When we stress a system with load is, as far as
> > I know, because usually, the large latencies don't appear on a quiet
> > system. Here we have a case where a large latency (the lat_worst number
> > I've gotten after more than 7 hours with the system fully loaded is even
> > less than this) appears on a quiet system and directly related to an
> > external event.
> >
> > > - second, an event has no relative effect on max latency, its effect
> > > is absolute: pressing a key over a telnet session causes, for unknown
> > > reason, a masking section of around 130us, which happens to also be
> > > the worst case latency that we measured properly with a loaded system.
> > I pointed out the lat_max field because, if you keep the latency test
> > running and hit a key on a telnet session, you can easily see how the
> > lat_max increase each time you hit the key, while the lat_worst increase
> > depends on the former history.
> >
> > >
> > > Now, if you want to know why you get such a masking section, you are
> > > free to investigate.
> >
> > I'll try to do it.
>
> Does the network driver use the AT91 PDC ? If yes, and if it is
> possible to disable it, could you try disabling it ?
Yes, the network driver uses DMA and is not easy to disable it. Anyway I
have, in my custom board, an additional ethernet chip (smc91x). I've
done the former tests with the smc91x (LAN91C111), which can't use DMA,
and the behaviour is similar.
Regards,
Juan Antonio
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-01-24 9:41 ` Juan Antonio Garcia Redondo
2008-01-24 10:02 ` Gilles Chanteperdrix
@ 2008-02-10 20:31 ` Gilles Chanteperdrix
2008-02-11 13:41 ` Jan Kiszka
1 sibling, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-02-10 20:31 UTC (permalink / raw)
To: Juan Antonio Garcia Redondo; +Cc: jagarcia, xenomai
Juan Antonio Garcia Redondo wrote:
> On 23/01/08 14:15, Gilles Chanteperdrix wrote:
> > On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
> > <gilles.chanteperdrix@xenomai.org> wrote:
> > > On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
> > >
> > > <juan-antonio.garcia@domain.hid> wrote:
> > > > I see everything OK except for the first samples of cyclictests. Any comments ?
> > >
> > > The load you apply does not load the cache, which is a source of
> > > jitter. You should run the cache calibrator (I do not find the cache
> > > calibrator URL, but it is somewhere in Xenomai distribution or wiki).
> >
> > It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
> >
> > --
> > Gilles Chanteperdrix
>
> Thanks Gilles, I've done more tests using the cache calibrator from
> http://www.cwi.nl/~manegold/Calibrator. The latency numbers are very
> similar althought I've found an strange behaviour related to telnet
> sessions.
>
> Environment:
> o Tests running from console over atmel serial port.
> o A telnet session over on-chip ethernet.
> o System without load.
>
> ./latency -p 500 -t0
> == All results in microseconds
> warming up...
> RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
> RTH|-RTH----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat
> worst
> RTD| 49.613| 52.190| 62.822| 0| 49.613| 62.822
> RTD| 42.203| 52.512| 66.365| 0| 42.203| 66.365
>
>
> Now If hit a key on the telnet session :
>
> RTD| 36.726| 57.989| 109.536| 0| 31.572| 109.536 <-------- Here I've hit the key.
> RTD| 36.404| 51.868| 69.587| 0| 31.572| 109.536
> RTD| 35.760| 51.868| 73.775| 0| 31.572| 109.536
>
> Now, I launch an script which executes four instances of cache
> calibrator.
>
> RTD| 45.103| 57.667| 75.708| 0| 32.538| 122.422
> RTD| 45.425| 57.023| 76.030| 0| 32.538| 122.422
> RTD| 46.069| 57.023| 75.708| 0| 32.538| 122.422
>
> Now, I can hit a key on the telnet session without effects over latency
> numbers:
>
> RTD| 44.136| 57.989| 75.386| 0| 27.384| 128.221
> RTD| 46.713| 57.345| 76.353| 0| 27.384| 128.221
> RTD| 44.780| 57.345| 76.675| 0| 27.384| 128.221
> RTD| 43.492| 56.701| 76.997| 0| 27.384| 128.221
>
> Now I stop the calibrator process and launch 'ping -f -s2048 192.168.2.82' from an external
> machine.
>
> RTD| 40.270| 68.621| 90.850| 0| 27.384| 128.221
> RTD| 36.082| 68.621| 88.273| 0| 27.384| 128.221
> RTD| 40.592| 67.976| 91.494| 0| 27.384| 128.221
> RTD| 41.237| 68.298| 89.239| 0| 27.384| 128.221
>
>
> Now If hit a key on the telnet session :
>
> RTD| 42.203| 67.976| 88.273| 0| 27.384| 128.221
> RTD| 32.216| 93.427| 128.543| 0| 27.384| 128.543 <---------- Here I've hit the key.
> RTD| 42.203| 68.298| 87.628| 0| 27.384| 128.543
>
> And again the calibrator execution results on eliminate the strange
> behaviour whith the telnet session.
>
> Any clues ?
Here is an update, follow-up on xenomai-core. I was finally able to
reproduce this behaviour: I run latency in the background and hit the
"Enter" key on my serial console, and get high latency figures.
I enabled the tracer, set xenomai latency to 300us and managed to get a
trace (220us latency). However, I do not understand what is going wrong
from reading the trace, so I post it here in case someone sees something.
Ah, and I added an ipipe_trace_special in ipipe_grab_irq to log the
number of the received irq. 1 is serial interrupt 18 (0x12) is timer
interrupt.
Inline, so that Jan can comment it.
--
Gilles Chanteperdrix.
---|------------|------------|------------|--------|-------------------------
RTS| -221.527| -152.777| 220.833| 0| 00:00:25/00:00:25
# cat /p\b\b# cat /proc/ipip# cat /proc/ipipe/tr# cat /proc/ipipe/trace/fr# cat /proc/ipipe/trace/frozen
I-pipe frozen back-tracing service on 2.6.20/ipipe-1.8-04
------------------------------------------------------------
CPU: 0, Freeze: 450692973 cycles, Trace Points: 1000 (+10)
Calibrated minimum trace-point overhead: 1.000 us
+----- Hard IRQs ('|': locked)
|+---- <unused>
||+--- <unused>
|||+-- Xenomai
||||+- Linux ('*': domain stalled, '+': current, '#': current+stalled)
||||| +---------- Delay flag ('+': > 1 us, '!': > 10 us)
||||| | +- NMI noise ('N')
||||| | |
Type User Val. Time Delay Function (Parent)
:| +func -10487+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -10485+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -10482+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -10478+ 2.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -10476+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -10466+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -10463+ 4.500 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -10459! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -10444+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -10437+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -10433+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -10430! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -10370! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -10352+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -10349+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -10345! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -10333+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -10325! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -10304+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -10301+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -10300+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -10296+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -10294+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -10288+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -10286+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -10283+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -10281+ 2.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -10278+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -10273+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -10269+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -10261! 50.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -10211! 19.500 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -10191+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -10184+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -10179+ 2.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -10176+ 6.500 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -10170+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -10167+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -10164+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -10162+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -10160+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -10159+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -10157+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -10155! 363.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -9791+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -9790+ 3.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -9786+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -9782+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -9779+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -9777+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -9774+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -9772+ 1.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -9770+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -9767+ 2.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -9764+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -9762+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -9758+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -9755+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -9745+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -9742+ 3.500 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -9739! 15.500 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -9723+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -9717+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -9713+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -9709! 59.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -9650! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -9632+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -9629+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -9625! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -9613+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -9605! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -9584+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -9581+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -9579+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -9575+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -9573+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -9567+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -9565+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -9563+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -9561+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -9558+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -9552+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -9549+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -9540! 50.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -9490! 19.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -9471+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -9464+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -9459+ 2.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -9456+ 6.500 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -9450+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -9447+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -9444+ 2.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -9442+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -9441+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -9439+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -9437+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -9435! 364.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -9071+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -9070+ 3.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -9066+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -9062+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -9059+ 3.000 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -9056+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -9054+ 1.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -9052+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -9050+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -9046+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -9044+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -9042+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -9038+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -9035+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -9026+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -9023+ 3.500 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -9019! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -9004+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -8998+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -8994+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -8990! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -8931! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -8913+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -8909+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -8905! 12.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -8893+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8885! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -8864+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -8861+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -8859+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8855+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -8853+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -8847+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -8845+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -8843+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -8840+ 2.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -8838+ 6.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -8832+ 3.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -8829+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -8820! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -8769! 19.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -8750+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -8743+ 3.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +func -8740+ 3.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -8737+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -8730+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -8725+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -8722+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -8720+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -8717+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -8715+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -8713+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -8710+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -8708+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -8705+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -8701+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -8698! 15.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -8683+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -8677+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -8672+ 2.500 rthal_irq_host_pend+0x10 (xnintr_clock_handler+0xbc)
:| # func -8670+ 4.000 __ipipe_schedule_irq+0x10 (rthal_irq_host_pend+0x20)
:| # func -8666+ 4.000 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
:| +func -8662+ 2.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +func -8660+ 6.000 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
:| #end 0x80000000 -8654+ 2.500 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
: #func -8651+ 3.000 asm_do_IRQ+0x10 (__ipipe_sync_stage+0x284)
: #func -8648+ 2.000 handle_level_irq+0x10 (asm_do_IRQ+0x50)
: #func -8646+ 2.500 _spin_lock+0x10 (handle_level_irq+0x24)
: #func -8644+ 3.000 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -8641+ 5.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -8636+ 1.500 _spin_unlock+0x10 (handle_level_irq+0x88)
: #func -8634+ 3.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -8631+ 2.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -8629+ 3.000 handle_IRQ_event+0x10 (handle_level_irq+0x94)
: #func -8626+ 2.000 at91_timer_interrupt+0x10 (handle_IRQ_event+0x3c)
: #func -8624+ 1.500 _spin_lock+0x10 (at91_timer_interrupt+0x1c)
: #func -8622+ 2.000 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -8620+ 3.500 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -8617+ 2.000 timer_tick+0x10 (at91_timer_interrupt+0x50)
: #func -8615+ 6.500 profile_tick+0x10 (timer_tick+0x18)
: #func -8608+ 6.500 do_timer+0x14 (timer_tick+0xe0)
: #func -8602+ 7.500 current_tick_length+0x10 (do_timer+0x144)
: #func -8594+ 2.000 update_process_times+0x10 (timer_tick+0xfc)
: #func -8592+ 5.000 account_system_time+0x10 (update_process_times+0x44)
: #func -8587+ 2.000 run_local_timers+0x10 (update_process_times+0x48)
: #func -8585+ 2.000 raise_softirq+0x10 (run_local_timers+0x18)
:| #begin 0x80000001 -8583+ 1.500 raise_softirq+0x34 (run_local_timers+0x18)
:| #end 0x80000001 -8582+ 3.500 raise_softirq+0x54 (run_local_timers+0x18)
: #func -8578+ 2.000 __ipipe_restore_root+0x10 (raise_softirq+0xb4)
:| #begin 0x80000001 -8576+ 2.500 __ipipe_restore_root+0x38 (raise_softirq+0xb4)
:| #end 0x80000001 -8574+ 2.000 __ipipe_restore_root+0x58 (raise_softirq+0xb4)
: #func -8572+ 4.500 softlockup_tick+0x14 (run_local_timers+0x1c)
: #func -8567+ 2.000 rcu_pending+0x10 (update_process_times+0x50)
: #func -8565+ 3.500 __rcu_pending+0x10 (rcu_pending+0x1c)
: #func -8562+ 3.000 __rcu_pending+0x10 (rcu_pending+0x34)
: #func -8559+ 2.000 scheduler_tick+0x10 (update_process_times+0x64)
: #func -8557+ 4.000 sched_clock+0x10 (scheduler_tick+0x14)
: #func -8553+ 5.000 run_posix_cpu_timers+0x14 (update_process_times+0x6c)
: #func -8548+ 1.500 _spin_unlock+0x10 (at91_timer_interrupt+0xc0)
: #func -8546+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -8545+ 3.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
:| #begin 0x80000001 -8542+ 2.000 handle_IRQ_event+0x84 (handle_level_irq+0x94)
:| #end 0x80000001 -8540+ 2.000 handle_IRQ_event+0xa4 (handle_level_irq+0x94)
: #func -8538+ 3.500 note_interrupt+0x14 (handle_level_irq+0xb0)
: #func -8534+ 1.500 _spin_lock+0x10 (handle_level_irq+0xb8)
: #func -8533+ 1.500 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -8531+ 2.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -8529+ 2.000 at91_aic_unmask_irq+0x10 (handle_level_irq+0xe4)
: #func -8527+ 1.500 _spin_unlock+0x10 (handle_level_irq+0xec)
: #func -8526+ 2.000 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -8524+ 2.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -8522+ 2.500 irq_exit+0x10 (asm_do_IRQ+0x54)
: #func -8519+ 3.000 __do_softirq+0x10 (irq_exit+0x44)
: #func -8516+ 2.000 __ipipe_unstall_root+0x10 (__do_softirq+0x40)
:| #begin 0x80000000 -8514+ 1.500 __ipipe_unstall_root+0x30 (__do_softirq+0x40)
:| +end 0x80000000 -8513+ 3.000 __ipipe_unstall_root+0x64 (__do_softirq+0x40)
: +func -8510+ 2.500 run_timer_softirq+0x14 (__do_softirq+0x58)
: +func -8507+ 6.000 hrtimer_run_queues+0x10 (run_timer_softirq+0x24)
: +func -8501+ 1.500 _spin_lock_irq+0x10 (run_timer_softirq+0x40)
:| +begin 0x80000001 -8500+ 2.500 _spin_lock_irq+0x34 (run_timer_softirq+0x40)
:| #end 0x80000001 -8497+ 2.000 _spin_lock_irq+0x54 (run_timer_softirq+0x40)
: #func -8495+ 1.500 ipipe_check_context+0x10 (_spin_lock_irq+0x60)
: #func -8494+ 4.500 _raw_spin_lock+0x14 (_spin_lock_irq+0x68)
: #func -8489+ 1.500 _spin_unlock_irq+0x10 (run_timer_softirq+0x1ac)
: #func -8488+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock_irq+0x14)
: #func -8486+ 1.500 __ipipe_unstall_root+0x10 (_spin_unlock_irq+0x18)
:| #begin 0x80000000 -8485+ 2.000 __ipipe_unstall_root+0x30 (_spin_unlock_irq+0x18)
:| +end 0x80000000 -8483+ 2.000 __ipipe_unstall_root+0x64 (_spin_unlock_irq+0x18)
: +func -8481+ 2.500 ipipe_check_context+0x10 (_spin_unlock_irq+0x20)
:| +begin 0x80000001 -8478+ 2.000 __do_softirq+0x90 (irq_exit+0x44)
:| #end 0x80000001 -8476+ 2.500 __do_softirq+0xb0 (irq_exit+0x44)
: #func -8474+ 3.000 _local_bh_enable+0x10 (__do_softirq+0xe8)
: #func -8471+ 2.500 ipipe_check_context+0x10 (irq_exit+0x4c)
:| #begin 0x80000000 -8468+ 3.500 __ipipe_sync_stage+0x31c (__ipipe_walk_pipeline+0xa8)
:| +end 0xffffffff -8465+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -8462+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -8460+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -8458+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -8456+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -8453+ 3.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -8450+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -8449+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -8447+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -8445+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -8444! 92.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -8352+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -8350+ 1.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -8348+ 1.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -8347+ 2.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -8345+ 1.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -8343+ 1.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -8342+ 1.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -8340+ 1.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -8339+ 2.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -8337+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -8335+ 1.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -8334+ 2.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -8332+ 1.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -8330+ 4.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -8326+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -8323+ 3.500 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -8320! 10.500 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -8309+ 3.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -8306+ 2.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -8304+ 4.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -8300! 66.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -8234! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -8216+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -8212+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -8208! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -8196+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8189! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -8168+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -8165+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -8163+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8159+ 2.500 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -8157+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -8151+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -8149+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -8146+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -8144+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -8141+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -8135+ 3.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -8132+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -8124! 50.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -8074! 19.500 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -8054+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -8048+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -8042+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -8039+ 6.500 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -8033+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -8030+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -8028+ 3.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -8025+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -8023+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -8021+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -8019+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -8017! 382.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -7635+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -7633+ 3.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -7630+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -7626+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -7623+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -7620+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -7618+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -7616+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -7614+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -7610+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -7608+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -7605+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -7602+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -7599+ 9.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -7590+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -7586+ 3.500 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -7583! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -7568+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -7561+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -7557+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -7554! 60.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -7494! 17.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -7476+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -7473+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -7469! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -7457+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -7449! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -7428+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -7426+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -7424+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -7420+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -7418+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -7412+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -7410+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -7407+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -7405+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -7402+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -7397+ 4.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -7393+ 8.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -7385! 50.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -7334! 19.500 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -7315+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -7308+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -7303+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -7300+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -7294+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -7291+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -7288+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -7286+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -7284+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -7283+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -7281+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -7279! 367.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -6912+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -6910+ 3.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -6907+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -6902+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -6899+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -6897+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -6894+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -6892+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -6890+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -6887+ 2.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -6884+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -6882+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -6878+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -6875+ 9.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -6866+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -6863+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -6859! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -6844+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -6837+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -6833+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -6830! 59.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -6771! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -6753+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -6749+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -6745! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -6734+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -6726! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -6705+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -6702+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -6700+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -6696+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -6694+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -6688+ 1.500 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -6686+ 3.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -6683+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -6681+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -6678+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -6673+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -6669+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -6661! 50.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -6610! 19.500 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -6591+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -6584+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -6579+ 2.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -6576+ 6.500 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -6570+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -6567+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -6564+ 2.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -6562+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -6561+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -6559+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -6557+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -6555! 363.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -6192+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -6190+ 3.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -6187+ 4.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -6183+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -6180+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -6177+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -6175+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -6173+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -6171+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -6167+ 2.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -6165+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -6162+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -6159+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -6156+ 9.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -6147+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -6144+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -6140! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -6125+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -6118+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -6114+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -6111! 59.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -6052! 17.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -6034+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -6031+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -6027! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -6015+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -6007! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -5986+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -5983+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -5981+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -5978+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -5976+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -5969+ 1.500 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -5968+ 3.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -5965+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -5963+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -5960+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -5954+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -5951+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -5942! 50.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -5892! 19.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -5873+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -5866+ 5.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -5861+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -5858+ 6.500 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -5851+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -5849+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -5846+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -5844+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -5842+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -5840+ 1.500 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -5839+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -5837! 365.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -5472+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -5470+ 3.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -5467+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -5462+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -5459+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -5457+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -5455+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -5452+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -5450+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -5447+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -5445+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -5442+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -5438+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -5435+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -5426+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -5423+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -5419! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -5404+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -5398+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -5393+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -5390! 60.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -5330! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -5312+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -5308+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -5304! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -5292+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -5285! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -5264+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -5261+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -5259+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -5255+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -5253+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -5247+ 1.500 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -5245+ 3.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -5242+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -5240+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -5237+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -5232+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -5228+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -5220! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -5169! 19.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -5150+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -5143+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -5138+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -5135+ 6.500 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -5128+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -5126+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -5123+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -5120+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -5119+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -5117+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -5115+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -5114! 362.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -4751+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -4750+ 3.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -4747+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -4742+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -4739+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -4737+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -4734+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -4732+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -4730+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -4727+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -4725+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -4722+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -4718+ 2.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -4716+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -4706+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -4703+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -4699! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -4684+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -4678+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -4674+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -4670! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -4611! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -4593+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -4589+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -4585! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -4574+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -4566! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -4545+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -4542+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -4541+ 4.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -4536+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -4534+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -4528+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -4526+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -4524+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -4521+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -4518+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -4513+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -4510+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -4501! 50.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -4451! 20.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -4431+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -4424+ 5.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -4419+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -4416+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -4410+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -4407+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -4405+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -4402+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -4401+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -4399+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -4397+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -4395! 363.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -4032+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -4030+ 3.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -4027+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -4022+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -4020+ 3.000 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -4017+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -4014+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -4012+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -4010+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -4007+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -4005+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -4002+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -3998+ 2.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -3996+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -3986+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -3983+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -3979! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -3964+ 7.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -3957+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -3953+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -3950! 59.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -3891! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -3873+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -3869+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -3865! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -3854+ 8.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3845! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -3824+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -3822+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -3820+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3816+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -3814+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -3808+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -3806+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -3803+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -3801+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -3798+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -3792+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -3789+ 8.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -3781! 50.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -3731! 20.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -3711+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -3704+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -3699+ 2.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -3696+ 6.500 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -3690+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -3687+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -3684+ 2.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -3682+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -3681+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -3679+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -3677+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -3675! 364.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -3311+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -3309+ 3.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -3306+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -3302+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -3299+ 3.000 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -3296+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -3294+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -3292+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -3290+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -3286+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -3284+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -3281+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -3278+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -3275+ 9.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -3266+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -3263+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -3259! 15.500 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -3243+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -3237+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -3233+ 4.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -3229! 60.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -3169! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -3151+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -3148+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -3144! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -3132+ 8.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3124! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -3103+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -3100+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -3098+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3094+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -3092+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -3086+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -3084+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -3082+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -3080+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -3077+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -3071+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -3068+ 8.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -3060! 50.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -3009! 19.500 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -2990+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -2983+ 5.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -2978+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -2975+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -2969+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -2966+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -2964+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -2961+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -2960+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -2958+ 1.500 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -2956+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -2954! 362.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -2592+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -2590+ 3.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -2587+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -2582+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -2579+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -2577+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -2574+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -2572+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -2570+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -2567+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -2565+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -2562+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -2558+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -2555+ 9.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -2546+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -2543+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -2539! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -2524+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -2518+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -2514+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -2510! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -2451! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -2433+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -2429+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -2425! 12.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -2413+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -2405! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -2384+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -2381+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -2379+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -2376+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -2374+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -2367+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -2365+ 2.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -2363+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -2361+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -2358+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -2352+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -2349+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -2340! 50.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -2290! 19.500 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -2270+ 6.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -2264+ 5.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -2259+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -2256+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -2250+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -2247+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -2244+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -2242+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -2240+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -2238+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -2236+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -2235! 363.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -1871+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -1870+ 3.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -1867+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -1862+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -1859+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -1857+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -1854+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -1852+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -1850+ 3.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -1847+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -1845+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -1842+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -1838+ 2.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -1836+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -1826+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -1823+ 3.500 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -1819! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -1804+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -1798+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -1794+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -1790! 59.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -1731! 17.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -1714+ 3.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -1710+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -1706! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -1695+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -1687! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -1666+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -1663+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -1661+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -1657+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -1655+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -1649+ 1.500 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -1647+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -1645+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -1642+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -1639+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -1634+ 3.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -1630+ 8.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -1622! 50.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -1571! 19.500 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -1552+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -1545+ 4.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +func -1541+ 2.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -1538+ 6.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -1532+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -1527+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -1524+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -1522+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -1519+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -1517+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -1515+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -1511+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -1510+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -1507+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -1503+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -1500! 14.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -1485+ 7.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -1478+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -1474+ 2.500 rthal_irq_host_pend+0x10 (xnintr_clock_handler+0xbc)
:| # func -1471+ 3.500 __ipipe_schedule_irq+0x10 (rthal_irq_host_pend+0x20)
:| # func -1468+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
:| +func -1463+ 2.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +func -1461+ 6.000 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
:| #end 0x80000000 -1455+ 2.000 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
: #func -1453+ 3.000 asm_do_IRQ+0x10 (__ipipe_sync_stage+0x284)
: #func -1450+ 2.000 handle_level_irq+0x10 (asm_do_IRQ+0x50)
: #func -1448+ 2.500 _spin_lock+0x10 (handle_level_irq+0x24)
: #func -1446+ 3.000 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -1443+ 5.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -1438+ 2.000 _spin_unlock+0x10 (handle_level_irq+0x88)
: #func -1436+ 3.000 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -1433+ 2.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -1431+ 3.000 handle_IRQ_event+0x10 (handle_level_irq+0x94)
: #func -1428+ 2.000 at91_timer_interrupt+0x10 (handle_IRQ_event+0x3c)
: #func -1426+ 2.000 _spin_lock+0x10 (at91_timer_interrupt+0x1c)
: #func -1424+ 1.500 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -1422+ 3.500 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -1419+ 2.000 timer_tick+0x10 (at91_timer_interrupt+0x50)
: #func -1417+ 6.500 profile_tick+0x10 (timer_tick+0x18)
: #func -1410+ 6.500 do_timer+0x14 (timer_tick+0xe0)
: #func -1404+ 7.500 current_tick_length+0x10 (do_timer+0x144)
: #func -1396+ 2.500 update_process_times+0x10 (timer_tick+0xfc)
: #func -1394+ 5.000 account_system_time+0x10 (update_process_times+0x44)
: #func -1389+ 2.000 run_local_timers+0x10 (update_process_times+0x48)
: #func -1387+ 2.000 raise_softirq+0x10 (run_local_timers+0x18)
:| #begin 0x80000001 -1385+ 2.500 raise_softirq+0x34 (run_local_timers+0x18)
:| #end 0x80000001 -1382+ 3.000 raise_softirq+0x54 (run_local_timers+0x18)
: #func -1379+ 2.000 __ipipe_restore_root+0x10 (raise_softirq+0xb4)
:| #begin 0x80000001 -1377+ 2.500 __ipipe_restore_root+0x38 (raise_softirq+0xb4)
:| #end 0x80000001 -1375+ 2.500 __ipipe_restore_root+0x58 (raise_softirq+0xb4)
: #func -1372+ 4.000 softlockup_tick+0x14 (run_local_timers+0x1c)
: #func -1368+ 2.000 rcu_pending+0x10 (update_process_times+0x50)
: #func -1366+ 4.000 __rcu_pending+0x10 (rcu_pending+0x1c)
: #func -1362+ 3.000 __rcu_pending+0x10 (rcu_pending+0x34)
: #func -1359+ 1.500 scheduler_tick+0x10 (update_process_times+0x64)
: #func -1358+ 4.000 sched_clock+0x10 (scheduler_tick+0x14)
: #func -1354+ 5.000 run_posix_cpu_timers+0x14 (update_process_times+0x6c)
: #func -1349+ 1.500 _spin_unlock+0x10 (at91_timer_interrupt+0xc0)
: #func -1347+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -1346+ 3.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
:| #begin 0x80000001 -1343+ 2.000 handle_IRQ_event+0x84 (handle_level_irq+0x94)
:| #end 0x80000001 -1341+ 2.000 handle_IRQ_event+0xa4 (handle_level_irq+0x94)
: #func -1339+ 3.500 note_interrupt+0x14 (handle_level_irq+0xb0)
: #func -1335+ 1.500 _spin_lock+0x10 (handle_level_irq+0xb8)
: #func -1334+ 1.500 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -1332+ 2.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -1330+ 2.000 at91_aic_unmask_irq+0x10 (handle_level_irq+0xe4)
: #func -1328+ 1.500 _spin_unlock+0x10 (handle_level_irq+0xec)
: #func -1327+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -1325+ 2.500 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -1323+ 2.500 irq_exit+0x10 (asm_do_IRQ+0x54)
: #func -1320+ 2.500 __do_softirq+0x10 (irq_exit+0x44)
: #func -1318+ 2.000 __ipipe_unstall_root+0x10 (__do_softirq+0x40)
:| #begin 0x80000000 -1316+ 2.000 __ipipe_unstall_root+0x30 (__do_softirq+0x40)
:| +end 0x80000000 -1314+ 3.000 __ipipe_unstall_root+0x64 (__do_softirq+0x40)
: +func -1311+ 2.500 run_timer_softirq+0x14 (__do_softirq+0x58)
: +func -1308+ 5.500 hrtimer_run_queues+0x10 (run_timer_softirq+0x24)
: +func -1303+ 2.000 _spin_lock_irq+0x10 (run_timer_softirq+0x40)
:| +begin 0x80000001 -1301+ 2.500 _spin_lock_irq+0x34 (run_timer_softirq+0x40)
:| #end 0x80000001 -1298+ 1.500 _spin_lock_irq+0x54 (run_timer_softirq+0x40)
: #func -1297+ 1.500 ipipe_check_context+0x10 (_spin_lock_irq+0x60)
: #func -1295+ 4.500 _raw_spin_lock+0x14 (_spin_lock_irq+0x68)
: #func -1291+ 1.500 _spin_unlock_irq+0x10 (run_timer_softirq+0x1ac)
: #func -1289+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock_irq+0x14)
: #func -1288+ 2.000 __ipipe_unstall_root+0x10 (_spin_unlock_irq+0x18)
:| #begin 0x80000000 -1286+ 2.000 __ipipe_unstall_root+0x30 (_spin_unlock_irq+0x18)
:| +end 0x80000000 -1284+ 1.500 __ipipe_unstall_root+0x64 (_spin_unlock_irq+0x18)
: +func -1282+ 2.500 ipipe_check_context+0x10 (_spin_unlock_irq+0x20)
:| +begin 0x80000001 -1280+ 2.000 __do_softirq+0x90 (irq_exit+0x44)
:| #end 0x80000001 -1278+ 2.500 __do_softirq+0xb0 (irq_exit+0x44)
: #func -1275+ 3.000 _local_bh_enable+0x10 (__do_softirq+0xe8)
: #func -1272+ 2.000 ipipe_check_context+0x10 (irq_exit+0x4c)
:| #begin 0x80000000 -1270+ 4.000 __ipipe_sync_stage+0x31c (__ipipe_walk_pipeline+0xa8)
:| +end 0xffffffff -1266+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
: +func -1263+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -1262+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -1260+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -1257+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -1254+ 2.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -1252+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -1251+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -1249+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -1247+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -1245! 94.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -1151+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -1150+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -1148+ 2.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -1146+ 1.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -1144+ 1.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -1143+ 1.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -1141+ 1.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +func -1140+ 1.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +func -1138+ 2.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| +func -1136+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| +func -1135+ 2.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -1133+ 1.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -1131+ 1.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -1130+ 4.000 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| # func -1126+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| # func -1123+ 3.500 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| # [ 753] -<?>- 99 -1119! 10.500 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| # func -1109+ 4.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| # func -1105+ 2.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -1103+ 4.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -1099! 66.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 753] -<?>- 99 -1033! 18.000 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -1015+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -1011+ 3.500 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -1007! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -995+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -987! 36.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -951+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -948+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -946+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -942+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -940+ 6.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -934+ 5.000 __rt_sem_v+0x14 (hisyscall_event+0x150)
: + func -929+ 2.000 xnregistry_fetch+0x10 (__rt_sem_v+0x60)
:| + begin 0x80000000 -927+ 3.500 xnregistry_fetch+0x34 (__rt_sem_v+0x60)
:| # func -924+ 1.500 __ipipe_restore_pipeline_head+0x10 (xnregistry_fetch+0xfc)
:| + end 0x80000000 -922+ 3.000 __ipipe_restore_pipeline_head+0x11c (xnregistry_fetch+0xfc)
: + func -919+ 2.000 rt_sem_v+0x10 (__rt_sem_v+0x70)
:| + begin 0x80000000 -917+ 3.500 rt_sem_v+0x38 (__rt_sem_v+0x70)
:| # func -914+ 7.500 xnsynch_wakeup_one_sleeper+0x14 (rt_sem_v+0x8c)
:| # func -906+ 3.500 xnpod_resume_thread+0x14 (xnsynch_wakeup_one_sleeper+0x324)
:| # [ 752] -<?>- 0 -903! 10.500 xnpod_resume_thread+0x84 (xnsynch_wakeup_one_sleeper+0x324)
:| # func -892+ 4.000 xnpod_schedule+0x14 (rt_sem_v+0x98)
:| # [ 753] -<?>- 99 -888+ 3.500 xnpod_schedule+0xc8 (rt_sem_v+0x98)
:| # func -885+ 2.000 __ipipe_restore_pipeline_head+0x10 (rt_sem_v+0xd4)
:| + end 0x80000000 -883+ 2.000 __ipipe_restore_pipeline_head+0x11c (rt_sem_v+0xd4)
:| + begin 0x80000001 -881+ 2.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -879+ 4.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -874+ 1.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -873+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -871+ 2.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -869+ 1.500 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -867+ 2.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -865+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -863+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -861+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -859+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -856+ 5.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -850+ 2.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 753] -<?>- 99 -848! 11.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # [ 752] -<?>- 0 -837+ 7.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| # func -829+ 2.000 __ipipe_restore_pipeline_head+0x10 (rt_sem_p+0x158)
:| + end 0x80000000 -827+ 4.500 __ipipe_restore_pipeline_head+0x11c (rt_sem_p+0x158)
:| + begin 0x80000001 -823+ 2.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -821! 352.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -468+ 6.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: + func -462+ 3.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -459+ 6.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -453+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -451+ 3.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -447+ 2.500 xnshadow_relax+0x14 (hisyscall_event+0x210)
:| + begin 0x80000000 -445+ 3.000 xnshadow_relax+0xd4 (hisyscall_event+0x210)
:| # func -442+ 5.000 schedule_linux_call+0x10 (xnshadow_relax+0x114)
:| # func -437+ 4.000 rthal_apc_schedule+0x10 (schedule_linux_call+0x1e8)
:| # func -433+ 5.000 __ipipe_schedule_irq+0x10 (rthal_apc_schedule+0xac)
:| # func -428+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
:| # func -423+ 3.500 rpi_push+0x14 (xnshadow_relax+0x11c)
:| # func -420+ 5.500 xnpod_suspend_thread+0x14 (xnshadow_relax+0x148)
:| # func -414+ 4.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
:| # [ 752] -<?>- 0 -410+ 7.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
:| # func -403! 56.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
:| # [ 0] -<?>- -1 -347! 20.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
:| +func -327+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +func -324+ 7.500 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
:| #end 0x80000000 -317+ 7.000 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
:| #func -310+ 2.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| #begin 0xffffffff -307+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| #(0x2a) 0x00000012 -300+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| #func -295+ 4.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| #func -291+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| #func -289+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| #func -286+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| #func -284+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| #func -282+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
:| #func -278+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
:| #func -276+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| #func -273+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| #*func -270+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| #*func -266+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
:| #*func -257+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
:| #*func -253+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
:| #*[ 753] -<?>- 99 -249! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
:| #*func -234+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
:| #*func -228+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| #*func -224+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| #*[ 0] -<?>- -1 -220! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| #*[ 753] -<?>- 99 -161! 17.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
:| #*func -143+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| #*func -139+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| +*end 0x80000000 -135+ 7.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| +*func -128+ 3.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +*begin 0xffffffff -125+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +*(0x2a) 0x00000001 -118+ 4.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +*func -114+ 5.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +*func -109+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_handle_irq+0xdc)
:| +*func -105+ 2.000 __ipipe_ack_irq+0x10 (__ipipe_handle_irq+0xfc)
:| +*func -103+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_irq+0x28)
:| +*func -100+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
:| +*func -97+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
:| +*func -95+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +*end 0xffffffff -92! 10.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +*begin 0x80000001 -81+ 4.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| +*end 0x80000001 -77! 39.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: +*func -38+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
: +*func -35+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| +*begin 0x80000001 -33+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| +*end 0x80000001 -29+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: +*func -27+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: +*func -21+ 2.500 xnshadow_sys_trace+0x10 (hisyscall_event+0x150)
: +*func -18+ 2.000 ipipe_trace_frozen_reset+0x10 (xnshadow_sys_trace+0x98)
: +*func -16+ 2.000 __ipipe_global_path_lock+0x10 (ipipe_trace_frozen_reset+0x14)
: +*func -14+ 2.000 __ipipe_spin_lock_irqsave+0x10 (__ipipe_global_path_lock+0x18)
:| +*begin 0x80000001 -12+ 8.000 __ipipe_spin_lock_irqsave+0x34 (__ipipe_global_path_lock+0x18)
:| #*func -4+ 2.500 __ipipe_spin_unlock_irqcomplete+0x10 (__ipipe_global_path_unlock+0x6c)
:| +*end 0x80000001 -2+ 2.000 __ipipe_spin_unlock_irqcomplete+0x4c (__ipipe_global_path_unlock+0x6c)
< +*freeze 0x00035ea1 0 2.500 xnshadow_sys_trace+0xa4 (hisyscall_event+0x150)
| +*begin 0x80000001 2 1.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
| +*end 0x80000001 4 5.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
+*func 9 2.000 __ipipe_syscall_root+0x10 (vector_swi+0x68)
+*func 11 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
| +*begin 0x80000001 12 2.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
| +*end 0x80000001 14 1.500 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
+*func 16 2.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
+*func 18 1.500 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
+*func 20 3.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
+*func 23 0.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
#
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-02-10 20:31 ` [Xenomai-core] " Gilles Chanteperdrix
@ 2008-02-11 13:41 ` Jan Kiszka
2008-02-11 14:05 ` Gilles Chanteperdrix
2008-02-12 7:04 ` Gilles Chanteperdrix
0 siblings, 2 replies; 27+ messages in thread
From: Jan Kiszka @ 2008-02-11 13:41 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
Gilles Chanteperdrix wrote:
> Juan Antonio Garcia Redondo wrote:
> > On 23/01/08 14:15, Gilles Chanteperdrix wrote:
> > > On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
> > > <gilles.chanteperdrix@xenomai.org> wrote:
> > > > On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
> > > >
> > > > <juan-antonio.garcia@domain.hid> wrote:
> > > > > I see everything OK except for the first samples of cyclictests. Any comments ?
> > > >
> > > > The load you apply does not load the cache, which is a source of
> > > > jitter. You should run the cache calibrator (I do not find the cache
> > > > calibrator URL, but it is somewhere in Xenomai distribution or wiki).
> > >
> > > It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
> > >
> > > --
> > > Gilles Chanteperdrix
> >
> > Thanks Gilles, I've done more tests using the cache calibrator from
> > http://www.cwi.nl/~manegold/Calibrator. The latency numbers are very
> > similar althought I've found an strange behaviour related to telnet
> > sessions.
> >
> > Environment:
> > o Tests running from console over atmel serial port.
> > o A telnet session over on-chip ethernet.
> > o System without load.
> >
> > ./latency -p 500 -t0
> > == All results in microseconds
> > warming up...
> > RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
> > RTH|-RTH----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat
> > worst
> > RTD| 49.613| 52.190| 62.822| 0| 49.613| 62.822
> > RTD| 42.203| 52.512| 66.365| 0| 42.203| 66.365
> >
> >
> > Now If hit a key on the telnet session :
> >
> > RTD| 36.726| 57.989| 109.536| 0| 31.572| 109.536 <-------- Here I've hit the key.
> > RTD| 36.404| 51.868| 69.587| 0| 31.572| 109.536
> > RTD| 35.760| 51.868| 73.775| 0| 31.572| 109.536
> >
> > Now, I launch an script which executes four instances of cache
> > calibrator.
> >
> > RTD| 45.103| 57.667| 75.708| 0| 32.538| 122.422
> > RTD| 45.425| 57.023| 76.030| 0| 32.538| 122.422
> > RTD| 46.069| 57.023| 75.708| 0| 32.538| 122.422
> >
> > Now, I can hit a key on the telnet session without effects over latency
> > numbers:
> >
> > RTD| 44.136| 57.989| 75.386| 0| 27.384| 128.221
> > RTD| 46.713| 57.345| 76.353| 0| 27.384| 128.221
> > RTD| 44.780| 57.345| 76.675| 0| 27.384| 128.221
> > RTD| 43.492| 56.701| 76.997| 0| 27.384| 128.221
> >
> > Now I stop the calibrator process and launch 'ping -f -s2048 192.168.2.82' from an external
> > machine.
> >
> > RTD| 40.270| 68.621| 90.850| 0| 27.384| 128.221
> > RTD| 36.082| 68.621| 88.273| 0| 27.384| 128.221
> > RTD| 40.592| 67.976| 91.494| 0| 27.384| 128.221
> > RTD| 41.237| 68.298| 89.239| 0| 27.384| 128.221
> >
> >
> > Now If hit a key on the telnet session :
> >
> > RTD| 42.203| 67.976| 88.273| 0| 27.384| 128.221
> > RTD| 32.216| 93.427| 128.543| 0| 27.384| 128.543 <---------- Here I've hit the key.
> > RTD| 42.203| 68.298| 87.628| 0| 27.384| 128.543
> >
> > And again the calibrator execution results on eliminate the strange
> > behaviour whith the telnet session.
> >
> > Any clues ?
>
> Here is an update, follow-up on xenomai-core. I was finally able to
> reproduce this behaviour: I run latency in the background and hit the
> "Enter" key on my serial console, and get high latency figures.
>
> I enabled the tracer, set xenomai latency to 300us and managed to get a
> trace (220us latency). However, I do not understand what is going wrong
> from reading the trace, so I post it here in case someone sees something.
>
> Ah, and I added an ipipe_trace_special in ipipe_grab_irq to log the
> number of the received irq. 1 is serial interrupt 18 (0x12) is timer
> interrupt.
>
> Inline, so that Jan can comment it.
Thanks, but TB is too "smart" - it cuts off everything that is marked as
footer ("--"). :-/
> I-pipe frozen back-tracing service on 2.6.20/ipipe-1.8-04
> ------------------------------------------------------------
> CPU: 0, Freeze: 450692973 cycles, Trace Points: 1000 (+10)
> Calibrated minimum trace-point overhead: 1.000 us
That is interesting. I tells us that we might subtract 1 us
_per_tracepoint_ from the given latencies due to the inherent tracer
overhead. We have about 50 entries in the critical path, so 50 us
compared to 220 us that were measured - roughly 170 us real latency.
What is the clock resolution btw? 500 ns?
So here is the interesting block, starting with the last larger IRQs-on
window.
> : + func -447+ 2.500 xnshadow_relax+0x14 (hisyscall_event+0x210)
> :| + begin 0x80000000 -445+ 3.000 xnshadow_relax+0xd4 (hisyscall_event+0x210)
> :| # func -442+ 5.000 schedule_linux_call+0x10 (xnshadow_relax+0x114)
> :| # func -437+ 4.000 rthal_apc_schedule+0x10 (schedule_linux_call+0x1e8)
> :| # func -433+ 5.000 __ipipe_schedule_irq+0x10 (rthal_apc_schedule+0xac)
> :| # func -428+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
> :| # func -423+ 3.500 rpi_push+0x14 (xnshadow_relax+0x11c)
> :| # func -420+ 5.500 xnpod_suspend_thread+0x14 (xnshadow_relax+0x148)
> :| # func -414+ 4.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
> :| # [ 752] -<?>- 0 -410+ 7.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
> :| # func -403! 56.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
> :| # [ 0] -<?>- -1 -347! 20.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
> :| +func -327+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
> :| +func -324+ 7.500 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
> :| #end 0x80000000 -317+ 7.000 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
OK, the clock starts ticking...
> :| #func -310+ 2.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
> :| #begin 0xffffffff -307+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
> :| #(0x2a) 0x00000012 -300+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
> :| #func -295+ 4.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
> :| #func -291+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
> :| #func -289+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
> :| #func -286+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
> :| #func -284+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
(Without looking at the arm code: Is this double invocation of
at91_aic_mask_irq correct and required?)
> :| #func -282+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
> :| #func -278+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
> :| #func -276+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
> :| #func -273+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
> :| #*func -270+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
> :| #*func -266+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
> :| #*func -257+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
> :| #*func -253+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
> :| #*[ 753] -<?>- 99 -249! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
Hmm, comparably costly, this simple resume. Hope it's not the
instrumentation (ipipe_trace_pid?) itself.
> :| #*func -234+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
> :| #*func -228+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
> :| #*func -224+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
> :| #*[ 0] -<?>- -1 -220! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
OK, this is the cache flushing thing, I guess. Expected.
> :| #*[ 753] -<?>- 99 -161! 17.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
Again this peak. Strange.
> :| #*func -143+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
> :| #*func -139+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
> :| +*end 0x80000000 -135+ 7.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
So we woke up the thread and are on the way back to userland. But then
the serial IRQ hits us...
> :| +*func -128+ 3.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
> :| +*begin 0xffffffff -125+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
> :| +*(0x2a) 0x00000001 -118+ 4.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
> :| +*func -114+ 5.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
> :| +*func -109+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_handle_irq+0xdc)
> :| +*func -105+ 2.000 __ipipe_ack_irq+0x10 (__ipipe_handle_irq+0xfc)
> :| +*func -103+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_irq+0x28)
> :| +*func -100+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
> :| +*func -97+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
> :| +*func -95+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
Serial done, the above 30 us (minus tracer overhead) are unavoidable.
> :| +*end 0xffffffff -92! 10.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
> :| +*begin 0x80000001 -81+ 4.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
This is the syscall epilogue related to wait_thread_period.
> :| +*end 0x80000001 -77! 39.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
> : +*func -38+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
After doing some latency calculation in user land, we are back to freeze
the trace now.
> : +*func -35+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
> :| +*begin 0x80000001 -33+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
> :| +*end 0x80000001 -29+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
> : +*func -27+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
> : +*func -21+ 2.500 xnshadow_sys_trace+0x10 (hisyscall_event+0x150)
> : +*func -18+ 2.000 ipipe_trace_frozen_reset+0x10 (xnshadow_sys_trace+0x98)
> : +*func -16+ 2.000 __ipipe_global_path_lock+0x10 (ipipe_trace_frozen_reset+0x14)
> : +*func -14+ 2.000 __ipipe_spin_lock_irqsave+0x10 (__ipipe_global_path_lock+0x18)
> :| +*begin 0x80000001 -12+ 8.000 __ipipe_spin_lock_irqsave+0x34 (__ipipe_global_path_lock+0x18)
> :| #*func -4+ 2.500 __ipipe_spin_unlock_irqcomplete+0x10 (__ipipe_global_path_unlock+0x6c)
> :| +*end 0x80000001 -2+ 2.000 __ipipe_spin_unlock_irqcomplete+0x4c (__ipipe_global_path_unlock+0x6c)
> < +*freeze 0x00035ea1 0 2.500 xnshadow_sys_trace+0xa4 (hisyscall_event+0x150)
So, all in all, nothing fishy, at least here. Any IRQ (actually even
multiple!) can hit us between wakeup and latency calculation, and this
happened here due to sending some characters over the serial console.
Plus we see the cache flushing overhead when switching processes, also
normal.
I would just check those small peaks around ipipe_trace_pid, if they are
just tracing related) (e.g. by disabling those instrumentations in the
nucleus).
Jan
--
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-02-11 13:41 ` Jan Kiszka
@ 2008-02-11 14:05 ` Gilles Chanteperdrix
2008-02-11 17:11 ` Jan Kiszka
2008-02-12 7:04 ` Gilles Chanteperdrix
1 sibling, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-02-11 14:05 UTC (permalink / raw)
To: Jan Kiszka; +Cc: jagarcia, xenomai
On Mon, Feb 11, 2008 at 2:41 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>
> Gilles Chanteperdrix wrote:
> > Juan Antonio Garcia Redondo wrote:
> > > On 23/01/08 14:15, Gilles Chanteperdrix wrote:
> > > > On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
> > > > <gilles.chanteperdrix@xenomai.org> wrote:
> > > > > On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
> > > > >
> > > > > <juan-antonio.garcia@domain.hid> wrote:
> > > > > > I see everything OK except for the first samples of cyclictests. Any comments ?
> > > > >
> > > > > The load you apply does not load the cache, which is a source of
> > > > > jitter. You should run the cache calibrator (I do not find the cache
> > > > > calibrator URL, but it is somewhere in Xenomai distribution or wiki).
> > > >
> > > > It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
> > > >
> > > > --
> > > > Gilles Chanteperdrix
> > >
> > > Thanks Gilles, I've done more tests using the cache calibrator from
> > > http://www.cwi.nl/~manegold/Calibrator. The latency numbers are very
> > > similar althought I've found an strange behaviour related to telnet
> > > sessions.
> > >
> > > Environment:
> > > o Tests running from console over atmel serial port.
> > > o A telnet session over on-chip ethernet.
> > > o System without load.
> > >
> > > ./latency -p 500 -t0
> > > == All results in microseconds
> > > warming up...
> > > RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
> > > RTH|-RTH----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat
> > > worst
> > > RTD| 49.613| 52.190| 62.822| 0| 49.613| 62.822
> > > RTD| 42.203| 52.512| 66.365| 0| 42.203| 66.365
> > >
> > >
> > > Now If hit a key on the telnet session :
> > >
> > > RTD| 36.726| 57.989| 109.536| 0| 31.572| 109.536 <-------- Here I've hit the key.
> > > RTD| 36.404| 51.868| 69.587| 0| 31.572| 109.536
> > > RTD| 35.760| 51.868| 73.775| 0| 31.572| 109.536
> > >
> > > Now, I launch an script which executes four instances of cache
> > > calibrator.
> > >
> > > RTD| 45.103| 57.667| 75.708| 0| 32.538| 122.422
> > > RTD| 45.425| 57.023| 76.030| 0| 32.538| 122.422
> > > RTD| 46.069| 57.023| 75.708| 0| 32.538| 122.422
> > >
> > > Now, I can hit a key on the telnet session without effects over latency
> > > numbers:
> > >
> > > RTD| 44.136| 57.989| 75.386| 0| 27.384| 128.221
> > > RTD| 46.713| 57.345| 76.353| 0| 27.384| 128.221
> > > RTD| 44.780| 57.345| 76.675| 0| 27.384| 128.221
> > > RTD| 43.492| 56.701| 76.997| 0| 27.384| 128.221
> > >
> > > Now I stop the calibrator process and launch 'ping -f -s2048 192.168.2.82' from an external
> > > machine.
> > >
> > > RTD| 40.270| 68.621| 90.850| 0| 27.384| 128.221
> > > RTD| 36.082| 68.621| 88.273| 0| 27.384| 128.221
> > > RTD| 40.592| 67.976| 91.494| 0| 27.384| 128.221
> > > RTD| 41.237| 68.298| 89.239| 0| 27.384| 128.221
> > >
> > >
> > > Now If hit a key on the telnet session :
> > >
> > > RTD| 42.203| 67.976| 88.273| 0| 27.384| 128.221
> > > RTD| 32.216| 93.427| 128.543| 0| 27.384| 128.543 <---------- Here I've hit the key.
> > > RTD| 42.203| 68.298| 87.628| 0| 27.384| 128.543
> > >
> > > And again the calibrator execution results on eliminate the strange
> > > behaviour whith the telnet session.
> > >
> > > Any clues ?
> >
> > Here is an update, follow-up on xenomai-core. I was finally able to
> > reproduce this behaviour: I run latency in the background and hit the
> > "Enter" key on my serial console, and get high latency figures.
> >
> > I enabled the tracer, set xenomai latency to 300us and managed to get a
> > trace (220us latency). However, I do not understand what is going wrong
> > from reading the trace, so I post it here in case someone sees something.
> >
> > Ah, and I added an ipipe_trace_special in ipipe_grab_irq to log the
> > number of the received irq. 1 is serial interrupt 18 (0x12) is timer
> > interrupt.
> >
> > Inline, so that Jan can comment it.
>
> Thanks, but TB is too "smart" - it cuts off everything that is marked as
> footer ("--"). :-/
>
>
> > I-pipe frozen back-tracing service on 2.6.20/ipipe-1.8-04
> > ------------------------------------------------------------
> > CPU: 0, Freeze: 450692973 cycles, Trace Points: 1000 (+10)
> > Calibrated minimum trace-point overhead: 1.000 us
>
> That is interesting. I tells us that we might subtract 1 us
> _per_tracepoint_ from the given latencies due to the inherent tracer
> overhead. We have about 50 entries in the critical path, so 50 us
> compared to 220 us that were measured - roughly 170 us real latency.
>
> What is the clock resolution btw? 500 ns?
It should be around 1us. A bit less. I think the frequency is 92 MHz /
64. Anyway, I checked once and the I-pipe rounding is really
approximative, it is equivalent to rounding frequency to 2 MHz.
>
> So here is the interesting block, starting with the last larger IRQs-on
> window.
>
>
> > : + func -447+ 2.500 xnshadow_relax+0x14 (hisyscall_event+0x210)
> > :| + begin 0x80000000 -445+ 3.000 xnshadow_relax+0xd4 (hisyscall_event+0x210)
> > :| # func -442+ 5.000 schedule_linux_call+0x10 (xnshadow_relax+0x114)
> > :| # func -437+ 4.000 rthal_apc_schedule+0x10 (schedule_linux_call+0x1e8)
> > :| # func -433+ 5.000 __ipipe_schedule_irq+0x10 (rthal_apc_schedule+0xac)
> > :| # func -428+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
> > :| # func -423+ 3.500 rpi_push+0x14 (xnshadow_relax+0x11c)
> > :| # func -420+ 5.500 xnpod_suspend_thread+0x14 (xnshadow_relax+0x148)
> > :| # func -414+ 4.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
> > :| # [ 752] -<?>- 0 -410+ 7.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
> > :| # func -403! 56.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
> > :| # [ 0] -<?>- -1 -347! 20.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
> > :| +func -327+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
> > :| +func -324+ 7.500 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
> > :| #end 0x80000000 -317+ 7.000 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
>
> OK, the clock starts ticking...
>
>
> > :| #func -310+ 2.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
> > :| #begin 0xffffffff -307+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
> > :| #(0x2a) 0x00000012 -300+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
> > :| #func -295+ 4.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
> > :| #func -291+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
> > :| #func -289+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
> > :| #func -286+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
> > :| #func -284+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
>
> (Without looking at the arm code: Is this double invocation of
> at91_aic_mask_irq correct and required?)
No, and I do not see where it is coming from.
>
>
> > :| #func -282+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
> > :| #func -278+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
> > :| #func -276+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
> > :| #func -273+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
> > :| #*func -270+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
> > :| #*func -266+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
> > :| #*func -257+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
> > :| #*func -253+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
> > :| #*[ 753] -<?>- 99 -249! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
>
> Hmm, comparably costly, this simple resume. Hope it's not the
> instrumentation (ipipe_trace_pid?) itself.
>
>
> > :| #*func -234+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
> > :| #*func -228+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
> > :| #*func -224+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
> > :| #*[ 0] -<?>- -1 -220! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
>
> OK, this is the cache flushing thing, I guess. Expected.
Yes, I took care of disabling the (still recent) irqs unmasking in
xnpod_schedule. So, we see the cache flushing with irqs disabled.
>
>
> > :| #*[ 753] -<?>- 99 -161! 17.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
>
> Again this peak. Strange.
>
>
> > :| #*func -143+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
> > :| #*func -139+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
> > :| +*end 0x80000000 -135+ 7.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
>
> So we woke up the thread and are on the way back to userland. But then
> the serial IRQ hits us...
>
>
> > :| +*func -128+ 3.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
> > :| +*begin 0xffffffff -125+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
> > :| +*(0x2a) 0x00000001 -118+ 4.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
> > :| +*func -114+ 5.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
> > :| +*func -109+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_handle_irq+0xdc)
> > :| +*func -105+ 2.000 __ipipe_ack_irq+0x10 (__ipipe_handle_irq+0xfc)
> > :| +*func -103+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_irq+0x28)
> > :| +*func -100+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
> > :| +*func -97+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
> > :| +*func -95+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
>
> Serial done, the above 30 us (minus tracer overhead) are unavoidable.
>
>
> > :| +*end 0xffffffff -92! 10.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
> > :| +*begin 0x80000001 -81+ 4.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
>
> This is the syscall epilogue related to wait_thread_period.
Is it normal to see the hw irqs disabled here ?
>
>
> > :| +*end 0x80000001 -77! 39.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
> > : +*func -38+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
>
> After doing some latency calculation in user land, we are back to freeze
> the trace now.
>
>
> > : +*func -35+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
> > :| +*begin 0x80000001 -33+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
> > :| +*end 0x80000001 -29+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
> > : +*func -27+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
> > : +*func -21+ 2.500 xnshadow_sys_trace+0x10 (hisyscall_event+0x150)
> > : +*func -18+ 2.000 ipipe_trace_frozen_reset+0x10 (xnshadow_sys_trace+0x98)
> > : +*func -16+ 2.000 __ipipe_global_path_lock+0x10 (ipipe_trace_frozen_reset+0x14)
> > : +*func -14+ 2.000 __ipipe_spin_lock_irqsave+0x10 (__ipipe_global_path_lock+0x18)
> > :| +*begin 0x80000001 -12+ 8.000 __ipipe_spin_lock_irqsave+0x34 (__ipipe_global_path_lock+0x18)
> > :| #*func -4+ 2.500 __ipipe_spin_unlock_irqcomplete+0x10 (__ipipe_global_path_unlock+0x6c)
> > :| +*end 0x80000001 -2+ 2.000 __ipipe_spin_unlock_irqcomplete+0x4c (__ipipe_global_path_unlock+0x6c)
> > < +*freeze 0x00035ea1 0 2.500 xnshadow_sys_trace+0xa4 (hisyscall_event+0x150)
>
> So, all in all, nothing fishy, at least here. Any IRQ (actually even
> multiple!) can hit us between wakeup and latency calculation, and this
> happened here due to sending some characters over the serial console.
> Plus we see the cache flushing overhead when switching processes, also
> normal.
The problem is that the nucleus latency is calibrated to 300 us (which
is very high), and that we get a 220us latency, which means that it
took 500us going from the timer interrupt to the xntrace syscall.
>
> I would just check those small peaks around ipipe_trace_pid, if they are
> just tracing related) (e.g. by disabling those instrumentations in the
> nucleus).
Will try that.
--
Gilles Chanteperdrix
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-02-11 14:05 ` Gilles Chanteperdrix
@ 2008-02-11 17:11 ` Jan Kiszka
2008-02-11 21:30 ` Gilles Chanteperdrix
2008-02-11 22:36 ` Gilles Chanteperdrix
0 siblings, 2 replies; 27+ messages in thread
From: Jan Kiszka @ 2008-02-11 17:11 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: jagarcia, xenomai
Gilles Chanteperdrix wrote:
> On Mon, Feb 11, 2008 at 2:41 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>> Gilles Chanteperdrix wrote:
>> > Juan Antonio Garcia Redondo wrote:
>> > > On 23/01/08 14:15, Gilles Chanteperdrix wrote:
>> > > > On Jan 23, 2008 11:04 AM, Gilles Chanteperdrix
>> > > > <gilles.chanteperdrix@xenomai.org> wrote:
>> > > > > On Jan 23, 2008 7:52 AM, Juan Antonio Garcia Redondo
>> > > > >
>> > > > > <juan-antonio.garcia@domain.hid> wrote:
>> > > > > > I see everything OK except for the first samples of cyclictests. Any comments ?
>> > > > >
>> > > > > The load you apply does not load the cache, which is a source of
>> > > > > jitter. You should run the cache calibrator (I do not find the cache
>> > > > > calibrator URL, but it is somewhere in Xenomai distribution or wiki).
>> > > >
>> > > > It is in the TROUBLESHOOTING guide, question "How do I adequately stress test".
>> > > >
>> > > > --
>> > > > Gilles Chanteperdrix
>> > >
>> > > Thanks Gilles, I've done more tests using the cache calibrator from
>> > > http://www.cwi.nl/~manegold/Calibrator. The latency numbers are very
>> > > similar althought I've found an strange behaviour related to telnet
>> > > sessions.
>> > >
>> > > Environment:
>> > > o Tests running from console over atmel serial port.
>> > > o A telnet session over on-chip ethernet.
>> > > o System without load.
>> > >
>> > > ./latency -p 500 -t0
>> > > == All results in microseconds
>> > > warming up...
>> > > RTT| 00:00:01 (periodic user-mode task, 500 us period, priority 99)
>> > > RTH|-RTH----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat
>> > > worst
>> > > RTD| 49.613| 52.190| 62.822| 0| 49.613| 62.822
>> > > RTD| 42.203| 52.512| 66.365| 0| 42.203| 66.365
>> > >
>> > >
>> > > Now If hit a key on the telnet session :
>> > >
>> > > RTD| 36.726| 57.989| 109.536| 0| 31.572| 109.536 <-------- Here I've hit the key.
>> > > RTD| 36.404| 51.868| 69.587| 0| 31.572| 109.536
>> > > RTD| 35.760| 51.868| 73.775| 0| 31.572| 109.536
>> > >
>> > > Now, I launch an script which executes four instances of cache
>> > > calibrator.
>> > >
>> > > RTD| 45.103| 57.667| 75.708| 0| 32.538| 122.422
>> > > RTD| 45.425| 57.023| 76.030| 0| 32.538| 122.422
>> > > RTD| 46.069| 57.023| 75.708| 0| 32.538| 122.422
>> > >
>> > > Now, I can hit a key on the telnet session without effects over latency
>> > > numbers:
>> > >
>> > > RTD| 44.136| 57.989| 75.386| 0| 27.384| 128.221
>> > > RTD| 46.713| 57.345| 76.353| 0| 27.384| 128.221
>> > > RTD| 44.780| 57.345| 76.675| 0| 27.384| 128.221
>> > > RTD| 43.492| 56.701| 76.997| 0| 27.384| 128.221
>> > >
>> > > Now I stop the calibrator process and launch 'ping -f -s2048 192.168.2.82' from an external
>> > > machine.
>> > >
>> > > RTD| 40.270| 68.621| 90.850| 0| 27.384| 128.221
>> > > RTD| 36.082| 68.621| 88.273| 0| 27.384| 128.221
>> > > RTD| 40.592| 67.976| 91.494| 0| 27.384| 128.221
>> > > RTD| 41.237| 68.298| 89.239| 0| 27.384| 128.221
>> > >
>> > >
>> > > Now If hit a key on the telnet session :
>> > >
>> > > RTD| 42.203| 67.976| 88.273| 0| 27.384| 128.221
>> > > RTD| 32.216| 93.427| 128.543| 0| 27.384| 128.543 <---------- Here I've hit the key.
>> > > RTD| 42.203| 68.298| 87.628| 0| 27.384| 128.543
>> > >
>> > > And again the calibrator execution results on eliminate the strange
>> > > behaviour whith the telnet session.
>> > >
>> > > Any clues ?
>> >
>> > Here is an update, follow-up on xenomai-core. I was finally able to
>> > reproduce this behaviour: I run latency in the background and hit the
>> > "Enter" key on my serial console, and get high latency figures.
>> >
>> > I enabled the tracer, set xenomai latency to 300us and managed to get a
>> > trace (220us latency). However, I do not understand what is going wrong
>> > from reading the trace, so I post it here in case someone sees something.
>> >
>> > Ah, and I added an ipipe_trace_special in ipipe_grab_irq to log the
>> > number of the received irq. 1 is serial interrupt 18 (0x12) is timer
>> > interrupt.
>> >
>> > Inline, so that Jan can comment it.
>>
>> Thanks, but TB is too "smart" - it cuts off everything that is marked as
>> footer ("--"). :-/
>>
>>
>> > I-pipe frozen back-tracing service on 2.6.20/ipipe-1.8-04
>> > ------------------------------------------------------------
>> > CPU: 0, Freeze: 450692973 cycles, Trace Points: 1000 (+10)
>> > Calibrated minimum trace-point overhead: 1.000 us
>>
>> That is interesting. I tells us that we might subtract 1 us
>> _per_tracepoint_ from the given latencies due to the inherent tracer
>> overhead. We have about 50 entries in the critical path, so 50 us
>> compared to 220 us that were measured - roughly 170 us real latency.
>>
>> What is the clock resolution btw? 500 ns?
>
> It should be around 1us. A bit less. I think the frequency is 92 MHz /
> 64. Anyway, I checked once and the I-pipe rounding is really
> approximative, it is equivalent to rounding frequency to 2 MHz.
>
>> So here is the interesting block, starting with the last larger IRQs-on
>> window.
>>
>>
>> > : + func -447+ 2.500 xnshadow_relax+0x14 (hisyscall_event+0x210)
>> > :| + begin 0x80000000 -445+ 3.000 xnshadow_relax+0xd4 (hisyscall_event+0x210)
>> > :| # func -442+ 5.000 schedule_linux_call+0x10 (xnshadow_relax+0x114)
>> > :| # func -437+ 4.000 rthal_apc_schedule+0x10 (schedule_linux_call+0x1e8)
>> > :| # func -433+ 5.000 __ipipe_schedule_irq+0x10 (rthal_apc_schedule+0xac)
>> > :| # func -428+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
>> > :| # func -423+ 3.500 rpi_push+0x14 (xnshadow_relax+0x11c)
>> > :| # func -420+ 5.500 xnpod_suspend_thread+0x14 (xnshadow_relax+0x148)
>> > :| # func -414+ 4.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
>> > :| # [ 752] -<?>- 0 -410+ 7.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
>> > :| # func -403! 56.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
>> > :| # [ 0] -<?>- -1 -347! 20.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
>> > :| +func -327+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
>> > :| +func -324+ 7.500 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
>> > :| #end 0x80000000 -317+ 7.000 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
>>
>> OK, the clock starts ticking...
>>
>>
>> > :| #func -310+ 2.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
>> > :| #begin 0xffffffff -307+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
>> > :| #(0x2a) 0x00000012 -300+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
>> > :| #func -295+ 4.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
>> > :| #func -291+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
>> > :| #func -289+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
>> > :| #func -286+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
>> > :| #func -284+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
>>
>> (Without looking at the arm code: Is this double invocation of
>> at91_aic_mask_irq correct and required?)
>
> No, and I do not see where it is coming from.
>
>>
>> > :| #func -282+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
>> > :| #func -278+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
>> > :| #func -276+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
>> > :| #func -273+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
>> > :| #*func -270+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
>> > :| #*func -266+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
>> > :| #*func -257+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
>> > :| #*func -253+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
>> > :| #*[ 753] -<?>- 99 -249! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
>>
>> Hmm, comparably costly, this simple resume. Hope it's not the
>> instrumentation (ipipe_trace_pid?) itself.
>>
>>
>> > :| #*func -234+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
>> > :| #*func -228+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
>> > :| #*func -224+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
>> > :| #*[ 0] -<?>- -1 -220! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
>>
>> OK, this is the cache flushing thing, I guess. Expected.
>
> Yes, I took care of disabling the (still recent) irqs unmasking in
> xnpod_schedule. So, we see the cache flushing with irqs disabled.
>
>>
>> > :| #*[ 753] -<?>- 99 -161! 17.500 xnpod_schedule+0xb14 (xnpod_suspend_thread+0x60c)
>>
>> Again this peak. Strange.
>>
>>
>> > :| #*func -143+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
>> > :| #*func -139+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
>> > :| +*end 0x80000000 -135+ 7.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
>>
>> So we woke up the thread and are on the way back to userland. But then
>> the serial IRQ hits us...
>>
>>
>> > :| +*func -128+ 3.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
>> > :| +*begin 0xffffffff -125+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
>> > :| +*(0x2a) 0x00000001 -118+ 4.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
>> > :| +*func -114+ 5.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
>> > :| +*func -109+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_handle_irq+0xdc)
>> > :| +*func -105+ 2.000 __ipipe_ack_irq+0x10 (__ipipe_handle_irq+0xfc)
>> > :| +*func -103+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_irq+0x28)
>> > :| +*func -100+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
>> > :| +*func -97+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
>> > :| +*func -95+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
>>
>> Serial done, the above 30 us (minus tracer overhead) are unavoidable.
>>
>>
>> > :| +*end 0xffffffff -92! 10.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
>> > :| +*begin 0x80000001 -81+ 4.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
>>
>> This is the syscall epilogue related to wait_thread_period.
>
> Is it normal to see the hw irqs disabled here ?
They were likely only disabled at the instrumentation point, but were
enabled shortly afterwards - just to be disabled again.
>
>>
>> > :| +*end 0x80000001 -77! 39.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
>> > : +*func -38+ 2.500 __ipipe_syscall_root+0x10 (vector_swi+0x68)
>>
>> After doing some latency calculation in user land, we are back to freeze
>> the trace now.
>>
>>
>> > : +*func -35+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
>> > :| +*begin 0x80000001 -33+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
>> > :| +*end 0x80000001 -29+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
>> > : +*func -27+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
>> > : +*func -21+ 2.500 xnshadow_sys_trace+0x10 (hisyscall_event+0x150)
>> > : +*func -18+ 2.000 ipipe_trace_frozen_reset+0x10 (xnshadow_sys_trace+0x98)
>> > : +*func -16+ 2.000 __ipipe_global_path_lock+0x10 (ipipe_trace_frozen_reset+0x14)
>> > : +*func -14+ 2.000 __ipipe_spin_lock_irqsave+0x10 (__ipipe_global_path_lock+0x18)
>> > :| +*begin 0x80000001 -12+ 8.000 __ipipe_spin_lock_irqsave+0x34 (__ipipe_global_path_lock+0x18)
>> > :| #*func -4+ 2.500 __ipipe_spin_unlock_irqcomplete+0x10 (__ipipe_global_path_unlock+0x6c)
>> > :| +*end 0x80000001 -2+ 2.000 __ipipe_spin_unlock_irqcomplete+0x4c (__ipipe_global_path_unlock+0x6c)
>> > < +*freeze 0x00035ea1 0 2.500 xnshadow_sys_trace+0xa4 (hisyscall_event+0x150)
>>
>> So, all in all, nothing fishy, at least here. Any IRQ (actually even
>> multiple!) can hit us between wakeup and latency calculation, and this
>> happened here due to sending some characters over the serial console.
>> Plus we see the cache flushing overhead when switching processes, also
>> normal.
>
> The problem is that the nucleus latency is calibrated to 300 us (which
> is very high), and that we get a 220us latency, which means that it
> took 500us going from the timer interrupt to the xntrace syscall.
There is a shadow relax procedure running before the timer IRQ fires,
and that takes another context switch. So the latency sum is:
- unrelated context switch
- timer IRQ
- switch to woken up RT process
- serial IRQ
Almost the theoretical worst case.
Jan
--
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-02-11 17:11 ` Jan Kiszka
@ 2008-02-11 21:30 ` Gilles Chanteperdrix
2008-02-11 22:36 ` Gilles Chanteperdrix
1 sibling, 0 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-02-11 21:30 UTC (permalink / raw)
To: Jan Kiszka; +Cc: jagarcia, xenomai
Jan Kiszka wrote:
> There is a shadow relax procedure running before the timer IRQ fires,
> and that takes another context switch. So the latency sum is:
>
> - unrelated context switch
> - timer IRQ
> - switch to woken up RT process
> - serial IRQ
>
> Almost the theoretical worst case.
The problem is that it does not happen when launching latency with -t 1
or -t 2, or if latency is not run in the background.
Anyway, I will investigate on the double ack issue, because it could
mean that there is a double EOI, which, I guess could cause some
troubles at interrupt controller level.
--
Gilles Chanteperdrix.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-02-11 17:11 ` Jan Kiszka
2008-02-11 21:30 ` Gilles Chanteperdrix
@ 2008-02-11 22:36 ` Gilles Chanteperdrix
1 sibling, 0 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-02-11 22:36 UTC (permalink / raw)
To: Jan Kiszka; +Cc: jagarcia, xenomai
Jan Kiszka wrote:
> There is a shadow relax procedure running before the timer IRQ fires,
> and that takes another context switch. So the latency sum is:
>
> - unrelated context switch
> - timer IRQ
> - switch to woken up RT process
> - serial IRQ
>
> Almost the theoretical worst case.
Here is another trace, I get traces similar to this one everytime I
capture the effect of hitting the "enter" key while latency is running
in the background. On this one, we can measure that the timer ticks
every 720 pseudo us (since latency is launched with -p 1000, so we see
that the I-pipe rounding is very approximative), except for the last
tick which is around 100us late, without a serial interrupt. We see a
strange spot around -415.
RTS| 65.972| 106.250| 415.277| 0| 00:00:14/00:00:14
# cat /proc/ipipe/trace/frozen
I-pipe frozen back-tracing service on 2.6.20/ipipe-1.8-04
------------------------------------------------------------
CPU: 0, Freeze: 112815123 cycles, Trace Points: 1000 (+10)
Calibrated minimum trace-point overhead: 1.000 us
+----- Hard IRQs ('|': locked)
|+---- <unused>
||+--- <unused>
|||+-- Xenomai
||||+- Linux ('*': domain stalled, '+': current, '#': current+stalled)
||||| +---------- Delay flag ('+': > 1 us, '!': > 10 us)
||||| | +- NMI noise ('N')
||||| | |
Type User Val. Time Delay Function (Parent)
: + func -10974+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -10972+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -10968+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -10966+ 7.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -10959+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -10957+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -10955+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -10952+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -10949+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -10944+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -10942+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -10937! 51.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -10885! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -10865+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -10858+ 3.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -10855+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -10850+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -10847+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -10841+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -10838+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -10836+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -10833+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -10832+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -10830+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -10828+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -10827! 381.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -10446+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -10444+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -10442+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -10437+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -10435+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -10432+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -10430+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -10427+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -10425+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -10421+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -10420+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -10417+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -10413+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -10410+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -10405+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -10402+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -10398+ 8.000 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -10390+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -10384+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -10380+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -10377! 52.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -10324! 18.000 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -10306+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -10302+ 4.500 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -10298! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -10286+ 7.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -10279! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -10258+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -10255+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -10253+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -10249+ 2.500 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -10247+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -10240+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -10238+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -10236+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -10233+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -10230+ 4.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -10225+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -10223+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -10218! 51.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -10166! 19.500 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -10147+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -10140+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -10137+ 5.000 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -10132+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -10129+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -10123+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -10120+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -10117+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -10115 1.000 default_idle+0x64 (cpu_idle+0x58)
: #func -10114+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -10112+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -10110+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -10108! 382.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -9726+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -9724+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -9722+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -9717+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -9715+ 3.000 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -9712+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -9709+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -9707+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -9705+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -9701+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -9699+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -9697+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -9693+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -9690+ 6.000 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -9684+ 2.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -9681+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -9678+ 9.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -9668+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -9662+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -9658+ 2.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -9655! 53.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -9602! 18.000 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -9584+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -9580+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -9576! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -9564+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -9556! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -9536+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -9533+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -9531+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -9527+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -9525+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -9518+ 2.500 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -9516+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -9513+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -9511+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -9508+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -9503+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -9500+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -9495! 51.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -9444! 16.500 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| # func -9427+ 2.500 rthal_irq_host_pend+0x10 (xnintr_clock_handler+0xbc)
:| # func -9425+ 4.000 __ipipe_schedule_irq+0x10 (rthal_irq_host_pend+0x20)
:| # func -9421+ 7.500 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
:| +func -9413+ 4.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +func -9409+ 7.000 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
:| #end 0x80000000 -9402+ 2.500 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
: #func -9400+ 3.500 asm_do_IRQ+0x10 (__ipipe_sync_stage+0x284)
: #func -9396+ 2.000 handle_level_irq+0x10 (asm_do_IRQ+0x50)
: #func -9394+ 2.500 _spin_lock+0x10 (handle_level_irq+0x24)
: #func -9392+ 3.000 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -9389+ 5.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -9384+ 2.500 _spin_unlock+0x10 (handle_level_irq+0x88)
: #func -9381+ 3.000 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -9378+ 2.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -9376+ 3.000 handle_IRQ_event+0x10 (handle_level_irq+0x94)
: #func -9373+ 5.500 at91_timer_interrupt+0x10 (handle_IRQ_event+0x3c)
: #func -9368+ 1.500 _spin_lock+0x10 (at91_timer_interrupt+0x1c)
: #func -9366+ 2.000 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -9364+ 3.500 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -9361+ 2.000 timer_tick+0x10 (at91_timer_interrupt+0x50)
: #func -9359+ 6.500 profile_tick+0x10 (timer_tick+0x18)
: #func -9352+ 6.500 do_timer+0x14 (timer_tick+0xe0)
: #func -9346+ 7.500 current_tick_length+0x10 (do_timer+0x144)
: #func -9338+ 2.500 update_process_times+0x10 (timer_tick+0xfc)
: #func -9336+ 5.000 account_system_time+0x10 (update_process_times+0x44)
: #func -9331+ 2.000 run_local_timers+0x10 (update_process_times+0x48)
: #func -9329+ 2.500 raise_softirq+0x10 (run_local_timers+0x18)
:| #begin 0x80000001 -9326+ 2.000 raise_softirq+0x34 (run_local_timers+0x18)
:| #end 0x80000001 -9324+ 3.500 raise_softirq+0x54 (run_local_timers+0x18)
: #func -9321+ 2.000 __ipipe_restore_root+0x10 (raise_softirq+0xb4)
:| #begin 0x80000001 -9319+ 2.000 __ipipe_restore_root+0x38 (raise_softirq+0xb4)
:| #end 0x80000001 -9317+ 2.500 __ipipe_restore_root+0x58 (raise_softirq+0xb4)
: #func -9314+ 4.500 softlockup_tick+0x14 (run_local_timers+0x1c)
: #func -9310+ 2.500 rcu_pending+0x10 (update_process_times+0x50)
: #func -9307+ 3.000 __rcu_pending+0x10 (rcu_pending+0x1c)
: #func -9304+ 3.500 __rcu_pending+0x10 (rcu_pending+0x34)
: #func -9301+ 1.500 scheduler_tick+0x10 (update_process_times+0x64)
: #func -9299+ 4.500 sched_clock+0x10 (scheduler_tick+0x14)
: #func -9295+ 4.500 run_posix_cpu_timers+0x14 (update_process_times+0x6c)
: #func -9290+ 1.500 _spin_unlock+0x10 (at91_timer_interrupt+0xc0)
: #func -9289+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -9287+ 3.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
:| #begin 0x80000001 -9284+ 2.000 handle_IRQ_event+0x84 (handle_level_irq+0x94)
:| #end 0x80000001 -9282+ 2.500 handle_IRQ_event+0xa4 (handle_level_irq+0x94)
: #func -9280+ 3.500 note_interrupt+0x14 (handle_level_irq+0xb0)
: #func -9276+ 1.500 _spin_lock+0x10 (handle_level_irq+0xb8)
: #func -9275+ 1.500 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -9273+ 3.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -9270+ 2.500 at91_aic_unmask_irq+0x10 (handle_level_irq+0xe4)
: #func -9268+ 1.500 _spin_unlock+0x10 (handle_level_irq+0xec)
: #func -9266+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -9265+ 2.500 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -9262+ 2.500 irq_exit+0x10 (asm_do_IRQ+0x54)
: #func -9260+ 3.000 __do_softirq+0x10 (irq_exit+0x44)
: #func -9257+ 1.500 __ipipe_unstall_root+0x10 (__do_softirq+0x40)
:| #begin 0x80000000 -9255+ 2.000 __ipipe_unstall_root+0x30 (__do_softirq+0x40)
:| +end 0x80000000 -9253+ 3.000 __ipipe_unstall_root+0x64 (__do_softirq+0x40)
: +func -9250+ 3.000 run_timer_softirq+0x14 (__do_softirq+0x58)
: +func -9247+ 5.500 hrtimer_run_queues+0x10 (run_timer_softirq+0x24)
: +func -9242+ 2.000 _spin_lock_irq+0x10 (run_timer_softirq+0x40)
:| +begin 0x80000001 -9240+ 2.500 _spin_lock_irq+0x34 (run_timer_softirq+0x40)
:| #end 0x80000001 -9237+ 1.500 _spin_lock_irq+0x54 (run_timer_softirq+0x40)
: #func -9236+ 1.500 ipipe_check_context+0x10 (_spin_lock_irq+0x60)
: #func -9234+ 4.500 _raw_spin_lock+0x14 (_spin_lock_irq+0x68)
: #func -9230+ 1.500 _spin_unlock_irq+0x10 (run_timer_softirq+0x1ac)
: #func -9228+ 2.000 _raw_spin_unlock+0x10 (_spin_unlock_irq+0x14)
: #func -9226+ 1.500 __ipipe_unstall_root+0x10 (_spin_unlock_irq+0x18)
:| #begin 0x80000000 -9225+ 2.000 __ipipe_unstall_root+0x30 (_spin_unlock_irq+0x18)
:| +end 0x80000000 -9223+ 2.000 __ipipe_unstall_root+0x64 (_spin_unlock_irq+0x18)
: +func -9221+ 2.000 ipipe_check_context+0x10 (_spin_unlock_irq+0x20)
:| +begin 0x80000001 -9219+ 2.000 __do_softirq+0x90 (irq_exit+0x44)
:| #end 0x80000001 -9217+ 2.500 __do_softirq+0xb0 (irq_exit+0x44)
: #func -9214+ 3.000 _local_bh_enable+0x10 (__do_softirq+0xe8)
: #func -9211+ 2.500 ipipe_check_context+0x10 (irq_exit+0x4c)
:| #begin 0x80000000 -9209+ 5.500 __ipipe_sync_stage+0x31c (__ipipe_walk_pipeline+0xa8)
:| +end 0xffffffff -9203+ 2.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -9201+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -9196+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -9195+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -9193+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -9190+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -9188+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -9185+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -9184+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -9182+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -9180+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -9178! 172.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -9006+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -9004+ 2.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -9002+ 4.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -8998+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -8995+ 2.000 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -8993+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -8991+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -8989+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -8987+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -8983+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -8981+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -8979+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -8975+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -8972+ 6.000 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -8966+ 2.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -8964+ 3.000 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -8961+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -8952+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -8946+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -8942+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -8939! 57.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -8881! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -8863+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -8859+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -8855! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -8843+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8835! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -8815+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -8811+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -8809+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8806+ 2.500 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -8803+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -8797+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -8795+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -8792+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -8790+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -8787+ 4.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -8782+ 3.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -8779+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -8774! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -8723! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -8703+ 7.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -8696+ 3.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -8692+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -8688+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -8685+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -8679+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -8676+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -8673+ 2.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -8671+ 2.000 default_idle+0x64 (cpu_idle+0x58)
: #func -8669+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -8668+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -8666+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -8664! 378.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -8286+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -8284+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -8282+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -8278+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -8275+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -8272+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -8270+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -8268+ 2.500 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -8265+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -8262+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -8260+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -8257+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -8254+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -8250+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -8245+ 2.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -8242+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -8239+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -8230+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -8224+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -8220+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -8217! 53.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -8164! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -8145+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -8141+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -8137! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -8125+ 7.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8118! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -8098+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -8094+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -8093+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -8089+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -8087+ 7.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -8080+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -8078+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -8075+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -8073+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -8070+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -8065+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -8062+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -8057! 51.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -8006! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -7986+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -7979+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -7976+ 5.000 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -7971+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -7968+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -7962+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -7959+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -7956+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -7954+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -7952+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -7951+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -7949+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -7947! 382.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -7565+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -7564+ 2.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -7561+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -7557+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -7554+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -7552+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -7549+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -7547+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -7545+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -7541+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -7539+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -7537+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -7533+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -7530+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -7524+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -7521+ 3.000 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -7518+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -7510+ 7.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -7503+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -7499+ 2.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -7496! 53.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -7443! 18.000 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -7425+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -7421+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -7417! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -7405+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -7398! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -7377+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -7374+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -7372+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -7368+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -7366+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -7360+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -7358+ 3.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -7355+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -7352+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -7349+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -7344+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -7342+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -7337! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -7286! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -7266+ 7.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -7258+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -7255+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -7251+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -7248+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -7242+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -7239+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -7236+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -7234+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -7232+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -7230+ 1.500 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -7229+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -7227! 381.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -6846+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -6844+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -6842+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -6838+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -6835+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -6832+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -6830+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -6828+ 2.500 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -6825+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -6822+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -6820+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -6817+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -6814+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -6810+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -6805+ 2.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -6802+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -6799+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -6790+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -6784+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -6780+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -6777! 52.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -6725! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -6706+ 4.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -6702+ 4.500 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -6697! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -6685+ 7.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -6678! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -6658+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -6654+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -6653+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -6649+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -6647+ 7.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -6640+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -6638+ 2.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -6636+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -6633+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -6630+ 4.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -6625+ 3.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -6622+ 4.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -6618! 52.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -6566! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -6546+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -6539+ 3.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -6536+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -6531+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -6528+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -6522+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -6519+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -6517+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -6514+ 2.000 default_idle+0x64 (cpu_idle+0x58)
: #func -6512+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -6510+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -6508+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -6507! 381.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -6126+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -6124+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -6122+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -6117+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -6115+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -6112+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -6110+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -6107+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -6105+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -6102+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -6100+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -6097+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -6094+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -6090+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -6085+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -6082+ 3.000 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -6079+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -6070+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -6064+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -6059+ 2.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -6057! 53.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -6004! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -5985+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -5981+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -5977! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -5965+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -5958! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -5937+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -5934+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -5932+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -5928+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -5926+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -5919+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -5917+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -5915+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -5912+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -5909+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -5904+ 3.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -5901+ 4.500 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -5897! 51.500 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -5845! 19.500 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -5826+ 7.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -5818+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -5815+ 5.000 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -5810+ 2.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -5808+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -5802+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -5799+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -5796+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -5794+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -5792+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -5790+ 1.500 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -5789+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -5787! 381.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -5406+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -5404+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -5402+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -5397+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -5395+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -5392+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -5390+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -5387+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -5385+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -5381+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -5380+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -5377+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -5373+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -5370+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -5364+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -5361+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -5358+ 8.000 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -5350+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -5344+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -5339+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -5336! 53.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -5283! 18.000 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -5265+ 4.500 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -5261+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -5257! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -5245+ 7.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -5238! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -5217+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -5213+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -5211+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -5208+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -5206+ 7.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -5199+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -5197+ 2.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -5195+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -5192+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -5189+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -5184+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -5181+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -5176! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -5125! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -5105+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -5098+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -5095+ 5.000 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -5090+ 2.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -5088+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -5082+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -5079+ 3.000 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -5076+ 2.000 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -5074+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -5072+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -5070+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -5068+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -5067! 381.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -4686+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -4684+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -4682+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -4677+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -4675+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -4672+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -4670+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -4667+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -4665+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -4661+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -4660+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -4657+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -4653+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -4650+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -4645+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -4642+ 3.000 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -4639+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -4630+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -4624+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -4620+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -4617! 52.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -4564! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -4546+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -4542+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -4538! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -4526+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -4519! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -4498+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -4495+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -4493+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -4489+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -4487+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -4481+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -4479+ 3.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -4476+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -4474+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -4470+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -4465+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -4463+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -4458! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -4407! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -4387+ 7.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -4379+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -4376+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -4372+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -4369+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -4363+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -4360+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -4357+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -4355+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -4353+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -4352+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -4350+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -4348! 382.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -3966+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -3964+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -3962+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -3958+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -3955+ 3.000 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -3952+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -3950+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -3947+ 2.500 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -3945+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -3941+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -3940+ 3.000 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -3937+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -3933+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -3930+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -3924+ 2.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -3922+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -3918+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -3910+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -3904+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -3899+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -3896! 53.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -3843! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -3825+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -3821+ 4.500 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -3816! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -3805+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3797! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -3777+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -3773+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -3771+ 3.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3768+ 2.500 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -3765+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -3759+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -3757+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -3754+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -3752+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -3748+ 4.500 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -3744+ 3.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -3741+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -3736! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -3685! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -3665+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -3658+ 3.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -3654+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -3650+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -3647+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -3641+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -3638+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -3635+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -3633+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -3631+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -3630+ 2.500 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -3627+ 1.500 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -3626! 380.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -3245+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -3244+ 2.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -3241+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -3237+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -3234+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -3232+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -3230+ 3.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -3227+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -3225+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -3221+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -3219+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -3217+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -3213+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -3210+ 6.000 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -3204+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -3201+ 3.000 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -3198+ 8.500 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -3189+ 6.000 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -3183+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -3179+ 2.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -3176! 53.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -3123! 18.000 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -3105+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -3101+ 4.500 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -3097! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -3085+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3078! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -3057+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -3054+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -3052+ 4.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -3048+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -3046+ 7.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -3039+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -3037+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -3034+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -3032+ 3.000 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -3029+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -3024+ 3.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -3021+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -3016! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -2965! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -2945+ 7.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -2938+ 3.000 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -2935+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -2930+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -2927+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -2921+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -2918+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -2916+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -2913+ 2.000 default_idle+0x64 (cpu_idle+0x58)
: #func -2911+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -2910+ 1.500 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -2908+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -2906! 380.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -2526+ 1.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -2525+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -2523+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -2518+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -2515+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -2513+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -2511+ 3.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -2508+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -2506+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -2502+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -2500+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -2498+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -2494+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -2491+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -2485+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -2482+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -2479+ 9.000 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -2470+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -2463+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -2459+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -2456! 53.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -2403! 18.000 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -2385+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -2381+ 4.500 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -2376! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -2364+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -2357! 20.500 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -2336+ 3.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -2333+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -2331+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -2327+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -2325+ 7.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -2318+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -2316+ 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -2314+ 2.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -2312+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -2308+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -2303+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -2301+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -2296! 52.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -2244! 16.500 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| # func -2227+ 2.000 rthal_irq_host_pend+0x10 (xnintr_clock_handler+0xbc)
:| # func -2225+ 4.000 __ipipe_schedule_irq+0x10 (rthal_irq_host_pend+0x20)
:| # func -2221+ 8.000 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
:| +func -2213+ 3.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +func -2210+ 6.500 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
:| #end 0x80000000 -2203+ 3.500 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
: #func -2200+ 3.000 asm_do_IRQ+0x10 (__ipipe_sync_stage+0x284)
: #func -2197+ 2.000 handle_level_irq+0x10 (asm_do_IRQ+0x50)
: #func -2195+ 2.500 _spin_lock+0x10 (handle_level_irq+0x24)
: #func -2192+ 3.000 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -2189+ 6.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -2183+ 1.500 _spin_unlock+0x10 (handle_level_irq+0x88)
: #func -2182+ 3.000 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -2179+ 2.500 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -2176+ 2.500 handle_IRQ_event+0x10 (handle_level_irq+0x94)
: #func -2174+ 6.000 at91_timer_interrupt+0x10 (handle_IRQ_event+0x3c)
: #func -2168+ 1.500 _spin_lock+0x10 (at91_timer_interrupt+0x1c)
: #func -2166+ 1.500 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -2165+ 3.500 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -2161+ 2.500 timer_tick+0x10 (at91_timer_interrupt+0x50)
: #func -2159+ 6.500 profile_tick+0x10 (timer_tick+0x18)
: #func -2152+ 6.500 do_timer+0x14 (timer_tick+0xe0)
: #func -2146+ 7.500 current_tick_length+0x10 (do_timer+0x144)
: #func -2138+ 2.500 update_process_times+0x10 (timer_tick+0xfc)
: #func -2136+ 4.500 account_system_time+0x10 (update_process_times+0x44)
: #func -2131+ 2.000 run_local_timers+0x10 (update_process_times+0x48)
: #func -2129+ 3.500 raise_softirq+0x10 (run_local_timers+0x18)
:| #begin 0x80000001 -2126+ 2.000 raise_softirq+0x34 (run_local_timers+0x18)
:| #end 0x80000001 -2124+ 3.500 raise_softirq+0x54 (run_local_timers+0x18)
: #func -2120+ 2.000 __ipipe_restore_root+0x10 (raise_softirq+0xb4)
:| #begin 0x80000001 -2118+ 2.500 __ipipe_restore_root+0x38 (raise_softirq+0xb4)
:| #end 0x80000001 -2116+ 2.000 __ipipe_restore_root+0x58 (raise_softirq+0xb4)
: #func -2114+ 4.500 softlockup_tick+0x14 (run_local_timers+0x1c)
: #func -2109+ 2.500 rcu_pending+0x10 (update_process_times+0x50)
: #func -2107+ 3.500 __rcu_pending+0x10 (rcu_pending+0x1c)
: #func -2103+ 3.000 __rcu_pending+0x10 (rcu_pending+0x34)
: #func -2100+ 1.500 scheduler_tick+0x10 (update_process_times+0x64)
: #func -2099+ 4.500 sched_clock+0x10 (scheduler_tick+0x14)
: #func -2094+ 5.000 run_posix_cpu_timers+0x14 (update_process_times+0x6c)
: #func -2089+ 1.500 _spin_unlock+0x10 (at91_timer_interrupt+0xc0)
: #func -2088+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -2086+ 3.000 ipipe_check_context+0x10 (_spin_unlock+0x1c)
:| #begin 0x80000001 -2083+ 2.000 handle_IRQ_event+0x84 (handle_level_irq+0x94)
:| #end 0x80000001 -2081+ 2.000 handle_IRQ_event+0xa4 (handle_level_irq+0x94)
: #func -2079+ 3.500 note_interrupt+0x14 (handle_level_irq+0xb0)
: #func -2076+ 1.500 _spin_lock+0x10 (handle_level_irq+0xb8)
: #func -2074+ 1.500 ipipe_check_context+0x10 (_spin_lock+0x1c)
: #func -2073+ 3.000 _raw_spin_lock+0x14 (_spin_lock+0x24)
: #func -2070+ 2.500 at91_aic_unmask_irq+0x10 (handle_level_irq+0xe4)
: #func -2067+ 1.500 _spin_unlock+0x10 (handle_level_irq+0xec)
: #func -2066+ 1.500 _raw_spin_unlock+0x10 (_spin_unlock+0x14)
: #func -2064+ 2.500 ipipe_check_context+0x10 (_spin_unlock+0x1c)
: #func -2062+ 2.000 irq_exit+0x10 (asm_do_IRQ+0x54)
: #func -2060+ 3.000 __do_softirq+0x10 (irq_exit+0x44)
: #func -2057+ 2.000 __ipipe_unstall_root+0x10 (__do_softirq+0x40)
:| #begin 0x80000000 -2055+ 2.000 __ipipe_unstall_root+0x30 (__do_softirq+0x40)
:| +end 0x80000000 -2053+ 3.000 __ipipe_unstall_root+0x64 (__do_softirq+0x40)
: +func -2050+ 2.500 run_timer_softirq+0x14 (__do_softirq+0x58)
: +func -2047+ 5.500 hrtimer_run_queues+0x10 (run_timer_softirq+0x24)
: +func -2042+ 2.000 _spin_lock_irq+0x10 (run_timer_softirq+0x40)
:| +begin 0x80000001 -2040+ 2.500 _spin_lock_irq+0x34 (run_timer_softirq+0x40)
:| #end 0x80000001 -2037+ 1.500 _spin_lock_irq+0x54 (run_timer_softirq+0x40)
: #func -2036+ 2.000 ipipe_check_context+0x10 (_spin_lock_irq+0x60)
: #func -2034+ 4.500 _raw_spin_lock+0x14 (_spin_lock_irq+0x68)
: #func -2029 1.000 _spin_unlock_irq+0x10 (run_timer_softirq+0x1ac)
: #func -2028+ 2.000 _raw_spin_unlock+0x10 (_spin_unlock_irq+0x14)
: #func -2026+ 1.500 __ipipe_unstall_root+0x10 (_spin_unlock_irq+0x18)
:| #begin 0x80000000 -2025+ 2.000 __ipipe_unstall_root+0x30 (_spin_unlock_irq+0x18)
:| +end 0x80000000 -2023+ 2.000 __ipipe_unstall_root+0x64 (_spin_unlock_irq+0x18)
: +func -2021+ 2.000 ipipe_check_context+0x10 (_spin_unlock_irq+0x20)
:| +begin 0x80000001 -2019+ 2.500 __do_softirq+0x90 (irq_exit+0x44)
:| #end 0x80000001 -2016+ 2.500 __do_softirq+0xb0 (irq_exit+0x44)
: #func -2014+ 2.500 _local_bh_enable+0x10 (__do_softirq+0xe8)
: #func -2011+ 2.500 ipipe_check_context+0x10 (irq_exit+0x4c)
:| #begin 0x80000000 -2009+ 5.500 __ipipe_sync_stage+0x31c (__ipipe_walk_pipeline+0xa8)
:| +end 0xffffffff -2003+ 2.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -2001+ 4.000 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -1997+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -1995+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -1993+ 2.500 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -1991+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -1988+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -1986+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -1984+ 2.000 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -1982+ 1.500 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -1981+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -1979! 173.000 default_idle+0x88 (cpu_idle+0x58)
:| +func -1806+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -1804+ 1.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -1802+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -1798+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -1795+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -1793+ 2.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -1791+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -1789+ 2.500 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -1786+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -1783+ 1.500 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -1781+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -1779+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -1775+ 2.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -1772+ 5.500 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -1767+ 2.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -1764+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -1761+ 8.000 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -1753+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -1746+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -1742+ 2.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -1739! 58.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -1681! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -1663+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -1659+ 4.500 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -1654! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -1643+ 7.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -1636! 21.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -1615+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -1612+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -1610+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -1606+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -1604+ 7.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -1597+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -1595+ 2.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -1593+ 2.500 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -1590+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -1587+ 5.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -1582+ 2.500 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -1579+ 5.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -1574! 51.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -1523! 20.500 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -1503+ 6.500 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +end 0xffffffff -1496+ 3.500 __ipipe_grab_irq+0x110 (__irq_svc+0x28)
:| +(0x2b) 0x00000001 -1493+ 4.500 __ipipe_grab_irq+0x11c (__irq_svc+0x28)
: +func -1488+ 3.000 __ipipe_unstall_root+0x10 (default_idle+0x90)
:| +begin 0x80000000 -1485+ 6.000 __ipipe_unstall_root+0x30 (default_idle+0x90)
:| +end 0x80000000 -1479+ 3.000 __ipipe_unstall_root+0x64 (default_idle+0x90)
: +func -1476+ 2.500 default_idle+0x10 (cpu_idle+0x58)
:| +begin 0x80000001 -1474+ 2.500 default_idle+0x44 (cpu_idle+0x58)
:| #end 0x80000001 -1471+ 1.500 default_idle+0x64 (cpu_idle+0x58)
: #func -1470+ 1.500 __ipipe_unstall_root+0x10 (default_idle+0x80)
:| #begin 0x80000000 -1468+ 2.000 __ipipe_unstall_root+0x30 (default_idle+0x80)
:| +end 0x80000000 -1466+ 2.000 __ipipe_unstall_root+0x64 (default_idle+0x80)
: +end 0x8000000e -1464! 378.500 default_idle+0x88 (cpu_idle+0x58)
:| +func -1086+ 2.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| +begin 0xffffffff -1084+ 2.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| +(0x2a) 0x00000012 -1082+ 4.500 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| +func -1077+ 2.500 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| +func -1075+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| +func -1072+ 2.500 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| +func -1070+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| +func -1067+ 2.000 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| +func -1065+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| +func -1062+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| +func -1060+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| +func -1057+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| # func -1053+ 3.000 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| # func -1050+ 6.000 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| # func -1044+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| # func -1041+ 3.000 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| # [ 745] -<?>- 99 -1038+ 8.000 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| # func -1030+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| # func -1024+ 4.500 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| # func -1019+ 2.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| # [ 0] -<?>- -1 -1017! 52.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| # [ 745] -<?>- 99 -964! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -946+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| # func -942+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| + end 0x80000000 -938! 12.000 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| + begin 0x80000001 -926+ 7.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -918! 35.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -883+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -880+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -878+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -874+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -872+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -866+ 4.500 __rt_sem_v+0x14 (hisyscall_event+0x150)
: + func -861+ 2.000 xnregistry_fetch+0x10 (__rt_sem_v+0x60)
:| + begin 0x80000000 -859+ 4.000 xnregistry_fetch+0x34 (__rt_sem_v+0x60)
:| # func -855+ 1.500 __ipipe_restore_pipeline_head+0x10 (xnregistry_fetch+0xfc)
:| + end 0x80000000 -854+ 2.500 __ipipe_restore_pipeline_head+0x11c (xnregistry_fetch+0xfc)
: + func -851+ 2.000 rt_sem_v+0x10 (__rt_sem_v+0x70)
:| + begin 0x80000000 -849+ 3.500 rt_sem_v+0x38 (__rt_sem_v+0x70)
:| # func -846+ 4.500 xnsynch_wakeup_one_sleeper+0x10 (rt_sem_v+0x8c)
:| # func -841+ 3.500 xnpod_resume_thread+0x10 (xnsynch_wakeup_one_sleeper+0xd0)
:| # [ 744] -<?>- 0 -838+ 6.500 xnpod_resume_thread+0x7c (xnsynch_wakeup_one_sleeper+0xd0)
:| # func -831+ 3.000 xnpod_schedule+0x14 (rt_sem_v+0x98)
:| # [ 745] -<?>- 99 -828+ 3.500 xnpod_schedule+0xc8 (rt_sem_v+0x98)
:| # func -825+ 2.000 __ipipe_restore_pipeline_head+0x10 (rt_sem_v+0xd4)
:| + end 0x80000000 -823+ 2.000 __ipipe_restore_pipeline_head+0x11c (rt_sem_v+0xd4)
:| + begin 0x80000001 -821+ 2.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -819+ 5.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -814+ 1.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -812+ 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -811+ 2.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -808 1.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -807+ 3.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -804+ 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
: + func -802+ 2.000 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
: + func -800+ 3.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
:| + begin 0x80000000 -797+ 3.500 xnpod_wait_thread_period+0x44 (rt_task_wait_period+0x44)
:| # func -794+ 4.000 xnpod_suspend_thread+0x14 (xnpod_wait_thread_period+0xb8)
:| # func -790+ 2.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 745] -<?>- 99 -788+ 9.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # [ 744] -<?>- 0 -779+ 7.000 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| # func -772+ 1.500 __ipipe_restore_pipeline_head+0x10 (rt_sem_p+0x158)
:| + end 0x80000000 -770+ 5.000 __ipipe_restore_pipeline_head+0x11c (rt_sem_p+0x158)
:| + begin 0x80000001 -765+ 2.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -763! 355.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: + func -408+ 6.000 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: + func -402+ 3.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| + begin 0x80000001 -399+ 6.500 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| + end 0x80000001 -392+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: + func -390+ 3.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: + func -387+ 2.000 xnshadow_relax+0x14 (hisyscall_event+0x210)
:| + begin 0x80000000 -385+ 2.500 xnshadow_relax+0x40 (hisyscall_event+0x210)
:| # func -382+ 4.500 schedule_linux_call+0x10 (xnshadow_relax+0x80)
:| # func -378+ 3.500 rthal_apc_schedule+0x10 (schedule_linux_call+0xb8)
:| # func -374+ 4.500 __ipipe_schedule_irq+0x10 (rthal_apc_schedule+0xac)
:| # func -370+ 5.000 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
:| # func -365+ 5.000 xnpod_suspend_thread+0x14 (xnshadow_relax+0xac)
:| # func -360+ 4.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x1d4)
:| # [ 744] -<?>- 0 -356+ 4.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x1d4)
:| # func -352! 58.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x2ac)
:| # [ 0] -<?>- -1 -294! 20.000 xnpod_schedule+0x590 (xnintr_clock_handler+0xa0)
:| +func -274+ 4.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
:| +func -270+ 7.500 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
:| #end 0x80000000 -262+ 6.500 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
:| #func -256+ 3.000 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
:| #begin 0xffffffff -253+ 6.500 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
:| #(0x2a) 0x00000012 -246+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
:| #func -241+ 3.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
:| #func -238+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
:| #func -236+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
:| #func -233+ 2.500 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x30)
:| #func -230+ 2.500 __ipipe_irq_finish+0x10 (__ipipe_ack_timerirq+0x38)
:| #func -228+ 3.500 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x3c)
:| #func -224+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x4c)
:| #func -222+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
:| #func -220+ 4.000 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
:| #*func -216+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
:| #*func -212+ 6.000 xntimer_tick_aperiodic+0x10 (xnintr_clock_handler+0x34)
:| #*func -206+ 3.000 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0xd8)
:| #*func -203+ 3.500 xnpod_resume_thread+0x10 (xnthread_periodic_handler+0x34)
:| #*[ 745] -<?>- 99 -200+ 8.000 xnpod_resume_thread+0x7c (xnthread_periodic_handler+0x34)
:| #*func -192+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x258)
:| #*func -185+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
:| #*func -181+ 3.000 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
:| #*[ 0] -<?>- -1 -178! 53.000 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
:| #*[ 745] -<?>- 99 -125! 18.500 xnpod_schedule+0x590 (xnpod_suspend_thread+0x1d4)
:| #*func -107+ 4.000 xntimer_get_overruns+0x14 (xnpod_wait_thread_period+0xe0)
:| #*func -103+ 4.000 __ipipe_restore_pipeline_head+0x10 (xnpod_wait_thread_period+0x114)
:| +*end 0x80000000 -99! 11.500 __ipipe_restore_pipeline_head+0x11c (xnpod_wait_thread_period+0x114)
:| +*begin 0x80000001 -87+ 8.000 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
:| +*end 0x80000001 -79! 40.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
: +*func -39+ 3.000 __ipipe_syscall_root+0x10 (vector_swi+0x64)
: +*func -36+ 2.000 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
:| +*begin 0x80000001 -34+ 4.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
:| +*end 0x80000001 -30+ 2.000 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
: +*func -28+ 6.500 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
: +*func -22+ 2.500 xnshadow_sys_trace+0x10 (hisyscall_event+0x150)
: +*func -19+ 2.000 ipipe_trace_frozen_reset+0x10 (xnshadow_sys_trace+0x98)
: +*func -17+ 2.000 __ipipe_global_path_lock+0x10 (ipipe_trace_frozen_reset+0x14)
: +*func -15+ 2.000 __ipipe_spin_lock_irqsave+0x10 (__ipipe_global_path_lock+0x18)
:| +*begin 0x80000001 -13+ 8.000 __ipipe_spin_lock_irqsave+0x34 (__ipipe_global_path_lock+0x18)
:| #*func -5+ 3.000 __ipipe_spin_unlock_irqcomplete+0x10 (__ipipe_global_path_unlock+0x6c)
:| +*end 0x80000001 -2+ 2.500 __ipipe_spin_unlock_irqcomplete+0x4c (__ipipe_global_path_unlock+0x6c)
< +*freeze 0x0006562d 0 2.500 xnshadow_sys_trace+0xa4 (hisyscall_event+0x150)
| +*begin 0x80000001 2 1.500 __ipipe_dispatch_event+0x108 (__ipipe_syscall_root+0x84)
| +*end 0x80000001 4 5.000 __ipipe_dispatch_event+0x1cc (__ipipe_syscall_root+0x84)
+*func 9 1.500 __ipipe_syscall_root+0x10 (vector_swi+0x64)
+*func 10 1.500 __ipipe_dispatch_event+0x14 (__ipipe_syscall_root+0x84)
| +*begin 0x80000001 12 2.000 __ipipe_dispatch_event+0x40 (__ipipe_syscall_root+0x84)
| +*end 0x80000001 14 1.500 __ipipe_dispatch_event+0xc8 (__ipipe_syscall_root+0x84)
+*func 15 3.000 hisyscall_event+0x14 (__ipipe_dispatch_event+0xe0)
+*func 18 2.000 __rt_task_wait_period+0x14 (hisyscall_event+0x150)
+*func 20 2.500 rt_task_wait_period+0x10 (__rt_task_wait_period+0x24)
+*func 23 0.000 xnpod_wait_thread_period+0x14 (rt_task_wait_period+0x44)
#
--
Gilles Chanteperdrix.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-02-11 13:41 ` Jan Kiszka
2008-02-11 14:05 ` Gilles Chanteperdrix
@ 2008-02-12 7:04 ` Gilles Chanteperdrix
2008-02-12 7:53 ` Gilles Chanteperdrix
1 sibling, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-02-12 7:04 UTC (permalink / raw)
To: Jan Kiszka; +Cc: jagarcia, xenomai
Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
> > I-pipe frozen back-tracing service on 2.6.20/ipipe-1.8-04
> > ------------------------------------------------------------
> > CPU: 0, Freeze: 450692973 cycles, Trace Points: 1000 (+10)
> > Calibrated minimum trace-point overhead: 1.000 us
>
> That is interesting. I tells us that we might subtract 1 us
> _per_tracepoint_ from the given latencies due to the inherent tracer
> overhead. We have about 50 entries in the critical path, so 50 us
> compared to 220 us that were measured - roughly 170 us real latency.
>
> What is the clock resolution btw? 500 ns?
>
> So here is the interesting block, starting with the last larger IRQs-on
> window.
>
> > : + func -447+ 2.500 xnshadow_relax+0x14 (hisyscall_event+0x210)
> > :| + begin 0x80000000 -445+ 3.000 xnshadow_relax+0xd4 (hisyscall_event+0x210)
> > :| # func -442+ 5.000 schedule_linux_call+0x10 (xnshadow_relax+0x114)
> > :| # func -437+ 4.000 rthal_apc_schedule+0x10 (schedule_linux_call+0x1e8)
> > :| # func -433+ 5.000 __ipipe_schedule_irq+0x10 (rthal_apc_schedule+0xac)
> > :| # func -428+ 4.500 __ipipe_set_irq_pending+0x10 (__ipipe_schedule_irq+0xa4)
> > :| # func -423+ 3.500 rpi_push+0x14 (xnshadow_relax+0x11c)
> > :| # func -420+ 5.500 xnpod_suspend_thread+0x14 (xnshadow_relax+0x148)
> > :| # func -414+ 4.000 xnpod_schedule+0x14 (xnpod_suspend_thread+0x60c)
> > :| # [ 752] -<?>- 0 -410+ 7.000 xnpod_schedule+0xc8 (xnpod_suspend_thread+0x60c)
> > :| # func -403! 56.000 xnheap_finalize_free_inner+0x10 (xnpod_schedule+0x82c)
Ok, we get a cache flush here
> > :| # [ 0] -<?>- -1 -347! 20.000 xnpod_schedule+0xb14 (xnintr_clock_handler+0xa0)
> > :| +func -327+ 3.000 __ipipe_walk_pipeline+0x10 (__ipipe_handle_irq+0x124)
> > :| +func -324+ 7.500 __ipipe_sync_stage+0x14 (__ipipe_walk_pipeline+0xa8)
> > :| #end 0x80000000 -317+ 7.000 __ipipe_sync_stage+0x250 (__ipipe_walk_pipeline+0xa8)
>
> OK, the clock starts ticking...
>
> > :| #func -310+ 2.500 __ipipe_grab_irq+0x10 (__irq_svc+0x28)
> > :| #begin 0xffffffff -307+ 7.000 __ipipe_grab_irq+0x20 (__irq_svc+0x28)
> > :| #(0x2a) 0x00000012 -300+ 5.000 __ipipe_grab_irq+0x2c (__irq_svc+0x28)
> > :| #func -295+ 4.000 __ipipe_handle_irq+0x10 (__ipipe_grab_irq+0x104)
> > :| #func -291+ 2.500 __ipipe_ack_timerirq+0x10 (__ipipe_handle_irq+0x74)
> > :| #func -289+ 3.000 __ipipe_ack_level_irq+0x10 (__ipipe_ack_timerirq+0x30)
> > :| #func -286+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x3c)
> > :| #func -284+ 2.000 at91_aic_mask_irq+0x10 (__ipipe_ack_level_irq+0x4c)
>
> (Without looking at the arm code: Is this double invocation of
> at91_aic_mask_irq correct and required?)
>
> > :| #func -282+ 4.000 __ipipe_mach_acktimer+0x10 (__ipipe_ack_timerirq+0x40)
> > :| #func -278+ 2.000 __ipipe_end_level_irq+0x10 (__ipipe_ack_timerirq+0x50)
> > :| #func -276+ 2.500 at91_aic_unmask_irq+0x10 (__ipipe_end_level_irq+0x28)
> > :| #func -273+ 3.500 __ipipe_dispatch_wired+0x14 (__ipipe_handle_irq+0x80)
> > :| #*func -270+ 3.500 xnintr_clock_handler+0x10 (__ipipe_dispatch_wired+0xe4)
> > :| #*func -266+ 9.500 xntimer_tick_aperiodic+0x14 (xnintr_clock_handler+0x34)
> > :| #*func -257+ 3.500 xnthread_periodic_handler+0x10 (xntimer_tick_aperiodic+0x354)
> > :| #*func -253+ 4.000 xnpod_resume_thread+0x14 (xnthread_periodic_handler+0x34)
> > :| #*[ 753] -<?>- 99 -249! 15.000 xnpod_resume_thread+0x84 (xnthread_periodic_handler+0x34)
>
> Hmm, comparably costly, this simple resume. Hope it's not the
> instrumentation (ipipe_trace_pid?) itself.
>
> > :| #*func -234+ 6.500 xntimer_next_local_shot+0x10 (xntimer_tick_aperiodic+0x7c0)
> > :| #*func -228+ 4.000 __ipipe_mach_set_dec+0x10 (xntimer_next_local_shot+0xbc)
> > :| #*func -224+ 3.500 xnpod_schedule+0x14 (xnintr_clock_handler+0xa0)
> > :| #*[ 0] -<?>- -1 -220! 59.500 xnpod_schedule+0xc8 (xnintr_clock_handler+0xa0)
>
> OK, this is the cache flushing thing, I guess. Expected.
And another here, whereas if I understand correctly, the mm did not
change. So, this is probably an unwanted effect of the cache flush
"optimization" in the arm patch.
I will now try to understand if this second cache flush is really normal.
--
Gilles Chanteperdrix.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Xenomai-core] [Xenomai-help] AT91SAM9260 latency
2008-02-12 7:04 ` Gilles Chanteperdrix
@ 2008-02-12 7:53 ` Gilles Chanteperdrix
0 siblings, 0 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2008-02-12 7:53 UTC (permalink / raw)
To: Jan Kiszka, Juan Antonio Garcia Redondo, jagarcia, xenomai
Gilles Chanteperdrix wrote:
>
> And another here, whereas if I understand correctly, the mm did not
> change. So, this is probably an unwanted effect of the cache flush
> "optimization" in the arm patch.
>
> I will now try to understand if this second cache flush is really normal.
Yes, it is normal: the first context switch, which xnshadow_relax does,
is a switch to whatever task Linux was running when preempted, not
necessarily latency (and it turns out to never be latency when we
capture the worst case) hence the first cache flush. We then re-interrupt
Linux after this context switch, and switch again to latency, and we get
a second cache flush.
So, the conclusion is: everything is normal. What we obtain when
pressing the enter key while latency is running in the background is a
wakeup of the shell process and this process uses cache, so that the
next latency context switches need to flush cache.
In other words: pressing the enter key yields the same latency as
running the cache calibrator because it has the same effect, it fills
the cache.
--
Gilles Chanteperdrix.
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2008-02-12 7:53 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-01-23 6:52 [Xenomai-help] AT91SAM9260 latency Juan Antonio Garcia Redondo
2008-01-23 10:04 ` Gilles Chanteperdrix
2008-01-23 13:15 ` Gilles Chanteperdrix
2008-01-24 9:41 ` Juan Antonio Garcia Redondo
2008-01-24 10:02 ` Gilles Chanteperdrix
2008-01-25 10:04 ` Juan Antonio Garcia Redondo
2008-01-25 17:00 ` Gilles Chanteperdrix
2008-01-28 8:51 ` Juan Antonio Garcia Redondo
2008-01-28 9:21 ` Juan Antonio Garcia Redondo
2008-01-28 13:19 ` Gilles Chanteperdrix
2008-01-28 13:34 ` Jan Kiszka
2008-01-28 13:35 ` Gilles Chanteperdrix
2008-01-28 13:46 ` Jan Kiszka
2008-01-28 13:51 ` Gilles Chanteperdrix
2008-01-28 14:10 ` Jan Kiszka
2008-01-29 8:09 ` Juan Antonio Garcia Redondo
2008-01-29 8:35 ` Gilles Chanteperdrix
2008-01-29 17:19 ` Gilles Chanteperdrix
2008-01-30 9:03 ` Juan Antonio Garcia Redondo
2008-02-10 20:31 ` [Xenomai-core] " Gilles Chanteperdrix
2008-02-11 13:41 ` Jan Kiszka
2008-02-11 14:05 ` Gilles Chanteperdrix
2008-02-11 17:11 ` Jan Kiszka
2008-02-11 21:30 ` Gilles Chanteperdrix
2008-02-11 22:36 ` Gilles Chanteperdrix
2008-02-12 7:04 ` Gilles Chanteperdrix
2008-02-12 7:53 ` Gilles Chanteperdrix
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.