[Xenomai-help] Large (negative) latency test results and fast PC clock

[Xenomai-help] Large (negative) latency test results and fast PC clock

[vikesh rambaran]

[-- Attachment #1.1: Type: text/plain, Size: 589 bytes --]

hi there

I built and installed Xenomai 2.40 with kernel 2.6.23.12 on an AMD Athlon 64
X 2 dual core PC.
The linux distribution used is Ubuntu 7.10, 32 bit version

When running the latency test, there are large negative and positive
latencies reported.
Sometimes the PC hangs while the latency test is running
The PC time updates at almost twice the normal speed (ie almost 2 seconds
for every one second that elapses)

Was not able to solve it using the xenomai TROUBLESHOOTING guide

The kernel config file and xeno-test output are attached.

Any help would be appreciated

Thanx
Vicki

[-- Attachment #1.2: Type: text/html, Size: 689 bytes --]

[-- Attachment #2: config-2.6.23.12-adeos-realtime.zip --]
[-- Type: application/zip, Size: 21402 bytes --]

[-- Attachment #3: xeno-test.txt.zip --]
[-- Type: application/zip, Size: 6315 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

vikesh rambaran wrote:
> hi there
> 
> I built and installed Xenomai 2.40 with kernel 2.6.23.12 on an AMD Athlon 64
> X 2 dual core PC.
> The linux distribution used is Ubuntu 7.10, 32 bit version
> 
> When running the latency test, there are large negative and positive
> latencies reported.
> Sometimes the PC hangs while the latency test is running
> The PC time updates at almost twice the normal speed (ie almost 2 seconds
> for every one second that elapses)
> 
> Was not able to solve it using the xenomai TROUBLESHOOTING guide
> 
> The kernel config file and xeno-test output are attached.
> 
> Any help would be appreciated

You should have seen this message while configuring Xenomai:

"NOTE: Xenomai needs either X86_LOCAL_APIC enabled or HPET_TIMER disabled."

Please check if this message is visible (if not, we would like to know).
Then adopt your .config appropriately. Hint: Choose a processor type
that is reflecting your real hardware more precisely. CONFIG_M586 means
that there is no TSC, which is not true and quite suboptimal.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[vikesh rambaran]

[-- Attachment #1.1: Type: text/plain, Size: 1593 bytes --]

Hi

Thanx for taking the time to reply over this festive season !

Although suboptimal for this PC, i am compiling using ubuntu's default
config file.
The idea is to make a general kernel image .deb file which can be used to
install
xenomai on older PC architectures (not necessarily AMD 64 bit CPUs)
Therefore the processor shows up as "CONFIG_M586"

This seems to be what Paul is making available as well - thanx i will try
them out.

Jan, i have checked the kernel config file. The following are set

CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y

CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y

Is this combination a problem ?
dmesg shows that the APIC is configured and enabled, but no information of
the HPET is displayed. (dmesg output is attached)

I have not been able to find the message "NOTE: Xenomai needs either
X86_LOCAL_APIC enabled or HPET_TIMER disabled."
during kernel configuration (make menuconfig) or during the xenomai
configuration. Is there a specific place where i should look ?

To build the kernel image package i used

fakeroot makekpkg --initrd --append_to_version "-adeos-realtime"
kernel_image kernel_headers

For the xenomai configuration i used

./configure --disable-x86-tsc


Note, on this same PC, using ubuntu 6.06, kernel 2.6.20.12, xenomai 2.3.1,
the latency test runs without large
(negative) latencies and no noticeable PC clock speedup. Test configuration
was also for CONFIG_M586=y
but with SMI workaround not enabled.

Please let me know if any other information that i can provide or tests that
can be done to solve this.

Thanx for your help
Vicki

[-- Attachment #1.2: Type: text/html, Size: 1798 bytes --]

[-- Attachment #2: dmesg.txt.zip --]
[-- Type: application/zip, Size: 6725 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2461 bytes --]

vikesh rambaran wrote:
> Hi
> 
> Thanx for taking the time to reply over this festive season !
> 
> Although suboptimal for this PC, i am compiling using ubuntu's default
> config file.
> The idea is to make a general kernel image .deb file which can be used to
> install
> xenomai on older PC architectures (not necessarily AMD 64 bit CPUs)
> Therefore the processor shows up as "CONFIG_M586"

But will there be CPUs included that don't have TSC support? If not,
choose at least a type that tells Xenomai it can use the TSC and does
not have to fall back to slow PIT-based emulation for timekeeping.

> 
> This seems to be what Paul is making available as well - thanx i will try
> them out.
> 
> Jan, i have checked the kernel config file. The following are set
> 
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> 
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> 
> Is this combination a problem ?

Yes, because with TSC being officially available, Xenomai uses the PIT
for timekeeping. At the same time, HPET_TIMER disables the PIT. That's
my theory ATM. Could you validate it by disabling CONFIG_HPET_TIMER?

> dmesg shows that the APIC is configured and enabled, but no information of
> the HPET is displayed. (dmesg output is attached)
> 
> I have not been able to find the message "NOTE: Xenomai needs either
> X86_LOCAL_APIC enabled or HPET_TIMER disabled."
> during kernel configuration (make menuconfig) or during the xenomai
> configuration. Is there a specific place where i should look ?

It would pop up on top of the Xenomai submenu. But as the local APIC is
on, this will not trigger anyway. If my theory is correct, we need to
enhance the condition for the warning.

> 
> To build the kernel image package i used
> 
> fakeroot makekpkg --initrd --append_to_version "-adeos-realtime"
> kernel_image kernel_headers
> 
> For the xenomai configuration i used
> 
> ./configure --disable-x86-tsc
> 
> 
> Note, on this same PC, using ubuntu 6.06, kernel 2.6.20.12, xenomai 2.3.1,

According to your logs, you are using Xenomai 2.4.

> the latency test runs without large
> (negative) latencies and no noticeable PC clock speedup. Test configuration
> was also for CONFIG_M586=y
> but with SMI workaround not enabled.
> 
> Please let me know if any other information that i can provide or tests that
> can be done to solve this.
> 
> Thanx for your help
> Vicki
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[vikesh rambaran]

[-- Attachment #1: Type: text/plain, Size: 3519 bytes --]

Hi

On Dec 31, 2007 4:43 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:

> vikesh rambaran wrote:
> > Hi
> >
> > Thanx for taking the time to reply over this festive season !
> >
> > Although suboptimal for this PC, i am compiling using ubuntu's default
> > config file.
> > The idea is to make a general kernel image .deb file which can be used
> to
> > install
> > xenomai on older PC architectures (not necessarily AMD 64 bit CPUs)
> > Therefore the processor shows up as "CONFIG_M586"
>
> But will there be CPUs included that don't have TSC support? If not,
> choose at least a type that tells Xenomai it can use the TSC and does
> not have to fall back to slow PIT-based emulation for timekeeping.


Thanx for the info. Will have a look at all our platforms and change the
processor type accordingly


> >
> > This seems to be what Paul is making available as well - thanx i will
> try
> > them out.
> >
> > Jan, i have checked the kernel config file. The following are set
> >
> > CONFIG_X86_LOCAL_APIC=y
> > CONFIG_X86_IO_APIC=y
> >
> > CONFIG_HPET_TIMER=y
> > CONFIG_HPET_EMULATE_RTC=y
> >
> > Is this combination a problem ?
>
> Yes, because with TSC being officially available, Xenomai uses the PIT
> for timekeeping. At the same time, HPET_TIMER disables the PIT. That's
> my theory ATM. Could you validate it by disabling CONFIG_HPET_TIMER?
>

Ok, will disable CONFIG_HPET_TIMER and provide feedback


>
> > dmesg shows that the APIC is configured and enabled, but no information
> of
> > the HPET is displayed. (dmesg output is attached)
> >
> > I have not been able to find the message "NOTE: Xenomai needs either
> > X86_LOCAL_APIC enabled or HPET_TIMER disabled."
> > during kernel configuration (make menuconfig) or during the xenomai
> > configuration. Is there a specific place where i should look ?
>
> It would pop up on top of the Xenomai submenu. But as the local APIC is
> on, this will not trigger anyway. If my theory is correct, we need to
> enhance the condition for the warning.


Yes, this is where the power management and speaker warning messages popped
up. After disabling
them, there were no more warning messages


>
> >
> > To build the kernel image package i used
> >
> > fakeroot makekpkg --initrd --append_to_version "-adeos-realtime"
> > kernel_image kernel_headers
> >
> > For the xenomai configuration i used
> >
> > ./configure --disable-x86-tsc
> >
> >
> > Note, on this same PC, using ubuntu 6.06, kernel 2.6.20.12, xenomai
> 2.3.1,
>
> According to your logs, you are using Xenomai 2.4.


You are correct, the logs which i have posted thus far are only for xenomai
2.4.
But i have a dual boot configuration on which Ubuntu 6.06 exists with
xenomai 2.3.1


>
> > the latency test runs without large
> > (negative) latencies and no noticeable PC clock speedup. Test
> configuration
> > was also for CONFIG_M586=y
> > but with SMI workaround not enabled.
> >
> > Please let me know if any other information that i can provide or tests
> that
> > can be done to solve this.
> >
> > Thanx for your help
> > Vicki
> >
>
> Jan
>

As an additional test i have downloaded and installed the xenomai kernel
image generated by Paul. The kernel boots
and there is no pc time speedup. Still need to get the latency test
installed and running. However, i've noticed that the
xenomai version displayed in dmesg logs is 2.4.1.

Is this an SVN snapshot ?

I will try to compile with this kernel config file and give feedback in the
"new year"

Keep well and have a great new years day!
Vicki

[-- Attachment #2: Type: text/html, Size: 5030 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[Paul]

On Monday 31 December 2007 15:54, vikesh rambaran wrote:
<snip>
> As an additional test i have downloaded and installed the xenomai kernel
> image generated by Paul. The kernel boots and there is no pc time speedup.

The kernel downloaded is a basic 586, no TSC, no APIC, or HPET timers enabled. 
If you want to compare performance, try the -686 package.


> Still need to get the latency test installed and running. However, i've
> noticed that the xenomai version displayed in dmesg logs is 2.4.1.

latency, xeno-test, along with the other utilities are all in the xenomai 
package (xeno-config is in xenomai-dev along with the header files).

> Is this an SVN snapshot ?

Snapshot of the v2.4.x branch, rev.3373.


Regards, Paul.

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[vikesh rambaran]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

Hi

Thanx for the help thus far, we're making progress :)

Unselecting HPET in the kernel configuration corrects the problem of the PC
time running faster
than normal. Jan's comment about "enhancing the Xenomai warning message" may
be valid here

However, the latency test still yields the same large results as before (:

How does xenomai select the timerdev and the clockdev from the following
kernel config parameters ?

a) X86_LOCAL_APIC
b) X86_IO_APIC
c) X86_GOOD_APIC
d) X86_UP_APIC
e) M586TSC
f) X86_TSC
g) HPET


On Dec 31, 2007 6:54 PM, Paul < paul_c@domain.hid> wrote:

> On Monday 31 December 2007 15:54, vikesh rambaran wrote:
> <snip>
> > As an additional test i have downloaded and installed the xenomai kernel
> > image generated by Paul. The kernel boots and there is no pc time
> speedup.
>
> The kernel downloaded is a basic 586, no TSC, no APIC, or HPET timers
> enabled.
> If you want to compare performance, try the -686 package.
>


The kernel .config file for the "Linux 2.6.23 image on
586/K5/5x86/6x86/6x86MX (patched with Xenomai)"
package has the following set

CONFIG_X86_GOOD_APIC=y
# CONFIG_X86_UP_APIC is not set

# CONFIG_M586TSC is not set
CONFIG_X86_TSC=y

# CONFIG_M586 is not set
CONFIG_M686=y

This does not seem to correspond to your comment above. Is there something
i've missed ?



> > Still need to get the latency test installed and running. However, i've
> > noticed that the xenomai version displayed in dmesg logs is 2.4.1.
>
> latency, xeno-test, along with the other utilities are all in the xenomai
> package (xeno-config is in xenomai-dev along with the header files).
>

In synaptic, the description for the xenomai package indicates that it is
compiled with TSC support.
Is there an appropriate xenomai deb package for the 586 without TSC
configuration ?

Thanx again for all your help
Vicki

[-- Attachment #2: Type: text/html, Size: 2515 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[Paul]

On Tuesday 01 January 2008 16:17, vikesh rambaran wrote:
> The kernel .config file for the "Linux 2.6.23 image on
> 586/K5/5x86/6x86/6x86MX (patched with Xenomai)"
> package has the following set
>
> CONFIG_X86_GOOD_APIC=y
> # CONFIG_X86_UP_APIC is not set
> # CONFIG_M586TSC is not set
> CONFIG_X86_TSC=y
> # CONFIG_M586 is not set
> CONFIG_M686=y
>
> This does not seem to correspond to your comment above. Is there something
> i've missed ?

/me curses.. Some of the basic 586 options appear to have been overrriden by 
options for the 686 flavour. With the release of Xenomai-2.4.1 anounced, I'll 
rebuild everything overnight and try to resolve the APIC/TSC issue.


Regards, Paul.

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[vikesh rambaran]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

hi

Sorry for taking a long time to reply. (Had to do some windpump maintenance
on a farm !)

On Jan 1, 2008 9:48 PM, Paul <paul_c@domain.hid> wrote:

> On Tuesday 01 January 2008 16:17, vikesh rambaran wrote:
> > The kernel .config file for the "Linux 2.6.23 image on
> > 586/K5/5x86/6x86/6x86MX (patched with Xenomai)"
> > package has the following set
> >
> > CONFIG_X86_GOOD_APIC=y
> > # CONFIG_X86_UP_APIC is not set
> > # CONFIG_M586TSC is not set
> > CONFIG_X86_TSC=y
> > # CONFIG_M586 is not set
> > CONFIG_M686=y
> >
> > This does not seem to correspond to your comment above. Is there
> something
> > i've missed ?
>
> /me curses.. Some of the basic 586 options appear to have been overrriden
> by
> options for the 686 flavour. With the release of Xenomai-2.4.1 anounced,
> I'll
> rebuild everything overnight and try to resolve the APIC/TSC issue.
>
>
> Regards, Paul.
>


Thanx for confirming that. Please let me know when your builds are
available,
as it helps to have a cross reference to figure out what i'm doing wrong

In the mean time, i will also do a build with 2.4.1 to see if it helps any

Keep well
Vicki

[-- Attachment #2: Type: text/html, Size: 1502 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[Paul]

On Friday 04 January 2008 12:39, vikesh rambaran wrote:
> Thanx for confirming that. Please let me know when your builds are
> available,
> as it helps to have a cross reference to figure out what i'm doing wrong

x86 kernels patched with Xenomai 2.4.1 were uploaded on Wednesday. Will 
probably do a 64bit build over the weekend.


Regards, Paul.

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[Jan Kiszka]

vikesh rambaran wrote:
> ...
> How does xenomai select the timerdev and the clockdev from the following
> kernel config parameters ?
> 
> a) X86_LOCAL_APIC
> b) X86_IO_APIC
> c) X86_GOOD_APIC
> d) X86_UP_APIC
> e) M586TSC
> f) X86_TSC
> g) HPET

To answer this question (for x86-32 aka i386):

Timer = X86_LOCAL_APIC ? LAPIC : PIT
Clock = X86_TSC ? TSC : PIT-emulated TSC

X86_UP_APIC controls the availability of X86_LOCAL_APIC on UP, M586TSC
(or better) enables X86_TSC. X86_IO_APIC and X86_GOOD_APIC have no
relevance here (the latter just influence the way the APIC is accessed
on i386). HPET_TIMER has that well-known side-effect of blocking the PIT
in case there is actually HPET hardware available and used by Linux.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[vikesh rambaran]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

Hi
Just some feedback.

Compared config files from my old xenomai 2.3.1, new xenomai
2.4.1installation and Paul's
- SMP settings were disabled all but my latest installation

Disabling the SMP option in the kernel configuration restores latencies to
acceptable values.
Any idea why this is so ?

According to the xenomai README.INSTALL the SMP option has a weak binding,
but i tried
compiling xenomai userspace with --enable-smp anyway as a test - did not
help for the
SMP enabled kernel

On Jan 4, 2008 6:26 PM, Paul <paul_c@domain.hid> wrote:

> On Friday 04 January 2008 12:39, vikesh rambaran wrote:
> > Thanx for confirming that. Please let me know when your builds are
> > available,
> > as it helps to have a cross reference to figure out what i'm doing wrong
>
> x86 kernels patched with Xenomai 2.4.1 were uploaded on Wednesday. Will
> probably do a 64bit build over the weekend.
>

Thanx!
There is a xenomai userspace package which goes with the 686  kernel image
Will there be a xenomai package to match the 586 kernel image?
[A separate on is required because of the feature-set differences eg tsc]


>
> Regards, Paul.
>


Keep well
Vicki

[-- Attachment #2: Type: text/html, Size: 1665 bytes --]

Re: [Xenomai-help] Large (negative) latency test results and fast PC clock

[vikesh rambaran]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

Correction :

On Jan 7, 2008 12:35 PM, vikesh rambaran <vikesh.rambaran@domain.hid> wrote:

> Hi
> Just some feedback.
>
> Compared config files from my old xenomai 2.3.1, new xenomai 2.4.1installation and Paul's
> - SMP settings were disabled all but my latest installation
>
> Disabling the SMP option in the kernel configuration restores latencies to
> acceptable values.
> Any idea why this is so ?
>

Disabling SMP AND enabling APIC gives acceptable latencies. Without the
APIC, latencies are in the
order of a few milliseconds (< 100ms)

Thanx
Vicki

[-- Attachment #2: Type: text/html, Size: 839 bytes --]