[Xenomai-core] rare brokenness in latency output - FPU bug!

[Xenomai-core] rare brokenness in latency output - FPU bug!

[Jan Kiszka]

[-- Attachment #1.1: Type: text/plain, Size: 1812 bytes --]

Hi,

it's now the second box I noticed this minor quirk with SVN head:
latency (tested with -t0 only so far) sometimes gives me a broken "lat
worst" number like this:

> RTD|      15.637|      17.842|      42.750|       0|       4.601|      54.264
> RTT|  00:04:55  (periodic user-mode task, 100 us period, priority 99)
> RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
> RTD|      14.463|      18.019|      41.907|       0|       4.601|         nan
> RTD|      15.197|      17.902|      37.396|       0|       4.601|      54.264

I also already had weird large numbers here (>200 digits) which makes me
think that this is not an issue of gmaxj (local long variable to store
the result before output conversion), but rather something with the FPU.
Anyone any thoughts?


Uhh, wait! I just fired up switchtest on the same box:

> == Testing FPU check routines...
> r0: 1 != 2
> r1: 1 != 2
> r2: 1 != 2
> r3: 1 != 2
> r4: 1 != 2
> r5: 1 != 2
> r6: 1 != 2
> r7: 1 != 2
> == FPU check routines: OK.
> == Threads: sleeper_ufps-0 rtk-1 rtk-2 rtk_fp-3 rtk_fp-4 rtk_fp_ufpp-5 rtk_fp_ufpp-6 rtup-7 rtup-8 rtup_ufpp-9 rtup_ufpp-10 rtus-11 rtus-12 rtus_ufps-13 rtus_ufps-14 rtuo-15 rtuo-16 rtuo_ufpp-17 rtuo_ufpp-18 rtuo_ufps-19 rtuo_ufps-20 rtuo_ufpp_ufps-21 rtuo_ufpp_ufps-22
> r0: 2147483648 != 10
> r1: 2147483648 != 10
> r2: 2147483648 != 10
> r3: 2147483648 != 10
> r4: 2147483648 != 10
> r5: 2147483648 != 10
> r6: 2147483648 != 10
> r7: 2147483648 != 10
> Error after context switch from task 8(rtup-8) to task 10(rtup_ufpp-10),
> FPU registers were set to 2147483648 (maybe task ???)
> Segmentation fault

Gilles, what info do you need? It's all the latest on x86 here:
2.6.17.13 (.config attached), ipipe-1.4, SVN #1646, gcc4.1.0.

Jan

[-- Attachment #1.2: config-2.6.17.bz2 --]
[-- Type: application/octet-stream, Size: 7132 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Jeff Webb]

Jan Kiszka wrote:
> Uhh, wait! I just fired up switchtest on the same box:
> ...
> Gilles, what info do you need? It's all the latest on x86 here:
> 2.6.17.13 (.config attached), ipipe-1.4, SVN #1646, gcc4.1.0.
> 
> Jan

I get a similar result here with xenomai-2.2.3, 2.6.17.13, x86 SMP (see below).  No seg fault, though.

-Jeff


*
*
* Type ^C to stop this application.
*
*
== Testing FPU check routines...
r0: 1 != 2
r1: 1 != 2
r2: 1 != 2
r3: 1 != 2
r4: 1 != 2
r5: 1 != 2
r6: 1 != 2
r7: 1 != 2
== FPU check routines: OK.
== Threads: sleeper_ufps0-0 rtk0-1 rtk0-2 rtk_fp0-3 rtk_fp0-4 rtk_fp_ufpp0-5 rtk_fp_ufpp0-6 rtup0-7 rtup0-8 rtup_ufpp0-9 rtup_ufpp0-10 rtus0-11 rtus0-12 rtus_ufps0-13 rtus_ufps0-14 rtuo0-15 rtuo0-16 rtuo_ufpp0-17 rtuo_ufpp0-18 rtuo_ufps0-19 rtuo_ufps0-20 rtuo_ufpp_ufps0-21 rtuo_ufpp_ufps0-22 sleeper_ufps1-0 rtk1-1 rtk1-2 rtk_fp1-3 rtk_fp1-4 rtk_fp_ufpp1-5 rtk_fp_ufpp1-6 rtup1-7 rtup1-8 rtup_ufpp1-9 rtup_ufpp1-10 rtus1-11 rtus1-12 rtus_ufps1-13 rtus_ufps1-14 rtuo1-15 rtuo1-16 rtuo_ufpp1-17 rtuo_ufpp1-18 rtuo_ufps1-19 rtuo_ufps1-20 rtuo_ufpp_ufps1-21 rtuo_ufpp_ufps1-22
r0: 2147483648 != 10
r1: 2147483648 != 10
r2: 2147483648 != 10
r3: 2147483648 != 10
r4: 2147483648 != 10
r5: 2147483648 != 10
r6: 2147483648 != 10
r7: 39868544 != 10
Error after context switch from task 8(rtup1-8) to task 10(rtup_ufpp1-10),
FPU registers were set to 39868544 (maybe task ???)
RTT|  00:00:00
RTH|---------cpu|ctx switches|-------total
RTD|           0|          23|          23
RTD|           1|          28|          28

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2142 bytes --]

Jeff Webb wrote:
> Jan Kiszka wrote:
>> Uhh, wait! I just fired up switchtest on the same box:
>> ...
>> Gilles, what info do you need? It's all the latest on x86 here:
>> 2.6.17.13 (.config attached), ipipe-1.4, SVN #1646, gcc4.1.0.
>>
>> Jan
> 
> I get a similar result here with xenomai-2.2.3, 2.6.17.13, x86 SMP (see
> below).  No seg fault, though.
> 
> -Jeff
> 
> 
> *
> *
> * Type ^C to stop this application.
> *
> *
> == Testing FPU check routines...
> r0: 1 != 2
> r1: 1 != 2
> r2: 1 != 2
> r3: 1 != 2
> r4: 1 != 2
> r5: 1 != 2
> r6: 1 != 2
> r7: 1 != 2
> == FPU check routines: OK.
> == Threads: sleeper_ufps0-0 rtk0-1 rtk0-2 rtk_fp0-3 rtk_fp0-4
> rtk_fp_ufpp0-5 rtk_fp_ufpp0-6 rtup0-7 rtup0-8 rtup_ufpp0-9 rtup_ufpp0-10
> rtus0-11 rtus0-12 rtus_ufps0-13 rtus_ufps0-14 rtuo0-15 rtuo0-16
> rtuo_ufpp0-17 rtuo_ufpp0-18 rtuo_ufps0-19 rtuo_ufps0-20
> rtuo_ufpp_ufps0-21 rtuo_ufpp_ufps0-22 sleeper_ufps1-0 rtk1-1 rtk1-2
> rtk_fp1-3 rtk_fp1-4 rtk_fp_ufpp1-5 rtk_fp_ufpp1-6 rtup1-7 rtup1-8
> rtup_ufpp1-9 rtup_ufpp1-10 rtus1-11 rtus1-12 rtus_ufps1-13 rtus_ufps1-14
> rtuo1-15 rtuo1-16 rtuo_ufpp1-17 rtuo_ufpp1-18 rtuo_ufps1-19
> rtuo_ufps1-20 rtuo_ufpp_ufps1-21 rtuo_ufpp_ufps1-22
> r0: 2147483648 != 10
> r1: 2147483648 != 10
> r2: 2147483648 != 10
> r3: 2147483648 != 10
> r4: 2147483648 != 10
> r5: 2147483648 != 10
> r6: 2147483648 != 10
> r7: 39868544 != 10
> Error after context switch from task 8(rtup1-8) to task 10(rtup_ufpp1-10),
> FPU registers were set to 39868544 (maybe task ???)
> RTT|  00:00:00
> RTH|---------cpu|ctx switches|-------total
> RTD|           0|          23|          23
> RTD|           1|          28|          28
> 

Welcome to the club!

Everyone's busy behind the curtain to analyse what's going on here. So
far we thought it's a post 2.2.2 regresssion (SVN rev 1595 for trunk,
1597 for 2.2.x), but the issue may be more tricky.

Do you get the switchtest failure also with plain 2.2.2?

What processor do you use? I was only able to trigger my problems on
Pentium MMX and older or an Athlon 800 here. A Pentium III worked fine.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Jeff Webb]

Jan Kiszka wrote:
> Welcome to the club!
> 
> Everyone's busy behind the curtain to analyse what's going on here. So
> far we thought it's a post 2.2.2 regresssion (SVN rev 1595 for trunk,
> 1597 for 2.2.x), but the issue may be more tricky.
> 
> Do you get the switchtest failure also with plain 2.2.2?
> 
> What processor do you use? I was only able to trigger my problems on
> Pentium MMX and older or an Athlon 800 here. A Pentium III worked fine.
> 
> Jan
> 

I don't have a plain xenomai-2.2.2 kernel handy, but I do have a patched version.  I do not have any problems with switchtest for xenomai-2.2.2 with the following patches:

  https://mail.gna.org/public/xenomai-help/2006-09/msg00084.html
  https://mail.gna.org/public/xenomai-help/2006-09/msg00096.html

These patches are to nucleus pipe.c and pipe.h / rtai fifo.c and fifo.h.

I am running an Athlon64 X2 in with 32-bit Fedora Core 5.

-Jeff

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Jeff Webb]

Jeff Webb wrote:
> I don't have a plain xenomai-2.2.2 kernel handy, but I do have a patched 
> version.  I do not have any problems with switchtest for xenomai-2.2.2 
> with the following patches:
> 
>  https://mail.gna.org/public/xenomai-help/2006-09/msg00084.html
>  https://mail.gna.org/public/xenomai-help/2006-09/msg00096.html
> 
> These patches are to nucleus pipe.c and pipe.h / rtai fifo.c and fifo.h.
> 
> I am running an Athlon64 X2 in with 32-bit Fedora Core 5.

I have done a couple more tests using xenomai from subversion.  Here is a summary of all the results I have so far:

xenomai-2.2.2 (patched as described above): switchtest runs
svn/xenomai/trunk (r1586): switchtest runs
svn/xenomai/trunk (r1589): switchtest runs
xenomai-2.2.3 : switchtest fails

This is on my Fedora Core 5 (32-bit x86), linux-2.6.17.13 SMP, ipipe-1.3 (except 1.4 for 2.2.3), gcc-4.1.1, Athlon64 X2.

Hopefully this will help,

-Jeff

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Philippe Gerum]

On Tue, 2006-09-19 at 09:48 -0500, Jeff Webb wrote:
> Jeff Webb wrote:
> > I don't have a plain xenomai-2.2.2 kernel handy, but I do have a patched 
> > version.  I do not have any problems with switchtest for xenomai-2.2.2 
> > with the following patches:
> > 
> >  https://mail.gna.org/public/xenomai-help/2006-09/msg00084.html
> >  https://mail.gna.org/public/xenomai-help/2006-09/msg00096.html
> > 
> > These patches are to nucleus pipe.c and pipe.h / rtai fifo.c and fifo.h.
> > 
> > I am running an Athlon64 X2 in with 32-bit Fedora Core 5.
> 
> I have done a couple more tests using xenomai from subversion.  Here is a summary of all the results I have so far:
> 
> xenomai-2.2.2 (patched as described above): switchtest runs
> svn/xenomai/trunk (r1586): switchtest runs
> svn/xenomai/trunk (r1589): switchtest runs
> xenomai-2.2.3 : switchtest fails
> 
> This is on my Fedora Core 5 (32-bit x86), linux-2.6.17.13 SMP, ipipe-1.3 (except 1.4 for 2.2.3), gcc-4.1.1, Athlon64 X2.
> 
> Hopefully this will help,

It does, thanks. This confirms Jan's findings that something went wrong
from #1595. The issue is spotted now, somewhere in the FPU save/restore
code in the x86 support layer.

> 
> -Jeff
> 
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core
-- 
Philippe.

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Jeff Webb]

Philippe Gerum wrote:
> On Tue, 2006-09-19 at 09:48 -0500, Jeff Webb wrote:
>> Jeff Webb wrote:
>>> I don't have a plain xenomai-2.2.2 kernel handy, but I do have a patched 
>>> version.  I do not have any problems with switchtest for xenomai-2.2.2 
>>> with the following patches:
>>>
>>>  https://mail.gna.org/public/xenomai-help/2006-09/msg00084.html
>>>  https://mail.gna.org/public/xenomai-help/2006-09/msg00096.html
>>>
>>> These patches are to nucleus pipe.c and pipe.h / rtai fifo.c and fifo.h.
>>>
>>> I am running an Athlon64 X2 in with 32-bit Fedora Core 5.
>> I have done a couple more tests using xenomai from subversion.  Here is a summary of all the results I have so far:
>>
>> xenomai-2.2.2 (patched as described above): switchtest runs
>> svn/xenomai/trunk (r1586): switchtest runs
>> svn/xenomai/trunk (r1589): switchtest runs
>> xenomai-2.2.3 : switchtest fails
>>
>> This is on my Fedora Core 5 (32-bit x86), linux-2.6.17.13 SMP, ipipe-1.3 (except 1.4 for 2.2.3), gcc-4.1.1, Athlon64 X2.
>>
>> Hopefully this will help,
> 
> It does, thanks. This confirms Jan's findings that something went wrong
> from #1595. The issue is spotted now, somewhere in the FPU save/restore
> code in the x86 support layer.

I just rebuilt 2.2.3 with the latest FPU fix patch posted by Philippe:

  https://mail.gna.org/public/xenomai-help/2006-09/msg00157.html

This appears to fix the FPU switchtest bug on my Athlon64 X2 (2.6 kernel) machine.  Great job!

Thanks for the hard work,

Jeff

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Philippe Gerum]

On Wed, 2006-09-20 at 12:20 -0500, Jeff Webb wrote:
> Philippe Gerum wrote:
> > On Tue, 2006-09-19 at 09:48 -0500, Jeff Webb wrote:
> >> Jeff Webb wrote:
> >>> I don't have a plain xenomai-2.2.2 kernel handy, but I do have a patched 
> >>> version.  I do not have any problems with switchtest for xenomai-2.2.2 
> >>> with the following patches:
> >>>
> >>>  https://mail.gna.org/public/xenomai-help/2006-09/msg00084.html
> >>>  https://mail.gna.org/public/xenomai-help/2006-09/msg00096.html
> >>>
> >>> These patches are to nucleus pipe.c and pipe.h / rtai fifo.c and fifo.h.
> >>>
> >>> I am running an Athlon64 X2 in with 32-bit Fedora Core 5.
> >> I have done a couple more tests using xenomai from subversion.  Here is a summary of all the results I have so far:
> >>
> >> xenomai-2.2.2 (patched as described above): switchtest runs
> >> svn/xenomai/trunk (r1586): switchtest runs
> >> svn/xenomai/trunk (r1589): switchtest runs
> >> xenomai-2.2.3 : switchtest fails
> >>
> >> This is on my Fedora Core 5 (32-bit x86), linux-2.6.17.13 SMP, ipipe-1.3 (except 1.4 for 2.2.3), gcc-4.1.1, Athlon64 X2.
> >>
> >> Hopefully this will help,
> > 
> > It does, thanks. This confirms Jan's findings that something went wrong
> > from #1595. The issue is spotted now, somewhere in the FPU save/restore
> > code in the x86 support layer.
> 
> I just rebuilt 2.2.3 with the latest FPU fix patch posted by Philippe:
> 
>   https://mail.gna.org/public/xenomai-help/2006-09/msg00157.html
> 
> This appears to fix the FPU switchtest bug on my Athlon64 X2 (2.6 kernel) machine.  Great job!
> 
> Thanks for the hard work,
> 

Gilles did the work actually. I've just relayed it.

> Jeff
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core
-- 
Philippe.

Re: [Xenomai-core] rare brokenness in latency output - FPU bug!

[Jeff Webb]

Jeff Webb wrote:
> I get a similar result here with xenomai-2.2.3, 2.6.17.13, x86 SMP (see 
> below).  No seg fault, though.

I get similar results if I compile a UP kernel instead of SMP.  I thought that might be a factor (since it seems most folks don't have this issue), but it seems that is not the case.

-Jeff