[Xenomai-help] fpu issue

[Xenomai-help] fpu issue

[Steven Seeger]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

Guys, the FPU issue I've been having with a Geode GX1 and an AMD Geode
LX800 still persists. It seems that we have always had this bug, and
it's timing related. I am pretty sure that the issue revolves around the
fact that xenomai doesn't initialize the FPU in a userspace created
thread until the FPU is actually used. I realize there is a performance
gain to be had here by not having to switch it every time, but to
confirm I would like to make a patch that initializes the FPU when a
thread is created. Can someone please point me in the right direction?

 

Thanks,

Steven

 

PS - Sorry to Gilles for all the bother. J

 


[-- Attachment #2: Type: text/html, Size: 5172 bytes --]

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> Guys, the FPU issue I've been having with a Geode GX1 and an AMD Geode
> LX800 still persists. It seems that we have always had this bug, and
> it's timing related. I am pretty sure that the issue revolves around the
> fact that xenomai doesn't initialize the FPU in a userspace created
> thread until the FPU is actually used. I realize there is a performance
> gain to be had here by not having to switch it every time, but to
> confirm I would like to make a patch that initializes the FPU when a
> thread is created. Can someone please point me in the right direction?
> 
>  
> 
> Thanks,
> 
> Steven
> 
>  
> 
> PS - Sorry to Gilles for all the bother. J

No problem, really, the only reason why I did not answer is that I was
busy. But having this discussion on the list is better anyway.

I suggest you use the code I sent you to dump the FPU state in various
places. The first place to try is before the initialization of the two
volatile double variables in the user-space real-time thread. At this
point you should have a clean fpu state, if it is not the case, then you
will have to start dumping the FPU state in kernel-space.

-- 
					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

I can do that. I have to be careful where I use printk, though. I
noticed my board freezes if I try to do a printk in
__xnpod_fault_init_fpu() so that makes it harder. I am not exactly sure
what to look for because I've never done any assembler with the 387. Any
pointers would be helpful!

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Tuesday, January 20, 2009 12:20 AM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > Guys, the FPU issue I've been having with a Geode GX1 and an AMD
> Geode
> > LX800 still persists. It seems that we have always had this bug, and
> > it's timing related. I am pretty sure that the issue revolves around
> the
> > fact that xenomai doesn't initialize the FPU in a userspace created
> > thread until the FPU is actually used. I realize there is a
> performance
> > gain to be had here by not having to switch it every time, but to
> > confirm I would like to make a patch that initializes the FPU when a
> > thread is created. Can someone please point me in the right
> direction?
> >
> >
> >
> > Thanks,
> >
> > Steven
> >
> >
> >
> > PS - Sorry to Gilles for all the bother. J
> 
> No problem, really, the only reason why I did not answer is that I was
> busy. But having this discussion on the list is better anyway.
> 
> I suggest you use the code I sent you to dump the FPU state in various
> places. The first place to try is before the initialization of the two
> volatile double variables in the user-space real-time thread. At this
> point you should have a clean fpu state, if it is not the case, then
> you
> will have to start dumping the FPU state in kernel-space.
> 
> --
> 					    Gilles.

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Tuesday, January 20, 2009 12:20 AM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > Guys, the FPU issue I've been having with a Geode GX1 and an AMD
> Geode
> > LX800 still persists. It seems that we have always had this bug, and
> > it's timing related. I am pretty sure that the issue revolves around
> the
> > fact that xenomai doesn't initialize the FPU in a userspace created
> > thread until the FPU is actually used. I realize there is a
> performance
> > gain to be had here by not having to switch it every time, but to
> > confirm I would like to make a patch that initializes the FPU when a
> > thread is created. Can someone please point me in the right
> direction?
> >
> >
> >
> > Thanks,
> >
> > Steven
> >
> >
> >
> > PS - Sorry to Gilles for all the bother. J
> 
> No problem, really, the only reason why I did not answer is that I was
> busy. But having this discussion on the list is better anyway.
> 
> I suggest you use the code I sent you to dump the FPU state in various
> places. The first place to try is before the initialization of the two
> volatile double variables in the user-space real-time thread. At this
> point you should have a clean fpu state, if it is not the case, then
> you
> will have to start dumping the FPU state in kernel-space.
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> I can do that. I have to be careful where I use printk, though. I
> noticed my board freezes if I try to do a printk in
> __xnpod_fault_init_fpu() so that makes it harder. I am not exactly sure
> what to look for because I've never done any assembler with the 387. Any
> pointers would be helpful!

You should not need any assembler at all. The header I have sent you
implements two functions save_fp to save the fp environment, and dump_fp
to dump a saved fp environment. If your cpu has no fxsr support (which
you can check in /proc/cpuinfo), you should use old_save_fp and
dump_old_fp instead.

-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

I do not have fxsr support. I noticed that instruction being used and
didn't find it in my datasheet. I guess my question is you want me to
dump the fpu state before I initialize the volatile doubles. However, I
don't have a saved state at that point. What exactly do I pass in? Or do
you want me to save and then dump?

Thanks,
Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Tuesday, January 20, 2009 7:50 AM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > I can do that. I have to be careful where I use printk, though. I
> > noticed my board freezes if I try to do a printk in
> > __xnpod_fault_init_fpu() so that makes it harder. I am not exactly
> sure
> > what to look for because I've never done any assembler with the 387.
> Any
> > pointers would be helpful!
> 
> You should not need any assembler at all. The header I have sent you
> implements two functions save_fp to save the fp environment, and
> dump_fp
> to dump a saved fp environment. If your cpu has no fxsr support (which
> you can check in /proc/cpuinfo), you should use old_save_fp and
> dump_old_fp instead.
> 
> --
>                                                  Gilles.

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> I do not have fxsr support. I noticed that instruction being used and
> didn't find it in my datasheet. I guess my question is you want me to
> dump the fpu state before I initialize the volatile doubles. However, I
> don't have a saved state at that point. What exactly do I pass in? Or do
> you want me to save and then dump?

Yes, save then dump.

-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

[-- Attachment #1: Type: text/plain, Size: 1667 bytes --]

Hello everyone. I have the dumps here from the header file that Gilles
was kind enough to provide to me.

When the problem does not occur:

sizeof(fpreg_t): 12
cwd: 0000037f,  swd: 00000000
fip: 08048c3e,  fcs: 01d00073
foo: 0959961c,  fos: 0000007b
FPU registers stack, top at 0:
7: Empty
6: Empty
5: Empty
4: Empty
3: Empty
2: Empty
1: Empty
0: Empty

When the problem does occur:

sizeof(fpreg_t): 12
cwd: 0000037f,  swd: 00000000
fip: 08048c3e,  fcs: 01d00073
foo: 00000000,  fos: 00000000
FPU registers stack, top at 0:
7:  10.000000000
6:  25.000000000
5: 0
4: 0
3: 0
2: 0
1: 0
0: - inf.

What's interesting is that 10.0 and 25.0 are coming from my other task.
I will attach my code as well. You can see that I'm dumping these values
in func2, and it seems that the FPU is perhaps left in an undesirable
state by func?

This problem occurs on a geode GX1 and a geode LX800. I hate geodes.

Steven



> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Tuesday, January 20, 2009 7:56 AM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > I do not have fxsr support. I noticed that instruction being used
and
> > didn't find it in my datasheet. I guess my question is you want me
to
> > dump the fpu state before I initialize the volatile doubles.
However,
> I
> > don't have a saved state at that point. What exactly do I pass in?
Or
> do
> > you want me to save and then dump?
> 
> Yes, save then dump.
> 
> --
>                                                  Gilles.

[-- Attachment #2: double_test.cpp --]
[-- Type: application/octet-stream, Size: 1950 bytes --]

#include <sys/mman.h>
#include <iostream>
#include <native/task.h>
#include "i387.h"

static RT_TASK task,task2;

void func(void*)
{
    volatile double min = 10.0;
    volatile double val = 25.0;
    
    for(;;)
    {
        if(val > min) {
            
        }
        else std::cout << "shouldn't see this " << min << ' ' << val << std::endl;
        
        rt_task_sleep(100000);
    }
    
}

//static const double min = 5.0;
//static const double val = 20.0;

void _func2(void*)
{
    fpenv_t env;
    
    old_save_fp(&env);
    dump_old_fp(&env);
    const double min = 5.0;
    const double val = 20.0;
    //const unsigned long long min = 5;
    //const unsigned long long val = 20;
    
    unsigned char garbage[8] = { 0, 0, 0, 0, 0, 0, -8, -1 };
    
    //rt_task_sleep(500000000);

    std::cout << "crap: " << *(double*)garbage << std::endl;
    std::cout << "should see this " << min << ' ' << val << ' ' << (void*)&min << ' ' << (void*)&val << std::endl;
    std::cout << (int)(char)((char*)&min)[0] << ", " << 
    (int)(char)((char*)&min)[1] << ", " << 
    (int)(char)((char*)&min)[2] << ", " << 
    (int)(char)((char*)&min)[3] << ", " << 
    (int)(char)((char*)&min)[4] << ", " << 
    (int)(char)((char*)&min)[5] << ", " << 
    (int)(char)((char*)&min)[6] << ", " << 
    (int)(char)((char*)&min)[7] << ", " <<  std::endl;
    
    for(;;)
    {
        if(val > min) {
            
        }
        else {
            std::cout << "shouldn't see this " << min << ' ' << val << ' ' << (void*)&min << ' ' << (void*)&val  << std::endl;
        }
        
        rt_task_sleep(100000);
    }
}

void func2(void*)
{
    //volatile double d = 123.0;
    //d += 5.0;
    
    _func2(NULL);
}

int main()
{
    mlockall(MCL_CURRENT | MCL_FUTURE);
    
    rt_task_spawn(&task, "dummy", 0, 60, T_FPU, func, NULL);
    rt_task_spawn(&task2, "dummy2", 0, 50, T_FPU, func2, NULL);
    
    usleep(500000);
    
    return 0;
}

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> Hello everyone. I have the dumps here from the header file that Gilles
> was kind enough to provide to me.
> 
> When the problem does not occur:
> 
> sizeof(fpreg_t): 12
> cwd: 0000037f,  swd: 00000000
> fip: 08048c3e,  fcs: 01d00073
> foo: 0959961c,  fos: 0000007b
> FPU registers stack, top at 0:
> 7: Empty
> 6: Empty
> 5: Empty
> 4: Empty
> 3: Empty
> 2: Empty
> 1: Empty
> 0: Empty
> 
> When the problem does occur:
> 
> sizeof(fpreg_t): 12
> cwd: 0000037f,  swd: 00000000
> fip: 08048c3e,  fcs: 01d00073
> foo: 00000000,  fos: 00000000
> FPU registers stack, top at 0:
> 7:  10.000000000
> 6:  25.000000000
> 5: 0
> 4: 0
> 3: 0
> 2: 0
> 1: 0
> 0: - inf.
> 
> What's interesting is that 10.0 and 25.0 are coming from my other task.
> I will attach my code as well. You can see that I'm dumping these values
> in func2, and it seems that the FPU is perhaps left in an undesirable
> state by func?
> 
> This problem occurs on a geode GX1 and a geode LX800. I hate geodes.

Ok. So the FPU state is garbled. Could you put printks in kernel-space
to trace the various FPU operations made by xenomai, you can use strcmp
to check that current->comm is "double_test", and you should put
different printks in all branches of the FPU handling functions, to know
which branches are taken.

-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

> Ok. So the FPU state is garbled. Could you put printks in kernel-space
> to trace the various FPU operations made by xenomai, you can use
strcmp
> to check that current->comm is "double_test", and you should put
> different printks in all branches of the FPU handling functions, to
> know which branches are taken.

Thanks for the suggestion. I can do that. I will get back to you today.

Thanks,
Steven

Re: [Xenomai-help] fpu issue

[Steven Seeger]

In pod.c:

__xnpod_init_fpu() is not called ever. However, init_fpu is called once
in i387.c

After the program runs, __xnpod_giveup_fpu is called twice (remember, I
have two threads)

__xnpod_release_fpu() never called

I can't put a printk in __xnpod_switch_fpu() because it freezes
(continues to print forever)

Steven

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> In pod.c:
> 
> __xnpod_init_fpu() is not called ever. However, init_fpu is called once
> in i387.c

__xnpod_init_fpu is for kernel-space thread. For user-space threads, the
FPU initialization happens either in Linux code if the fault happens in
secondary mode, or in the fault handler if it happens in primary mode.

> 
> After the program runs, __xnpod_giveup_fpu is called twice (remember, I
> have two threads)
> 
> __xnpod_release_fpu() never called
> 
> I can't put a printk in __xnpod_switch_fpu() because it freezes
> (continues to print forever)

Which is why I suggested printing only if !strcmp(current->comm,
"double_test"), and why you should stop double_test as soon as you
detect an error. Furthermore, what interest me is what branches are
taken in the various ifs found in xnarch_save_fpu, xnarch_restore_fpu,
xnarch_enable_fpu.

-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

> __xnpod_init_fpu is for kernel-space thread. For user-space threads,
> the
> FPU initialization happens either in Linux code if the fault happens
in
> secondary mode, or in the fault handler if it happens in primary mode.

Interesting. Shouldn't rt_task_spawn() start up in primary mode even if
it's called in user space? Is it normal to start in secondary mode? I
know std::cout will migrate to secondary, but I do my fpu stuff before
that.
 
> Which is why I suggested printing only if !strcmp(current->comm,
> "double_test"), and why you should stop double_test as soon as you
> detect an error. Furthermore, what interest me is what branches are
> taken in the various ifs found in xnarch_save_fpu, xnarch_restore_fpu,
> xnarch_enable_fpu.

My current test is called double_test_wlib and I notice current->comm
only prints "double_test_wli" so I guess there is a limit to the number
of characters in it. I'll put that in there now.

I changed the program so it doesn't need to loop. Just doing everything
in func() and func2() one time is enough to see the problem.

I will let you know about the branches in the mentioned routines
shortly.

Steven

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
>> __xnpod_init_fpu is for kernel-space thread. For user-space threads,
>> the
>> FPU initialization happens either in Linux code if the fault happens
> in
>> secondary mode, or in the fault handler if it happens in primary mode.
> 
> Interesting. Shouldn't rt_task_spawn() start up in primary mode even if
> it's called in user space? Is it normal to start in secondary mode? I
> know std::cout will migrate to secondary, but I do my fpu stuff before
> that.

When you first use fpu in primary mode, a fault happen, is trapped in
ksrc/nucleus/pod.c and directly calls xnarch_init_fpu, not
__xnpod_init_fpu. Now, the fact that xnarch_init_fpu is not called may
indeed be your problem, but we have to know exactly what happens before
to understand why this happens.

-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

Well, is what I said correct that a new userspace thread created with
rt_task_spawn() should start in primary mode instead of secondary mode?

Also, there is no need to stop my test when it detects an issue because
all it does now is print something out and then quit. The issue is there
if the printout shows me the memory holding the double is corrupt. So
that makes it easier that I've further redcuded the complexity.

I will have you results in a few minutes. Thank you for helping me with
your suggestions.

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Wednesday, January 21, 2009 10:12 AM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> >> __xnpod_init_fpu is for kernel-space thread. For user-space
threads,
> >> the
> >> FPU initialization happens either in Linux code if the fault
happens
> > in
> >> secondary mode, or in the fault handler if it happens in primary
> mode.
> >
> > Interesting. Shouldn't rt_task_spawn() start up in primary mode even
> if
> > it's called in user space? Is it normal to start in secondary mode?
I
> > know std::cout will migrate to secondary, but I do my fpu stuff
> before
> > that.
> 
> When you first use fpu in primary mode, a fault happen, is trapped in
> ksrc/nucleus/pod.c and directly calls xnarch_init_fpu, not
> __xnpod_init_fpu. Now, the fact that xnarch_init_fpu is not called may
> indeed be your problem, but we have to know exactly what happens
before
> to understand why this happens.
> 
> --
>                                                  Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

I have your output and it is repeatable every tiem.

Good case:

i387.c init_fpu()

xnarch_save_fpu() enters and takes the following path:

if(!tcb->is_root)
if(task)
if(!wrap_test_fpu_used(task)) return;

then, I see xnarch_restore_fpu take the following path:

tcb->is_root is set to 1, so it goes to else
tcb->cr0_ts is true, so it calls stts() and returns;

bad case:

all the same, except tcb->cr0_ts is false. It leaves that if block,
calls clts(), and then frstor instruction is called because I don't have
fxsr

Is this helpful?

There are no calls to xnarch_enable_fpu or xnarch_init_fpu

Thanks,
Steven

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> I have your output and it is repeatable every tiem.
> 
> Good case:
> 
> i387.c init_fpu()
> 
> xnarch_save_fpu() enters and takes the following path:
> 
> if(!tcb->is_root)
> if(task)
> if(!wrap_test_fpu_used(task)) return;
> 
> then, I see xnarch_restore_fpu take the following path:
> 
> tcb->is_root is set to 1, so it goes to else
> tcb->cr0_ts is true, so it calls stts() and returns;
> 
> bad case:
> 
> all the same, except tcb->cr0_ts is false. It leaves that if block,
> calls clts(), and then frstor instruction is called because I don't have
> fxsr
> 
> Is this helpful?
> 
> There are no calls to xnarch_enable_fpu or xnarch_init_fpu

Errr. Some things are missing, this is only one context switch (probably
the last ?). I would be interested in all the context switches since the
beginning of the process (hopefully, there should not be too much of them).

-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

Gilles,

Just to clarify, that is all the information I have. I see a single call
to init_fpu() in i387.c, a single call to xnarch_save_fpu(), and a
single call to xnarch_restore_fpu().

It almost seems like the second thread doesn't know the FPU was used. I
should see at least two inits, but seeing one context switch may make
sense since I now break from my loops in both routines. Plus, with the
const doubles in func2(), the comparison is optimized out when I look at
the disassembled output.

Steven

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> Gilles,
> 
> Just to clarify, that is all the information I have. I see a single call
> to init_fpu() in i387.c, a single call to xnarch_save_fpu(), and a
> single call to xnarch_restore_fpu().
> 
> It almost seems like the second thread doesn't know the FPU was used. I
> should see at least two inits, but seeing one context switch may make
> sense since I now break from my loops in both routines. Plus, with the
> const doubles in func2(), the comparison is optimized out when I look at
> the disassembled output.

Yes, the strcmp trick sucks. You should trace the switch when
current->comm is either "double_test" or "gatekeeper/0".

-- 
					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

Gilles,

That gave me more stuff to look at. Let me sift through it and type up
my findings shortly.

Thanks again for all your help.

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Wednesday, January 21, 2009 12:49 PM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > Gilles,
> >
> > Just to clarify, that is all the information I have. I see a single
> call
> > to init_fpu() in i387.c, a single call to xnarch_save_fpu(), and a
> > single call to xnarch_restore_fpu().
> >
> > It almost seems like the second thread doesn't know the FPU was
used.
> I
> > should see at least two inits, but seeing one context switch may
make
> > sense since I now break from my loops in both routines. Plus, with
> the
> > const doubles in func2(), the comparison is optimized out when I
look
> at
> > the disassembled output.
> 
> Yes, the strcmp trick sucks. You should trace the switch when
> current->comm is either "double_test" or "gatekeeper/0".
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

The increased output has changed timing and now I never get the bad
case. I'll have to try and reduce some extra messages. :(

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Wednesday, January 21, 2009 12:49 PM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > Gilles,
> >
> > Just to clarify, that is all the information I have. I see a single
> call
> > to init_fpu() in i387.c, a single call to xnarch_save_fpu(), and a
> > single call to xnarch_restore_fpu().
> >
> > It almost seems like the second thread doesn't know the FPU was
used.
> I
> > should see at least two inits, but seeing one context switch may
make
> > sense since I now break from my loops in both routines. Plus, with
> the
> > const doubles in func2(), the comparison is optimized out when I
look
> at
> > the disassembled output.
> 
> Yes, the strcmp trick sucks. You should trace the switch when
> current->comm is either "double_test" or "gatekeeper/0".
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

Gilles,

I can't seem to reproduce the corruption anymore with gatekeeper and
double_test. I've really reduced what I print out but for some reason
there is no more scrambled FPU registers. What I do notice, however, is
that if(!wrap_test_fpu_used(task)) always evaluates to true in every
xnarch_save_fpu() which seems odd to me.

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Wednesday, January 21, 2009 12:49 PM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > Gilles,
> >
> > Just to clarify, that is all the information I have. I see a single
> call
> > to init_fpu() in i387.c, a single call to xnarch_save_fpu(), and a
> > single call to xnarch_restore_fpu().
> >
> > It almost seems like the second thread doesn't know the FPU was
used.
> I
> > should see at least two inits, but seeing one context switch may
make
> > sense since I now break from my loops in both routines. Plus, with
> the
> > const doubles in func2(), the comparison is optimized out when I
look
> at
> > the disassembled output.
> 
> Yes, the strcmp trick sucks. You should trace the switch when
> current->comm is either "double_test" or "gatekeeper/0".
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

I spoke too soon. I managed to make it fail by removing the break in
func() in my test, so that it does its comparison over and over again.

The failure case starts off like the normal case where xnarch_save_fpu
always exists early due to wrap_test_fpu_used always returning false.
However, when it fails, as I pointed out earlier it seems that the
cr0_ts is not set so stts() is not called in restore. I never see this
case when things work properly. When it fails, I see two restores in a
row where the task switch bit is not set in the FPU.

Where would I want to look into this?

Thanks,
Steven

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> I spoke too soon. I managed to make it fail by removing the break in
> func() in my test, so that it does its comparison over and over again.
> 
> The failure case starts off like the normal case where xnarch_save_fpu
> always exists early due to wrap_test_fpu_used always returning false.
> However, when it fails, as I pointed out earlier it seems that the
> cr0_ts is not set so stts() is not called in restore. I never see this
> case when things work properly. When it fails, I see two restores in a
> row where the task switch bit is not set in the FPU.
> 
> Where would I want to look into this?

Please send me traces, so that I can understand what goes wrong.

-- 
					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

After I see init_fpu() called, this is what I see:

s4r6s4r6s4r6s4r7s4r7s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6
s4r6s4r6

"s4" is printed in xnarch_save_fpu() right after
if(!wrap_test_fpu_used(task)) where it is false and causes the save
function to quit

"r6" is printed in xnarch_restore_fpu() right after if(tcb->cr0_ts)
evaluates to 1

"r7" is printed in xnarch_restore_fpu() right after if(tcb->cr0_ts)
evalutes to 0

These calls are made from __xnpod_switch_fpu()

There are no calls to xnarch_enable_fpu()

It seems to me that we are missing an fpu init, and it seems that
xnarch_save_fpu() never actually saves because fpu used is never set.

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Wednesday, January 21, 2009 2:14 PM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > I spoke too soon. I managed to make it fail by removing the break in
> > func() in my test, so that it does its comparison over and over
again.
> >
> > The failure case starts off like the normal case where
> xnarch_save_fpu
> > always exists early due to wrap_test_fpu_used always returning
false.
> > However, when it fails, as I pointed out earlier it seems that the
> > cr0_ts is not set so stts() is not called in restore. I never see
> this
> > case when things work properly. When it fails, I see two restores in
> a
> > row where the task switch bit is not set in the FPU.
> >
> > Where would I want to look into this?
> 
> Please send me traces, so that I can understand what goes wrong.
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> After I see init_fpu() called, this is what I see:
> 
> s4r6s4r6s4r6s4r7s4r7s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6
> s4r6s4r6
> 
> "s4" is printed in xnarch_save_fpu() right after
> if(!wrap_test_fpu_used(task)) where it is false and causes the save
> function to quit
> 
> "r6" is printed in xnarch_restore_fpu() right after if(tcb->cr0_ts)
> evaluates to 1
> 
> "r7" is printed in xnarch_restore_fpu() right after if(tcb->cr0_ts)
> evalutes to 0
> 
> These calls are made from __xnpod_switch_fpu()
> 
> There are no calls to xnarch_enable_fpu()
> 
> It seems to me that we are missing an fpu init, and it seems that
> xnarch_save_fpu() never actually saves because fpu used is never set.

Ok. Please let me some time to try and understand. In the meantime,
could you put a printk in the kernel_fpu_begin function to see if it is
ever called?

-- 
					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

Yes I can. Let me also point out that with my trace, the if below "r7"
in if(tcb->usedfpu) never enters.

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Wednesday, January 21, 2009 2:27 PM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > After I see init_fpu() called, this is what I see:
> >
> >
>
s4r6s4r6s4r6s4r7s4r7s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r6s4r
> 6
> > s4r6s4r6
> >
> > "s4" is printed in xnarch_save_fpu() right after
> > if(!wrap_test_fpu_used(task)) where it is false and causes the save
> > function to quit
> >
> > "r6" is printed in xnarch_restore_fpu() right after if(tcb->cr0_ts)
> > evaluates to 1
> >
> > "r7" is printed in xnarch_restore_fpu() right after if(tcb->cr0_ts)
> > evalutes to 0
> >
> > These calls are made from __xnpod_switch_fpu()
> >
> > There are no calls to xnarch_enable_fpu()
> >
> > It seems to me that we are missing an fpu init, and it seems that
> > xnarch_save_fpu() never actually saves because fpu used is never
set.
> 
> Ok. Please let me some time to try and understand. In the meantime,
> could you put a printk in the kernel_fpu_begin function to see if it
is
> ever called?
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

Yes, kernel_fpu_begin() is being called. 

Steven
 
> Ok. Please let me some time to try and understand. In the meantime,
> could you put a printk in the kernel_fpu_begin function to see if it
is
> ever called?
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> Yes, kernel_fpu_begin() is being called. 

If kernel_fpu_begin() is called during the run-time (and not only at
initialization time when benchmarking RAID functions), then it is
probably the source of the problem. This situation was not well tested.

-- 
					    Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

It is being called a lot. Running my test with the break in both loops
yields approximately 60 calls to it. Half of those are called even
before i387.c's init_fpu().

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Wednesday, January 21, 2009 2:56 PM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > Yes, kernel_fpu_begin() is being called.
> 
> If kernel_fpu_begin() is called during the run-time (and not only at
> initialization time when benchmarking RAID functions), then it is
> probably the source of the problem. This situation was not well
tested.
> 
> --
> 					    Gilles.

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
> It is being called a lot. Running my test with the break in both loops
> yields approximately 60 calls to it. Half of those are called even
> before i387.c's init_fpu().

For the records: we found a solution to this FPU issue. The issue was
two-fold:
- when xenomai was preempting some linux kernel code which has called
kernel_fpu_begin, it was saving the current FPU context over the current
thread user-space FPU backup area, erasing its value previously saved by
kernel_fpu_begin();
- xenomai could preempt the linux kernel anywhere within the
kernel_fpu_begin function, which caused it to observe an incoherent
state (namely incoherent values of
current_thread_info()->status & TS_USEDFPU and TS bit in CR0).

These issues were easy to observe on Geode, because Linux on these
processors use MMX, thus kernel_fpu_begin routinely.

So, the following patch should be integrated into the I-pipe patch:

diff --git a/include/asm-x86/i387.h b/include/asm-x86/i387.h
index 56d00e3..e850fa1 100644
--- a/include/asm-x86/i387.h
+++ b/include/asm-x86/i387.h
@@ -222,11 +222,14 @@ static inline void __clear_fpu(struct task_struct
*tsk)
 static inline void kernel_fpu_begin(void)
 {
 	struct thread_info *me = current_thread_info();
+	unsigned long flags;
 	preempt_disable();
+	local_irq_save_hw_cond(flags);
 	if (me->status & TS_USEDFPU)
 		__save_init_fpu(me->task);
 	else
 		clts();
+	local_irq_restore_hw_cond(flags);
 }

 static inline void kernel_fpu_end(void)


And the following patch should be applied to Xenomai:

Index: include/asm-x86/bits/pod_32.h
===================================================================
--- include/asm-x86/bits/pod_32.h	(revision 4572)
+++ include/asm-x86/bits/pod_32.h	(working copy)
@@ -63,7 +63,12 @@ static inline void xnarch_leave_root(xna
 	rootcb->ts_usedfpu = wrap_test_fpu_used(current) != 0;
 	rootcb->cr0_ts = (read_cr0() & 8) != 0;
 	/* So that xnarch_save_fpu() will operate on the right FPU area. */
-	rootcb->fpup = x86_fpustate_ptr(&rootcb->user_task->thread);
+	if (rootcb->cr0_ts || rootcb->ts_usedfpu)
+		rootcb->fpup = x86_fpustate_ptr(&rootcb->user_task->thread);
+	else
+		/* the kernel is currently using fpu in kernel-space, do not
+		   clobber the user-space fpu backup area. */
+		rootcb->fpup = &rootcb->i387;
 }

 #define xnarch_enter_root(rootcb)  do { } while(0)





-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue on PPC ?

[roderik.wildenburg]

As you certainly allready know, we are facing some FPU problems on PPC (2.4 Kernel) too (switchtest reports inconsistent FPU-register; our application crashs with FPU-exception or shows sometimes unreasonable FP-values). 
Is the solution you found for i386 applicable for PPC too (does the issue, you described, apply to PPC too) ?

Roderik


> -----Ursprüngliche Nachricht-----
> Von: xenomai-help-bounces@domain.hid
> [mailto:xenomai-help-bounces@domain.hid] Im Auftrag von Gilles 
> Chanteperdrix
> Gesendet: Montag, 26. Januar 2009 17:13
> An: Steven Seeger
> Cc: xenomai@xenomai.org; Adeos
> Betreff: Re: [Xenomai-help] fpu issue
> 
> Steven Seeger wrote:
> > It is being called a lot. Running my test with the break in 
> both loops
> > yields approximately 60 calls to it. Half of those are called even
> > before i387.c's init_fpu().
> 
> For the records: we found a solution to this FPU issue. The issue was
> two-fold:
> - when xenomai was preempting some linux kernel code which has called
> kernel_fpu_begin, it was saving the current FPU context over 
> the current
> thread user-space FPU backup area, erasing its value 
> previously saved by
> kernel_fpu_begin();
> - xenomai could preempt the linux kernel anywhere within the
> kernel_fpu_begin function, which caused it to observe an incoherent
> state (namely incoherent values of
> current_thread_info()->status & TS_USEDFPU and TS bit in CR0).
> 
> These issues were easy to observe on Geode, because Linux on these
> processors use MMX, thus kernel_fpu_begin routinely.
> 
> So, the following patch should be integrated into the I-pipe patch:
> 
> diff --git a/include/asm-x86/i387.h b/include/asm-x86/i387.h
> index 56d00e3..e850fa1 100644
> --- a/include/asm-x86/i387.h
> +++ b/include/asm-x86/i387.h
> @@ -222,11 +222,14 @@ static inline void __clear_fpu(struct 
> task_struct
> *tsk)
>  static inline void kernel_fpu_begin(void)
>  {
>  	struct thread_info *me = current_thread_info();
> +	unsigned long flags;
>  	preempt_disable();
> +	local_irq_save_hw_cond(flags);
>  	if (me->status & TS_USEDFPU)
>  		__save_init_fpu(me->task);
>  	else
>  		clts();
> +	local_irq_restore_hw_cond(flags);
>  }
> 
>  static inline void kernel_fpu_end(void)
> 
> 
> And the following patch should be applied to Xenomai:
> 
> Index: include/asm-x86/bits/pod_32.h
> ===================================================================
> --- include/asm-x86/bits/pod_32.h	(revision 4572)
> +++ include/asm-x86/bits/pod_32.h	(working copy)
> @@ -63,7 +63,12 @@ static inline void xnarch_leave_root(xna
>  	rootcb->ts_usedfpu = wrap_test_fpu_used(current) != 0;
>  	rootcb->cr0_ts = (read_cr0() & 8) != 0;
>  	/* So that xnarch_save_fpu() will operate on the right 
> FPU area. */
> -	rootcb->fpup = x86_fpustate_ptr(&rootcb->user_task->thread);
> +	if (rootcb->cr0_ts || rootcb->ts_usedfpu)
> +		rootcb->fpup = 
> x86_fpustate_ptr(&rootcb->user_task->thread);
> +	else
> +		/* the kernel is currently using fpu in 
> kernel-space, do not
> +		   clobber the user-space fpu backup area. */
> +		rootcb->fpup = &rootcb->i387;
>  }
> 
>  #define xnarch_enter_root(rootcb)  do { } while(0)
> 
> 
> 
> 
> 
> -- 
>                                                  Gilles.
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
>

--------------------------------------------------------
manroland AG
Vorsitzender des Aufsichtsrates: Hanno C. Fiedler
Vorstand: Gerd Finkbeiner (Vorsitzender), Dr. Ingo Koch, Dr. Markus Rall, Paul Steidle   
Sitz der Gesellschaft: Offenbach am Main, Registergericht: Amtsgericht Offenbach HRB-Nr. 42592
USt-Ident-Nr. DE 250200933

Re: [Xenomai-help] fpu issue on PPC ?

[Gilles Chanteperdrix]

roderik.wildenburg@domain.hid wrote:
> As you certainly allready know, we are facing some FPU problems on
> PPC (2.4 Kernel) too (switchtest reports inconsistent FPU-register;
> our application crashs with FPU-exception or shows sometimes
> unreasonable FP-values). Is the solution you found for i386
> applicable for PPC too (does the issue, you described, apply to PPC
> too) ?

No. The FPU handling is very different on ppc and x86. I far as I know,
the PPC kernel only uses Altivec in kernel-space which is handled
separately from plain FPU registers and not touched at all by Xenomai.

I am investigating the ppc fpu issue, and the current situation is that
the FPU handling on ppc is indeed broken. I am working on a fix.

-- 
                                                 Gilles.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

Oh, and I never see __xnpod_fault_init_fpu() called either. It looks
like Linux is trapping this and initializing the FPU in i387.c

> -----Original Message-----
> From: Steven Seeger
> Sent: Wednesday, January 21, 2009 9:42 AM
> To: 'Gilles Chanteperdrix'
> Cc: xenomai@xenomai.org
> Subject: RE: [Xenomai-help] fpu issue
> 
> In pod.c:
> 
> __xnpod_init_fpu() is not called ever. However, init_fpu is called
once
> in i387.c
> 
> After the program runs, __xnpod_giveup_fpu is called twice (remember,
I
> have two threads)
> 
> __xnpod_release_fpu() never called
> 
> I can't put a printk in __xnpod_switch_fpu() because it freezes
> (continues to print forever)
> 
> Steven

[Xenomai-help] fpu issue

[Steven Seeger]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

I found the source of my fpu issue. A single kernel module with two
tasks was also active. I noticed the failure occurred whenever sound
would play. (This is one of the tasks.) It seems the problem is that the
sound task doesn't use the FPU, so I didn't turn on T_FPU. The tone task
(not in use at that time) does, so it has T_FPU. When I turned on T_FPU
for the sound task, I never saw the problem again. With it off, if I
played sound while running the program, I'd regularly see the
corruption.

 

Here is the task code:

 

static void sound_func(void *data)

{       

    for (;;) //go until suspended or thread deletion

    { 

        if (play==stop) //time to stop if this happens

        {

            unsigned long evt = 0;

            

            play=stop=0; // just in case

            rt_event_clear(&sound_event, SOUND, 0);

            rt_task_set_periodic(&sound_task, TM_NOW, TM_INFINITE); /*
disable periodic mode */

            rt_event_wait(&sound_event, SOUND, &evt, EV_ALL,
TM_INFINITE);

            rt_task_set_periodic(&sound_task, TM_NOW,
rt_timer_ns2ticks(ST_AUDIO_PERIOD));

        }

        

#if BUFSIZE != 65536

        if (play==BUFSIZE) play=0;

#endif

        

        if(rt_task_wait_period(0)) rt_task_wait_period(0); //wait for
next tick and ignore overruns (do the best we can)

        // Since the index is incrementd subtract 2 from the size before
comparing the index

        

        outb(buf[play++], DAC_ADDRESS); //send the next byte to the dac

    }

}

 

The buffer is filled with a linux userland process and an appropriate
wait queue. This same code worked fine in RTAI without the FPU flag.

 

I've tested this for several minutes.

 

Any idea?

 

Steven

 


[-- Attachment #2: Type: text/html, Size: 15394 bytes --]

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
 > I found the source of my fpu issue. A single kernel module with two
 > tasks was also active. I noticed the failure occurred whenever sound
 > would play. (This is one of the tasks.) It seems the problem is that the
 > sound task doesn't use the FPU, so I didn't turn on T_FPU. The tone task
 > (not in use at that time) does, so it has T_FPU. When I turned on T_FPU
 > for the sound task, I never saw the problem again. With it off, if I
 > played sound while running the program, I'd regularly see the
 > corruption.
 > 
 >  
 > 
 > Here is the task code:
 > 
 >  
 > 
 > static void sound_func(void *data)
 > 
 > {       
 > 
 >     for (;;) //go until suspended or thread deletion
 > 
 >     { 
 > 
 >         if (play==stop) //time to stop if this happens
 > 
 >         {
 > 
 >             unsigned long evt = 0;
 > 
 >             
 > 
 >             play=stop=0; // just in case
 > 
 >             rt_event_clear(&sound_event, SOUND, 0);
 > 
 >             rt_task_set_periodic(&sound_task, TM_NOW, TM_INFINITE); /*
 > disable periodic mode */
 > 
 >             rt_event_wait(&sound_event, SOUND, &evt, EV_ALL,
 > TM_INFINITE);
 > 
 >             rt_task_set_periodic(&sound_task, TM_NOW,
 > rt_timer_ns2ticks(ST_AUDIO_PERIOD));
 > 
 >         }
 > 
 >         
 > 
 > #if BUFSIZE != 65536
 > 
 >         if (play==BUFSIZE) play=0;
 > 
 > #endif
 > 
 >         
 > 
 >         if(rt_task_wait_period(0)) rt_task_wait_period(0); //wait for
 > next tick and ignore overruns (do the best we can)
 > 
 >         // Since the index is incrementd subtract 2 from the size before
 > comparing the index
 > 
 >         
 > 
 >         outb(buf[play++], DAC_ADDRESS); //send the next byte to the dac
 > 
 >     }
 > 
 > }
 > 
 >  
 > 
 > The buffer is filled with a linux userland process and an appropriate
 > wait queue. This same code worked fine in RTAI without the FPU flag.
 > 
 >  
 > 
 > I've tested this for several minutes.
 > 
 >  
 > 
 > Any idea?

An idea is to disassemble the kernel task code and the function it calls
to see if FPU is really never used.

If this task really never calls any function using FPU, it would be nice
if you could (as we always ask, you may have noticed) reduce your test
case to a simple code where only FPU operations and every thing which
causes task switches (blocking calls, and calls waking up other tasks)
is kept.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] fpu issue

[Steven Seeger]

 > An idea is to disassemble the kernel task code and the function it
calls
> to see if FPU is really never used.
> 
> If this task really never calls any function using FPU, it would be
nice
> if you could (as we always ask, you may have noticed) reduce your test
> case to a simple code where only FPU operations and every thing which
> causes task switches (blocking calls, and calls waking up other tasks)
> is kept.

My application has 12 threads most of which wait on queues. How am I
going to reduce this to a simple case? Most of my threads are userspace
except for 2. 

I guess I'll see what I can do. I'll try the disassembler, too.

Steven 

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
 >  > An idea is to disassemble the kernel task code and the function it
 > calls
 > > to see if FPU is really never used.
 > > 
 > > If this task really never calls any function using FPU, it would be
 > nice
 > > if you could (as we always ask, you may have noticed) reduce your test
 > > case to a simple code where only FPU operations and every thing which
 > > causes task switches (blocking calls, and calls waking up other tasks)
 > > is kept.
 > 
 > My application has 12 threads most of which wait on queues. How am I
 > going to reduce this to a simple case? Most of my threads are userspace
 > except for 2. 

You should start with an example having the 12 same threads as the real
code, then try removing threads one by one and look if the bug is still
there. If you need the 12 threads, then send us your example with the 12
threads.

 > 
 > I guess I'll see what I can do. I'll try the disassembler, too.
 > 
 > Steven 


-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] fpu issue

[Gilles Chanteperdrix]

Steven Seeger wrote:
 >  > An idea is to disassemble the kernel task code and the function it
 > calls
 > > to see if FPU is really never used.
 > > 
 > > If this task really never calls any function using FPU, it would be
 > nice
 > > if you could (as we always ask, you may have noticed) reduce your test
 > > case to a simple code where only FPU operations and every thing which
 > > causes task switches (blocking calls, and calls waking up other tasks)
 > > is kept.
 > 
 > My application has 12 threads most of which wait on queues. How am I
 > going to reduce this to a simple case? Most of my threads are userspace
 > except for 2. 
 > 
 > I guess I'll see what I can do. I'll try the disassembler, too.

It was a bad idea: if ever your task had used FPU without the FPU bit,
Xenomai would have suspended this task, with a warning on the console.

-- 


					    Gilles Chanteperdrix.