FP in kernelspace

FP in kernelspace

[Jiri Slaby]

Hello,

I have a driver written for 2.4 + RT patches with FP support. I want it to work 
in 2.6. How to implement FP? Has anybody developped some "protocol" between KS 
and US yet? If not, could somebody point me, how to do it the best -- with low 
latency.
The device doesn't generate irqs *), I need to quickly respond to timer call, 
because interval between two posts of data to the device has to be equal as much 
as possible (BTW is there any way how to gain up to 5000Hz).
I've one idea: have a thread with RT priority and wake the app in US waiting in 
read of character device when timer ticks, post a struct with 2 floats and 
operation and wait in write for the result. App computes, writes the result, we 
are woken and can post it to the device. But I'm afraid it would be tooo slow.

*) I don't know how to persuade it (standard PLX chip with unknown piece of 
logic behind) to generate, because official driver is closed and _very_ 
expensive. Old (2.4) driver was implemented with RT thread and timer, where FP 
is implemented within RT and computed directly in KS.

So 2 questions are:
1) howto FP in kernel
2) howto precise timer (may mingo RT patches help?)
3) any way to have faster ticks (up to 5000Hz)?

Any suggestions, please?

thanks,
-- 
<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Re: FP in kernelspace

[Jiri Slaby]

Jiri Slaby wrote:
> Hello,
> 
> I have a driver written for 2.4 + RT patches with FP support. I want it 
> to work in 2.6. How to implement FP? Has anybody developped some 
> "protocol" between KS and US yet? If not, could somebody point me, how 
> to do it the best -- with low latency.
One more thing, it can't be table-precomputed, positions are read from the 
device, needs to be recomputed and pushed back.

regards,
-- 
<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Re: FP in kernelspace

[Arjan van de Ven]

On Sun, 2006-07-30 at 13:27 +0159, Jiri Slaby wrote:
> 
> I have a driver written for 2.4 + RT patches with FP support. I want
> it to work 
> in 2.6. How to implement FP? Has anybody developped some "protocol"
> between KS 
> and US yet? If not, could somebody point me, how to do it the best --
> with low 
> latency.
> The device doesn't generate irqs *), I need to quickly respond to
> timer call, 
> because interval between two posts of data to the device has to be
> equal as much 
> as possible (BTW is there any way how to gain up to 5000Hz).
> I've one idea: have a thread with RT priority and wake the app in US
> waiting in 
> read of character device when timer ticks, post a struct with 2 floats
> and 
> operation and wait in write for the result. App computes, writes the
> result, we 
> are woken and can post it to the device. But I'm afraid it would be
> tooo slow.

real floating point in the kernel is basically no-go; especially if you
enable preemption. The entire floating point exception, context save and
restore engine is designed with the assumption of no kernel space FPU
usage... and there are a LOT of nasty corner cases to deal with... and
so far the kernel decided to cheap out on many of those by just not
allowing kernel FPU use (mmx is easier so that can be dealt with).

Volume 3 of the Intel architecture manuals are a good start if you want
to know what some of these corner cases are...


-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com

Re: FP in kernelspace

[Avi Kivity]

Jiri Slaby wrote:
>
> Hello,
>
> I have a driver written for 2.4 + RT patches with FP support. I want 
> it to work
> in 2.6. How to implement FP? Has anybody developped some "protocol" 
> between KS
> and US yet? If not, could somebody point me, how to do it the best -- 
> with low
> latency.
> The device doesn't generate irqs *), I need to quickly respond to 
> timer call,
> because interval between two posts of data to the device has to be 
> equal as much
> as possible (BTW is there any way how to gain up to 5000Hz).
> I've one idea: have a thread with RT priority and wake the app in US 
> waiting in
> read of character device when timer ticks, post a struct with 2 floats 
> and
> operation and wait in write for the result. App computes, writes the 
> result, we
> are woken and can post it to the device. But I'm afraid it would be 
> tooo slow.
>
> *) I don't know how to persuade it (standard PLX chip with unknown 
> piece of
> logic behind) to generate, because official driver is closed and _very_
> expensive. Old (2.4) driver was implemented with RT thread and timer, 
> where FP
> is implemented within RT and computed directly in KS.
>
> So 2 questions are:
> 1) howto FP in kernel
>
kernel_fpu_begin();
c = d * 3.14;
kernel_fpu_end();

-- 
error compiling committee.c: too many arguments to function

Re: FP in kernelspace

[Arjan van de Ven]

> > So 2 questions are:
> > 1) howto FP in kernel
> >
> kernel_fpu_begin();
> c = d * 3.14;
> kernel_fpu_end();
> 

unfortunately this only works for MMX not for real fpu (due to exception
handling uglies)

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com

Re: FP in kernelspace

[Jiri Slaby]

Arjan van de Ven wrote:
>>> So 2 questions are:
>>> 1) howto FP in kernel
>>>
>> kernel_fpu_begin();
>> c = d * 3.14;
>> kernel_fpu_end();

Yup, I know about this possibility, but this is only x86 specific?!

> unfortunately this only works for MMX not for real fpu (due to exception
> handling uglies)

concludes it's not multiplatform at all... For that reasen I (maybe) want some 
"protocol" for communication with US, where I can easily compute it.

Another way could be rtai (there is FP implemented IIRC), but it means having 
out-of-kernel driver.

thanks,
-- 
<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Re: FP in kernelspace

[Arjan van de Ven]

On Sun, 2006-07-30 at 17:08 +0159, Jiri Slaby wrote:
> Arjan van de Ven wrote:
> >>> So 2 questions are:
> >>> 1) howto FP in kernel
> >>>
> >> kernel_fpu_begin();
> >> c = d * 3.14;
> >> kernel_fpu_end();
> 
> Yup, I know about this possibility, but this is only x86 specific?!
> 
> > unfortunately this only works for MMX not for real fpu (due to exception
> > handling uglies)
> 
> concludes it's not multiplatform at all... For that reasen I (maybe) want some 
> "protocol" for communication with US, where I can easily compute it.]

easiest might be a shared mmap ring buffer (well 2 of these actually),
kernel puts data in buffer, updates ring pointer (and wakes userspace if
needed). Userspace notices data in ring, does math, puts data back in
the 2nd ringbuffer and updates that ring pointer. Then maybe does a
syscall to 'wake' the kernel. Kernel then sees data in the 2nd
ringbuffer and gets it. In addition to the wakeups I mentioned you can
also poll for the buffers; the wakeups are just there for extra safety.

Re: FP in kernelspace

[Avi Kivity]

Jiri Slaby wrote:
>
> > unfortunately this only works for MMX not for real fpu (due to 
> exception
> > handling uglies)
>
> concludes it's not multiplatform at all... For that reasen I (maybe) 
> want some
> "protocol" for communication with US, where I can easily compute it.
>
Well, usually such a protocol is by means of read(2) and write(2) to a 
character device provided by your driver: userspace reads some data, 
does the fp ops, and writes it back.  If you want realtime you'll have 
to use a realtime thread with the appropriate priority.

-- 
error compiling committee.c: too many arguments to function

Re: FP in kernelspace

[Avi Kivity]

Arjan van de Ven wrote:
>
> > > So 2 questions are:
> > > 1) howto FP in kernel
> > >
> > kernel_fpu_begin();
> > c = d * 3.14;
> > kernel_fpu_end();
> >
>
> unfortunately this only works for MMX not for real fpu (due to exception
> handling uglies)
>

Perhaps there should be a comment to that effect?  Neither the code nor 
Documentation/preemt-locking (which mentions the fpu) says anything 
about this little fact.

It's also broken for x86-64, which uses sse for floating point, not the 
x87 fpu.

-- 
error compiling committee.c: too many arguments to function

Re: FP in kernelspace

[Andi Kleen]

Avi Kivity <avi@argo.co.il> writes:
> 
> It's also broken for x86-64, which uses sse for floating point, not
> the x87 fpu.

Sorry, that doesn't make sense.

-Andi

Re: FP in kernelspace

[Avi Kivity]

Andi Kleen wrote:
>
> Avi Kivity <avi@argo.co.il> writes:
> >
> > It's also broken for x86-64, which uses sse for floating point, not
> > the x87 fpu.
>
> Sorry, that doesn't make sense.
>

 > > kernel_fpu_begin();
 > > c = d * 3.14;
 > > kernel_fpu_end();

Cannot work on x86-64, even disregarding fp exceptions, because 
kernel_fpu_begin() doesn't save the sse state which is used by fp math.

No?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: FP in kernelspace

[Andi Kleen]

> 
> Cannot work on x86-64, even disregarding fp exceptions, because 
> kernel_fpu_begin() doesn't save the sse state which is used by fp math.
> 
> No?

It does - FXSAVE saves everything.

BTW you can use x87 on x86-64 too, you just need to use long double.

-Andi

Re: FP in kernelspace

[Avi Kivity]

Andi Kleen wrote:
>
> >
> > Cannot work on x86-64, even disregarding fp exceptions, because
> > kernel_fpu_begin() doesn't save the sse state which is used by fp math.
> >
> > No?
>
> It does - FXSAVE saves everything.
>

I see.  Thanks for clearing it up.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: FP in kernelspace

[Jan Engelhardt]

>
> Cannot work on x86-64, even disregarding fp exceptions, because
> kernel_fpu_begin() doesn't save the sse state which is used by fp math.
>
> No?

No. You can - if you want - exclusively use x87 math on x64 (at least for 
userspace) and don't care about SSE. Otherwise, GCC having a -mfpmath=387 
would be pretty useless.

No?


Jan Engelhardt
-- 

Re: FP in kernelspace

[Andi Kleen]

Arjan van de Ven <arjan@infradead.org> writes:
> 
> unfortunately this only works for MMX not for real fpu (due to exception
> handling uglies)

He can always disable all FPU exceptions in his code or make sure
there aren't any.

-Andi

Re: FP in kernelspace

[Jan Engelhardt]

> kernel_fpu_begin();
> c = d * 3.14;
> kernel_fpu_end();

static inline void kernel_fpu_begin() {
    ...
    preempt_disable();
    ...
}


Jan Engelhardt
-- 

Re: FP in kernelspace

[Avi Kivity]

Jan Engelhardt wrote:
>
> > kernel_fpu_begin();
> > c = d * 3.14;
> > kernel_fpu_end();
>
> static inline void kernel_fpu_begin() {
>     ...
>     preempt_disable();
>     ...
> }
>
Is the kernel allowed to clobber userspace's sse registers?

What about interrupt code?

xor.h at least appears to save the sse state before use.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: FP in kernelspace

[Jan Engelhardt]

>> > kernel_fpu_begin();
>> > c = d * 3.14;
>> > kernel_fpu_end();
>> 
>> static inline void kernel_fpu_begin() {
>> ...
>> preempt_disable();
>> ...
>> }
>> 
> Is the kernel allowed to clobber userspace's sse registers?

As long as you save and restore it properly, and make it look like to all 
other threads that nothing happened, you are (I hope) free to do anything.

> What about interrupt code?

You do not want to go there...

> xor.h at least appears to save the sse state before use.
>
> -- 
> Do not meddle in the internals of kernels, for they are subtle and quick to
> panic.

Follow that advice, don't do FP. But OTOH......<long dots> there is already 
an SSE player in the kernel: drivers/md/raid6sse*.c


Jan Engelhardt
-- 

Re: FP in kernelspace

[Andi Kleen]

Jiri Slaby <jirislaby@gmail.com> writes:

> I have a driver written for 2.4 + RT patches with FP support. I want
> it to work in 2.6. How to implement FP? Has anybody developped some
> "protocol" between KS and US yet? If not, could somebody point me, how
> to do it the best -- with low latency.
> The device doesn't generate irqs 

Doesn't sound like something that should be supported for mainline
then.

> So 2 questions are:
> 1) howto FP in kernel

You can use kernel_fpu_begin()/_end() around the FP code in the
kernel, but the real time people will probably hate you for it because
it disables preemption.

> 3) any way to have faster ticks (up to 5000Hz)?

There are plenty of interval timers in various hardware in a typical
system that can be used.  e.g. RTC is a common choice. Drawback is
that you might conflict with other users, especially in user space.
 
But please don't try to submit such a hack.

-Andi

Re: FP in kernelspace

[Steven Rostedt]

On Sun, 2006-07-30 at 13:27 +0159, Jiri Slaby wrote:
> Hello,
> 
> I have a driver written for 2.4 + RT patches with FP support. I want it to work 
> in 2.6. How to implement FP? Has anybody developped some "protocol" between KS 
> and US yet? If not, could somebody point me, how to do it the best -- with low 
> latency.
> The device doesn't generate irqs *), I need to quickly respond to timer call, 
> because interval between two posts of data to the device has to be equal as much 
> as possible (BTW is there any way how to gain up to 5000Hz).
> I've one idea: have a thread with RT priority and wake the app in US waiting in 
> read of character device when timer ticks, post a struct with 2 floats and 
> operation and wait in write for the result. App computes, writes the result, we 
> are woken and can post it to the device. But I'm afraid it would be tooo slow.
> 
> *) I don't know how to persuade it (standard PLX chip with unknown piece of 
> logic behind) to generate, because official driver is closed and _very_ 
> expensive. Old (2.4) driver was implemented with RT thread and timer, where FP 
> is implemented within RT and computed directly in KS.
> 
> So 2 questions are:
> 1) howto FP in kernel
> 2) howto precise timer (may mingo RT patches help?)

Well Ingo's RT patch set has the high resolution timers developed by
Thomas Gleixner, which may help you here.

> 3) any way to have faster ticks (up to 5000Hz)?

Why do you need faster ticks?  The high res timers are done in nano
secs, and the resolution is up to the hardware.

> 
> Any suggestions, please?

--  Steve