How to avoid data copies in a driver ?

How to avoid data copies in a driver ?

[Francis Moreau]

Hello,

I'd like to optimize my driver, which receives data through a fifo and gives
them to a user space application. In turns this application moves this data
into a file.

To avoid several useless copies, I'd like the application to pass to the driver
a  file descriptor (?) to  the driver and then the driver can directly move the
received data to that file.

Could anybody give me some example of such scheme ?

Thanks,
-- 
Francis

Re: How to avoid data copies in a driver ?

[Lennart Sorensen]

On Wed, May 14, 2008 at 09:54:03PM +0200, Francis Moreau wrote:
> Hello,
> 
> I'd like to optimize my driver, which receives data through a fifo and gives
> them to a user space application. In turns this application moves this data
> into a file.
> 
> To avoid several useless copies, I'd like the application to pass to the driver
> a  file descriptor (?) to  the driver and then the driver can directly move the
> received data to that file.
> 
> Could anybody give me some example of such scheme ?

If the application memory mapped the file, would it be able to simply
pass a pointer to that mapped file as part of the call to the driver and
the driver would place the data directly at the requested location which
would then be directly to the file?

-- 
Len Sorensen

Re: How to avoid data copies in a driver ?

[Francis Moreau]

On Wed, May 14, 2008 at 10:02 PM, Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:
> On Wed, May 14, 2008 at 09:54:03PM +0200, Francis Moreau wrote:
>> Hello,
>>
>> I'd like to optimize my driver, which receives data through a fifo and gives
>> them to a user space application. In turns this application moves this data
>> into a file.
>>
>> To avoid several useless copies, I'd like the application to pass to the driver
>> a  file descriptor (?) to  the driver and then the driver can directly move the
>> received data to that file.
>>
>> Could anybody give me some example of such scheme ?
>
> If the application memory mapped the file, would it be able to simply
> pass a pointer to that mapped file as part of the call to the driver and
> the driver would place the data directly at the requested location which
> would then be directly to the file?
>

So I would need to map this pointer into the kernel space, then fill it, take
care of cache coherency, unmap the kernel pointer.

Do you have any example of that in the kernel tree ?

BTW, data are received in interrupt context. Is it safe to put them in mapped
memory  (can I have page fault ?) in this context ?

Thanks
-- 
Francis

Re: How to avoid data copies in a driver ?

[Lennart Sorensen]

On Thu, May 15, 2008 at 09:44:37AM +0200, Francis Moreau wrote:
> So I would need to map this pointer into the kernel space, then fill it, take
> care of cache coherency, unmap the kernel pointer.
> 
> Do you have any example of that in the kernel tree ?

I was asking if that would work.  Does the to_user and from_user work on
a pointer from user space if that pointer points at a memory mapped
file?

> BTW, data are received in interrupt context. Is it safe to put them in mapped
> memory  (can I have page fault ?) in this context ?

Oh I thought you just wanted user space to be able to ask for a chunk of
data.  I wouldn't want to write to a file in an interrupt handler.
Compared to the disk I/O a memory copy sounds rather insignificant to
the overall process.

-- 
Len Sorensen

Re: How to avoid data copies in a driver ?

[Francis Moreau]

On Thu, May 15, 2008 at 3:59 PM, Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:
> On Thu, May 15, 2008 at 09:44:37AM +0200, Francis Moreau wrote:
>> So I would need to map this pointer into the kernel space, then fill it, take
>> care of cache coherency, unmap the kernel pointer.
>>
>> Do you have any example of that in the kernel tree ?
>
> I was asking if that would work. Does the to_user and from_user work on
> a pointer from user space if that pointer points at a memory mapped
> file?

I would say yes but I'm not really confident.

-- 
Francis

Re: How to avoid data copies in a driver ?

[linux-os (Dick Johnson)]

On Wed, 14 May 2008, Francis Moreau wrote:

> Hello,
>
> I'd like to optimize my driver, which receives data through a fifo and gives
> them to a user space application. In turns this application moves this data
> into a file.
>
> To avoid several useless copies, I'd like the application to pass to
> the driver
> a file descriptor (?) to  the driver and then the driver can directly
> move the
> received data to that file.
>
> Could anybody give me some example of such scheme ?
>
> Thanks,
> -- 
> Francis
> --

You memory-map the data. Impliment mmap() in your driver.
You can also impliment poll()  { select() } so your
application knows when new data are available.

You cannot use a user-mode file-descriptor in the kernel.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips).
My book : http://www.AbominableFirebug.com/
_


****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: How to avoid data copies in a driver ?

[Francis Moreau]

Hello,

On Wed, May 14, 2008 at 11:23 PM, linux-os (Dick Johnson)
<linux-os@analogic.com> wrote:
> You memory-map the data. Impliment mmap() in your driver.
> You can also impliment poll()  { select() } so your
> application knows when new data are available.
>
> You cannot use a user-mode file-descriptor in the kernel.
>

Why not ?

I'm suprised because what I need doens't seem so uncommon, usually
devices send or
receive data to/from files. So a helper (system call ?) to achieve
that other than the basic
read/write seems needed, no ?

-- 
Francis

Re: How to avoid data copies in a driver ?

[Arnd Hannemann]

Francis Moreau wrote:
> Hello,
> 
> On Wed, May 14, 2008 at 11:23 PM, linux-os (Dick Johnson)
> <linux-os@analogic.com> wrote:
>> You memory-map the data. Impliment mmap() in your driver.
>> You can also impliment poll()  { select() } so your
>> application knows when new data are available.
>>
>> You cannot use a user-mode file-descriptor in the kernel.
>>
> 
> Why not ?

http://kernelnewbies.org/FAQ/WhyWritingFilesFromKernelIsBad

> 
> I'm suprised because what I need doens't seem so uncommon, usually
> devices send or
> receive data to/from files. So a helper (system call ?) to achieve
> that other than the basic
> read/write seems needed, no ?

Usually devices send or receive just data, and they shouldn't care about
file format, filesystems, permissions and all this stuff...

Regards,
Arnd

Re: How to avoid data copies in a driver ?

[Francis Moreau]

On Thu, May 15, 2008 at 1:50 PM, Arnd Hannemann <arnd@arndnet.de> wrote:
> Francis Moreau wrote:
>> Hello,
>>
>> On Wed, May 14, 2008 at 11:23 PM, linux-os (Dick Johnson)
>> <linux-os@analogic.com> wrote:
>>> You memory-map the data. Impliment mmap() in your driver.
>>> You can also impliment poll()  { select() } so your
>>> application knows when new data are available.
>>>
>>> You cannot use a user-mode file-descriptor in the kernel.
>>>
>>
>> Why not ?
>
> http://kernelnewbies.org/FAQ/WhyWritingFilesFromKernelIsBad
>

ok, relayfs might be what I need, thanks.

>>
>> I'm suprised because what I need doens't seem so uncommon, usually
>> devices send or
>> receive data to/from files. So a helper (system call ?) to achieve
>> that other than the basic
>> read/write seems needed, no ?
>
> Usually devices send or receive just data, and they shouldn't care about
> file format, filesystems, permissions and all this stuff...
>

The fact is that the data usually ends into a file.

thanks
-- 
Francis

Re: How to avoid data copies in a driver ?

[linux-os (Dick Johnson)]

On Thu, 15 May 2008, Francis Moreau wrote:

> Hello,
>
> On Wed, May 14, 2008 at 11:23 PM, linux-os (Dick Johnson)
> <linux-os@analogic.com> wrote:
>> You memory-map the data. Impliment mmap() in your driver.
>> You can also impliment poll()  { select() } so your
>> application knows when new data are available.
>>
>> You cannot use a user-mode file-descriptor in the kernel.
>>
>
> Why not ?
>
> I'm suprised because what I need doens't seem so uncommon, usually
> devices send or
> receive data to/from files. So a helper (system call ?) to achieve
> that other than the basic
> read/write seems needed, no ?
>
> -- 
> Francis
>

The kernel is designed to perform services on behalf of
a caller. The kernel itself doesn't have a process context.
Therefore, a file-descriptor, which requires a process context
to mean anything, is not useful within the kernel unless the
kernel either uses your process context (which happens
efficiently when YOU call the kernel) or it steals one
from somebody else, which means their context gets trashed.
Note that every process is created with at file descriptors
0, 1, and 2. It's only the process context that keeps them
separate.

That said, you can create a kernel-mode task in your driver.
That task would have a context. However it wouldn't be YOUR
context, so you would need to signal it when data was
available, adding overhead and communicate with it from.
your user-space program context. It would share the same
data-space as your driver so it wouldn't need to copy.
However, such data would get copied into kernel buffers
by the I/O code so it's a waste anyway. You save one
copy, which you would do with memory-mapping, plus you
have the added communications overhead.

Using memory mapping as previously advised, lets you DMA
data directly to a user if your hardware does DMA. It also
would allow you to save one copy, even if you don't have
DMA capabilities. As far as copies are concerned, there are
many copies before your data actually gets to a disk platter.
Those copies (usually) occur when the kernel doesn't have
anything else to do.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips).
My book : http://www.AbominableFirebug.com/
_


****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: How to avoid data copies in a driver ?

[Jeremy Fitzhardinge]

Francis Moreau wrote:
> I'm suprised because what I need doens't seem so uncommon, usually
> devices send or
> receive data to/from files. So a helper (system call ?) to achieve
> that other than the basic
> read/write seems needed, no ?
>   

It's fairly rare to have an application which requires moving data to 
file with absolutely no processing; normally there's at least a bit of 
massaging/parsing/etc.  If that's really what you want to do, maybe you 
can do it with splice?  I haven't looked at it at all, but the intention 
is that you can splice file descriptors together, so you can splice your 
device fd to a file fd and have it all just work...

Alternatively you could read() from your device into a mmaped file.  
That's a single copy from device to file, which is about the best you 
can do without going to heroic lengths.

Also, it really depends on your application.  Is it a high-bandwidth 
thing in which the copy is a huge cost?  Or do you want to eliminate the 
copies because it seems like a nice thing to do?


    J

Re: How to avoid data copies in a driver ?

[Francis Moreau]

Hello,

On Fri, May 16, 2008 at 10:10 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Francis Moreau wrote:
>>
>> I'm suprised because what I need doens't seem so uncommon, usually
>> devices send or
>> receive data to/from files. So a helper (system call ?) to achieve
>> that other than the basic
>> read/write seems needed, no ?
>>
>
> It's fairly rare to have an application which requires moving data to file
> with absolutely no processing; normally there's at least a bit of
> massaging/parsing/etc.

Well, every cases where a client <-> server exchange files. I wouldn't call
that a rare case...

> If that's really what you want to do, maybe you can
> do it with splice?  I haven't looked at it at all, but the intention is that
> you can splice file descriptors together, so you can splice your device fd
> to a file fd and have it all just work...
>

yes but the kernel I'm working on (2.6.16) doens't have splice support.

But relay may sound a good idea, and the version I use has sendfile support.

The drawback of relay: the user application can receive a file from the kernel
but the app can't send a file to the kernel.

But I can be wrong since I haven't look at relay closely yet.

> Alternatively you could read() from your device into a mmaped file.  That's
> a single copy from device to file, which is about the best you can do
> without going to heroic lengths.
>

Don't know for now.

I think using sendfile with relay may be a good answer without me becoming
a hero ;)

> Also, it really depends on your application.  Is it a high-bandwidth thing
> in which the copy is a huge cost?  Or do you want to eliminate the copies
> because it seems like a nice thing to do?

No it has a real cost. The application receives files which are usually larger
than 1Go.

Thanks
-- 
Francis

Re: How to avoid data copies in a driver ?

[Pavel Machek]

On Wed 2008-05-14 21:54:03, Francis Moreau wrote:
> Hello,
> 
> I'd like to optimize my driver, which receives data through a fifo and gives
> them to a user space application. In turns this application moves this data
> into a file.
> 
> To avoid several useless copies, I'd like the application to pass to the driver
> a  file descriptor (?) to  the driver and then the driver can directly move the
> received data to that file.
> 
> Could anybody give me some example of such scheme ?

man splice?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html