sendfile() expert advice sought

sendfile() expert advice sought

[Patrick J. LoPresti]

Executive summary:  Can I get the benefits of sendfile() for anonymous pages?

I have an application that generates hundreds of gigabytes of data per
hour.  I want to push that data out over a TCP socket.  (The network
connection will be fast; multiple bonded GigE lines or 10GigE.)

I gather that sendfile() is pretty efficient, so I would like to use
it.  But I do not want to write all of my data to disk first.  So I am
considering an approach like this:

  int fd = shm_open("/foo", O_RDWR|O_TRUNC);
  ftruncate(fd, length);
  void *p = mmap (0, length, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
  // (fill memory block at p with some data)
  sendfile(fd, sock, 0, length);

Questions:

1) Will this work at all?  (Some on-line sources suggest sendfile()
does not work with tmpfs files.  But I think this was fixed at some
point...)

2) Will it provide zero-copy behavior, or does the fact that the pages
are mapped in my process cause sendfile() to copy them?

3) If it is zero-copy, what happens if I overwrite the memory block
after sendfile() returns?  Do I risk corrupting my data?  (In
particular, suppose I have TCP_CORK set on the socket.  Will
sendfile() return before all of the data has actually been sent,
giving me a window to corrupt my data?  If so, how do I know when it
is "safe" to re-use the memory?)

4) If sendfile() is not zero-copy in this example, would I expect a
performance boost anyway, because sendfile() does not need to crawl
page tables or something?

Any responses or references will be appreciated.

Thanks!

 - Pat

P.S.  I know I could also try mmap()'ing "/dev/zero" and using
vmsplice().  Same set of questions, though.

Re: sendfile() expert advice sought

[Robert Hancock]

On 02/16/2010 01:53 PM, Patrick J. LoPresti wrote:
> Executive summary:  Can I get the benefits of sendfile() for anonymous pages?
>
> I have an application that generates hundreds of gigabytes of data per
> hour.  I want to push that data out over a TCP socket.  (The network
> connection will be fast; multiple bonded GigE lines or 10GigE.)
>
> I gather that sendfile() is pretty efficient, so I would like to use
> it.  But I do not want to write all of my data to disk first.  So I am
> considering an approach like this:
>
>    int fd = shm_open("/foo", O_RDWR|O_TRUNC);
>    ftruncate(fd, length);
>    void *p = mmap (0, length, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
>    // (fill memory block at p with some data)
>    sendfile(fd, sock, 0, length);
>
> Questions:
>
> 1) Will this work at all?  (Some on-line sources suggest sendfile()
> does not work with tmpfs files.  But I think this was fixed at some
> point...)
>
> 2) Will it provide zero-copy behavior, or does the fact that the pages
> are mapped in my process cause sendfile() to copy them?
>
> 3) If it is zero-copy, what happens if I overwrite the memory block
> after sendfile() returns?  Do I risk corrupting my data?  (In
> particular, suppose I have TCP_CORK set on the socket.  Will
> sendfile() return before all of the data has actually been sent,
> giving me a window to corrupt my data?  If so, how do I know when it
> is "safe" to re-use the memory?)
>
> 4) If sendfile() is not zero-copy in this example, would I expect a
> performance boost anyway, because sendfile() does not need to crawl
> page tables or something?
>
> Any responses or references will be appreciated.

I can't really answer definitively, but as far as I know you wouldn't 
get any magic performance benefits from playing games with sendfile, 
splice, etc. as long as the pages were mapped into userspace already, 
they only really help when the data being written out is coming from 
another file, pipe, etc. not when your application is generating the 
data internally.

Re: sendfile() expert advice sought

[Bryan Donlan]

On Tue, Feb 16, 2010 at 2:53 PM, Patrick J. LoPresti <lopresti@gmail.com> wrote:
> Executive summary:  Can I get the benefits of sendfile() for anonymous pages?
>
> I have an application that generates hundreds of gigabytes of data per
> hour.  I want to push that data out over a TCP socket.  (The network
> connection will be fast; multiple bonded GigE lines or 10GigE.)
>
> I gather that sendfile() is pretty efficient, so I would like to use
> it.  But I do not want to write all of my data to disk first.  So I am
> considering an approach like this:
>
>  int fd = shm_open("/foo", O_RDWR|O_TRUNC);
>  ftruncate(fd, length);
>  void *p = mmap (0, length, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
>  // (fill memory block at p with some data)
>  sendfile(fd, sock, 0, length);
>
> Questions:
>
> 1) Will this work at all?  (Some on-line sources suggest sendfile()
> does not work with tmpfs files.  But I think this was fixed at some
> point...)

If you add a msync() call in there it should work (it might work
without it, but this is only an implementation detail :).

> 2) Will it provide zero-copy behavior, or does the fact that the pages
> are mapped in my process cause sendfile() to copy them?

sendfile() always copies pages; the performance benefit on regular
files comes from the fact that you don't need to copy _twice_ - once
to userspace from DMA buffers, then once back into the kernel network
buffers. Of course, in this case you only need one copy either way...

>
> 3) If it is zero-copy, what happens if I overwrite the memory block
> after sendfile() returns?  Do I risk corrupting my data?  (In
> particular, suppose I have TCP_CORK set on the socket.  Will
> sendfile() return before all of the data has actually been sent,
> giving me a window to corrupt my data?  If so, how do I know when it
> is "safe" to re-use the memory?)

sendfile() copies the data it needs, so it's fine to re-use the data
immediately.

>
> 4) If sendfile() is not zero-copy in this example, would I expect a
> performance boost anyway, because sendfile() does not need to crawl
> page tables or something?

Doubtful - user-to-kernel copies using write() and friends generally
use the CPU's builtin page translation circuitry anyway, which is
probably faster than any software, in-kernel mechanism. You'll
probably only get a benefit if you're sendfile()ing from a disk file
(and this is likely to be on the same order as from mmap()ing the file
and using write() from the mmap'd buffer).

Re: sendfile() expert advice sought

[Patrick J. LoPresti]

On Wed, Feb 17, 2010 at 6:19 PM, Bryan Donlan <bdonlan@gmail.com> wrote:
>
> sendfile() always copies pages

Really?  So the description in the following article (see "Figure 4") is wrong?

http://www.linuxjournal.com/article/6345?page=0,1

(To be clear, I want to replace the step where the kernel buffers get
populated from disk with a step where I populate them programatically.
 That is what I mean by "sendfile() for anonymous pages".)

According to that article, among others, if your network device
supports scatter/gather and the pages are already resident, then
sendfile() is a true zero-copy operation.  That is, it takes page
cache pages and arranges to DMA them to the network card without
creating any additional copies in system RAM.

My questions make a lot more sense if that is how it works, anyway :-).

 - Pat