Is splice() useful without page stealing?

Is splice() useful without page stealing?

[Patrick J. LoPresti]

I just spent a couple of hours tracking down documentation and
discussion on splice().  I apologize in advance if I missed something.

As near as I can tell, SPLICE_F_MOVE (aka. "page stealing") was
removed in 2.6.21
(http://lkml.indiana.edu/hypermail/linux/kernel/0703.1/2592.html) and
never reinstated.

My question is, doesn't this make splice() nearly useless from a
performance perspective?

For example, consider implementing a simple file copy.  You might
argue that even without SPLICE_F_MOVE, splice() can be used to avoid
one copy of the data:  The standard/portable "read() into a user
buffer and write() from that buffer" is two copies, whereas splice()
to and from a pipe would be one copy.

But that argument is bogus, because modern processors have sizable
caches.  If you simply write the portable code to read() into a small
buffer (16K or so) and write() from that buffer, and you use the same
buffer over and over, the data in the buffer should get overwritten in
the L1 cache before it ever gets flushed to the RAM behind it.  In
effect, this simple loop would move data into the L1 cache (on read())
and then back into the page cache (on write()), for a net of ONE
round-trip to physical memory.

Similarly for copying data from a socket to a file in an attempt to
implement "recvfile()".

Now, if splice() actually let you do things like flip pages from the
network stack into the page cache with zero copies, then I could see
the point.  As it is, I am curious to know:

1) Does anybody have any actual benchmarks showing performance
advantages for splice() over read()/write()?   If so, can I obtain the
benchmark code?

2) Are there any plans to reinstate SPLICE_F_MOVE or something equivalent?

Thank you.

 - Pat

Re: Is splice() useful without page stealing?

[Willy Tarreau]

On Thu, Dec 17, 2009 at 06:47:19PM -0800, Patrick J. LoPresti wrote:
(...)
> Now, if splice() actually let you do things like flip pages from the
> network stack into the page cache with zero copies, then I could see
> the point.  As it is, I am curious to know:
> 
> 1) Does anybody have any actual benchmarks showing performance
> advantages for splice() over read()/write()?   If so, can I obtain the
> benchmark code?

I did this for TCP->TCP sockets, and the gain is huge at high
bit rates (10 Gbps). Basically, 10 Gbps at 20% CPU with splice,
100% CPU with recv/send. You can find the test results here :

   http://haproxy.1wt.eu/10g.html

I'm not using splice for disk-to-net nor net-to-disk transfers though.

Willy