Re: using splice/vmsplice to improve file receive performance

[Jens Axboe <jens.axboe@oracle.com>]

On Thu, Dec 21 2006, saeed bishara wrote:
> Hi,
> I'm trying to use the splice/vmsplice system calls to improve the
> samba server write throughput, but before touching the smbd, I started
> to improve the ttcp tool since it simple and has the same flow. I'm
> expecting to avoid the "copy_from_user" path when using those
> syscalls.
> so far, I couldn't make any improvement, actually the throughput get
> worst. the new receive flow looks like this (code also attached):
> 1. read tcp packet (64 pages) to page aligned buffer.
> 2. vmsplice the buffer to pipe with SPLICE_F_MOVE.
> 3. splice the pipe to the file, also with SPLICE_F_MOVE.
> 
> the strace shows that the splice takes a lot of time. also when
> profiling the kernel, I found that the memcpy() called to often !!

(didn't see this until now, axboe@suse.de doesn't work anymore)

I'm assuming that you mean you vmsplice with SPLICE_F_GIFT, to hand
ownership of the pages to the kernel (in which case SPLICE_F_MOVE will
work, otherwise you get a copy)? If not, that'll surely cost you a data
copy.

This sounds remarkably like a recent thread on lkml, you may want to
read up on that. Basically using splice for network receive is a bit of
a work-around now, since you do need the one copy and then vmsplice that
into a pipe. To realize the full potential of splice, we first need
socket receive support so you can skip that step (splice from socket to
pipe, splice pipe to file).

There was no test code attached, btw.

-- 
Jens Axboe