Re: splice() performance for TCP socket forwarding

[Eric Dumazet <eric.dumazet@gmail.com>]

On 12/13/2018 06:03 AM, Eric Dumazet wrote:
> 
> 
> On 12/13/2018 05:33 AM, Marek Majkowski wrote:
>> Ok, 4.19 does seem to kinda fix the SO_RCVLOWAT with splice, but I
>> don't fully understand it:
>>
>> fcntl(8, F_SETPIPE_SZ, 1048576)         = 1048576 <0.000033>
>> setsockopt(4, SOL_SOCKET, SO_RCVLOWAT, [131072], 4) = 0 <0.000014>
>> splice(4, NULL, 9, NULL, 1048576, SPLICE_F_MOVE) = 121435 <71.039385>
>> splice(8, NULL, 5, NULL, 121435, SPLICE_F_MOVE) = 121435 <0.000118>
>> splice(4, NULL, 9, NULL, 1048576, SPLICE_F_MOVE) = 11806 <0.000019>
>> splice(8, NULL, 5, NULL, 11806, SPLICE_F_MOVE) = 11806 <0.000018>
>>
> 
> Good point.
> 
> At this moment SO_RCVLOWAT only tries to reduce number of POLLIN events.
> 
> But if your splice() system call is performed while there are already
> available skbs in the receive queue, splice() wont block and deliver
> what is available in the queue.
> 
> I guess that we would need to add some logic in recvmsg() and tcp_splice_read()
> to truly implement SO_RCVLOWAT : block until at least sk->sk_rcvlowat bytes are
> available in receive queue.
> 

You could also work around the problem by inserting a poll() system call before the splice(),
since poll() would only deliver the POLLIN event when sk->sk_rcvlowat bytes are present in the queue.