Re: sunrpc: socket buffer size tuneable

[Ben Myers <bpm@sgi.com>]

Hey,
	
On Fri, Jan 25, 2013 at 04:57:12PM -0500, J. Bruce Fields wrote:
> On Fri, Jan 25, 2013 at 09:45:12PM +0000, Myklebust, Trond wrote:
> > > -----Original Message----- From: J. Bruce Fields
> > > [mailto:bfields@fieldses.org] Sent: Friday, January 25, 2013 4:35 PM
> > > To: Myklebust, Trond Cc: Ben Myers; Olga Kornievskaia;
> > > linux-nfs@vger.kernel.org; Jim Rees Subject: Re: sunrpc: socket
> > > buffer size tuneable
> > > 
> > > On Fri, Jan 25, 2013 at 09:29:09PM +0000, Myklebust, Trond wrote:
> > > > > -----Original Message----- From: J. Bruce Fields
> > > > > [mailto:bfields@fieldses.org] Sent: Friday, January 25, 2013
> > > > > 4:21 PM To: Myklebust, Trond Cc: Ben Myers; Olga Kornievskaia;
> > > > > linux-nfs@vger.kernel.org; Jim Rees Subject: Re: sunrpc: socket
> > > > > buffer size tuneable
> > > > >
> > > > > On Fri, Jan 25, 2013 at 09:12:55PM +0000, Myklebust, Trond
> > > > > wrote:
> > > >
> > > > > > Why is it not sufficient to clamp the TCP values of 'snd' and
> > > > > > 'rcv' using
> > > > > sysctl_tcp_wmem/sysctl_tcp_rmem?
> > > > > > ...and clamp the UDP values using
> > > > > sysctl_[wr]mem_min/sysctl_[wr]mem_max?.
> > > > >
> > > > > Yeah, I was just looking at that--so, Ben, something like:
> > > > >
> > > > > 	echo "1048576 1048576 4194304"
> > > > > 	>/proc/sys/net/ipv4/tcp_wmem
> > > > >
> > > > > But I'm unclear on some of the details: do we need to set the
> > > > > minimum or only the default?  And does it need any more
> > > > > allowance for protocol overhead?
> > > >
> > > > I meant adding a check either to svc_sock_setbufsize or to the 2
> > > > call-sites
> > > that enforces the above limits.
> > > 
> > > I lost you.
> > > 
> > > It's not svc_sock_setbufsize that's setting too-small values, if
> > > that's what you mean.
> > > 
> > 
> > I understood that the problem was svc_udp_recvfrom() and
> > svc_setup_socket() were using negative values in the calls to
> > svc_sock_setbufsize().  Looking again at svc_setup_socket(), I don't
> > see how that could do so, but svc_udp_recvfrom() definitely has
> > potential to cause damage.
> 
> Right, the changelog was confusing, the problem they're actually hitting
> is with tcp.  Looks like tcp autotuning is decreasing the send buffer
> below the size we requested in svc_sock_setbufsize().

echo "1048576 1048576 4194304" > /proc/sys/net/ipv4/tcp_wmem 

Seems to have been effective.  I'll be toasting to you gents tonight.

I think it would be good if the server enforced a setting that is large enough.

Thanks,
	Ben