Re: [PATCH] NFSD: fix use of setsockopt

[Dean Hildebrand <seattleplus@gmail.com>]

Hi Olga,

This makes sense, if NFSD is going to ignore global Linux TCP settings 
and 'go it alone', then it shouldn't be constrained by them.

At least now we can increase the rcv buffer size by increasing the 
number of NFSDs.  I would still like to pursue my sysctl patch for the 
rcv and snd buffer though since we have seen situations where too many 
NFSDs can increase the randomness of requests to the underlying file 
system, reducing the effectiveness readahead/write gathering.

Dean

Olga Kornievskaia wrote:
> The following patch fixes NFS server's use of setsockopt. For this 
> function to take an effect it first needs be called after socket 
> creation but before sock binding.
>
> This patch also changes the size of the receive sock buffer to be same 
> as the send sock buffer. Both buffers are now a multiple of maxpayload 
> and number of nfsd threads.
>
> This patch fixes the problem that receive window never opens beyond 
> the default TCP receive window size set by the 2nd parameter of the 
> net.ipv4.tcp_rmem sysctl.
>
> Signed-off-by: Olga Kornievskaia <aglo@citi.umich.edu>
Signed-off-by: Olga Kornievskaia <aglo@citi.umich.edu>

------------------------------------------------------------------------

diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index c75bffe..178b397 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1191,7 +1191,7 @@ svc_tcp_recvfrom(struct svc_rqst *rqstp)
 		 */
 		svc_sock_setbufsize(svsk->sk_sock,
 				    (serv->sv_nrthreads+3) * serv->sv_max_mesg,
-				    3 * serv->sv_max_mesg);
+				    (serv->sv_nrthreads+3) * serv->sv_max_mesg);
 
 	clear_bit(SK_DATA, &svsk->sk_flags);
 
@@ -1372,11 +1372,6 @@ svc_tcp_init(struct svc_sock *svsk)
 		 * receive and respond to one request.
 		 * svc_tcp_recvfrom will re-adjust if necessary
 		 */
-		svc_sock_setbufsize(svsk->sk_sock,
-				    3 * svsk->sk_server->sv_max_mesg,
-				    3 * svsk->sk_server->sv_max_mesg);
-
-		set_bit(SK_CHNGBUF, &svsk->sk_flags);

 		set_bit(SK_DATA, &svsk->sk_flags);
 		if (sk->sk_state != TCP_ESTABLISHED)
 			set_bit(SK_CLOSE, &svsk->sk_flags);
@@ -1761,6 +1756,8 @@ static int svc_create_socket(struct svc_serv *serv, int protocol,
 
 	if (type == SOCK_STREAM)
 		sock->sk->sk_reuse = 1;		/* allow address reuse */
+	svc_sock_setbufsize(sock, (serv->sv_nrthreads+3) * serv->sv_max_mesg,
+				(serv->sv_nrthreads+3) * serv->sv_max_mesg);
 	error = kernel_bind(sock, sin, len);
 	if (error < 0)
 		goto bummer;