[PATCH] SVC sockets don't disable Nagle

[PATCH] SVC sockets don't disable Nagle

[Olof Johansson]

I noticed that the Nagle algorithm was disabled just recently on the 
client side, while it still seems to be enabled on the server side. The 
previous patch came from Chuck Lever.

Below patch disables it on the server side as well. For latency reasons, 
this should be the desired behaviour NFS at both client and server.

Diff is against 2.4.21-rc1.

---------------------------

--- linux-2.4.21-rc1/net/sunrpc/svcsock.c.orig
+++ linux-2.4.21-rc1/net/sunrpc/svcsock.c
@@ -932,6 +932,7 @@ static int
  svc_tcp_init(struct svc_sock *svsk)
  {
         struct sock     *sk = svsk->sk_sk;
+       struct tcp_opt  *tp = &(sk->tp_pinfo.af_tcp);

         svsk->sk_recvfrom = svc_tcp_recvfrom;
         svsk->sk_sendto = svc_tcp_sendto;
@@ -948,6 +949,8 @@ svc_tcp_init(struct svc_sock *svsk)
                 svsk->sk_reclen = 0;
                 svsk->sk_tcplen = 0;

+               tp->nonagle = 1;        /* disable Nagle's algorithm */
+
                 /* initialise setting must have enough space to
                  * receive and respond to one request.
                  * svc_tcp_recvfrom will re-adjust if necessary


---------------------------


Thanks,

Olof

-- 
Olof Johansson                                        Office: 4E002/905
pSeries Linux Development                             IBM Systems Group
Email: olof@austin.ibm.com                          Phone: 512-838-9858
All opinions are my own and not those of IBM.



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] SVC sockets don't disable Nagle

[Bogdan Costescu]

On Tue, 29 Apr 2003, Olof Johansson wrote:

> Below patch disables it on the server side as well. For latency reasons, 
> this should be the desired behaviour NFS at both client and server.

I disagree with this general statement. Nagle is there in the TCP protocol 
for a specific reason and enabling or disabling it gives the opportunity 
to better tune the connection.
It seems like disabling Nagle is the bolierplate answer for anyone who 
encounters GigEth or better networking simply because the TCP protocol was 
not designed for such fast networks. But what happens to others that do 
not have GigEth ? What is the impact of this change for a slower/congested 
network ?
If anything, I would like to see this as a per-mount-point and 
per-export-entry option...

--
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu@IWR.Uni-Heidelberg.De



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] SVC sockets don't disable Nagle

[Olof Johansson]

Bogdan Costescu wrote:
> On Tue, 29 Apr 2003, Olof Johansson wrote:
> 
>>Below patch disables it on the server side as well. For latency reasons, 
>>this should be the desired behaviour NFS at both client and server.
> 
> 
> I disagree with this general statement. Nagle is there in the TCP protocol 
> for a specific reason and enabling or disabling it gives the opportunity 
> to better tune the connection.

Yes, Nagle is there to reduce overhead by aggregating data into segments
and piggybacking them with ACKs. Most significant differences will be
when dealing with a steady but slow pace of characters, say text being
typed over a 9600bps serial line.

> It seems like disabling Nagle is the bolierplate answer for anyone who 
> encounters GigEth or better networking simply because the TCP protocol was 
> not designed for such fast networks. But what happens to others that do 
> not have GigEth ? What is the impact of this change for a slower/congested 
> network ?

There is a chance that small requests and replies (~100 bytes) will not
be aggregated into the same segments but instead sent out separately. In
essence, this will result in some additional network overhead due to
headers, but the response times will be perceived as faster even for 
slower networks.

Since NFS traffic is bursty, it is unlikely that any individual request
or reply will be split up between two segments when it didn't need to.

> If anything, I would like to see this as a per-mount-point and 
> per-export-entry option...

I think it would just clutter the documentation and list of options,
since I don't forsee any practical scenarios in which anyone would have
a positive performance impact from having Nagle enabled.


On a side note: As far as I know, most (all?) other NFS implementations 
out there already disable Nagle.


-Olof

-- 
Olof Johansson                                        Office: 4E002/905
pSeries Linux Development                             IBM Systems Group
Email: olof@austin.ibm.com                          Phone: 512-838-9858




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] SVC sockets don't disable Nagle

[Bogdan Costescu]

On Tue, 29 Apr 2003, Olof Johansson wrote:

> There is a chance that small requests and replies (~100 bytes) will not
> be aggregated into the same segments but instead sent out separately.

Although I haven't looked at the code recently, I think that what you said 
is true when this is the only request from the client. But what happens 
when there are several requests or when GETATTR calls happen at the same 
time with some data transfer (which is normally the case for a multiuser 
NFS client) ?

> In essence, this will result in some additional network overhead due to
> headers, but the response times will be perceived as faster even for
> slower networks.

True, until the network becomes too slow, then you see "server not 
responding" messages...

> I think it would just clutter the documentation and list of options,

Yes, but Samba has this option and many others and still lots of people 
are using it :-)

> since I don't forsee any practical scenarios in which anyone would have
> a positive performance impact from having Nagle enabled.

Well, my previous message included this question: "What is the impact of 
this change for a slower/congested network ?" So I'll rephrase it: do you 
have some data to show that disabling Nagle does not have a negative 
impact ?

> On a side note: As far as I know, most (all?) other NFS implementations 
> out there already disable Nagle.

I didn't say that the other NFS implementations are better than the Linux 
implementation :-)

-- 
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu@IWR.Uni-Heidelberg.De




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] SVC sockets don't disable Nagle

[Trond Myklebust]

>>>>> " " == Bogdan Costescu <bogdan.costescu@iwr.uni-heidelberg.de> writes:

    >> In essence, this will result in some additional network
    >> overhead due to headers, but the response times will be
    >> perceived as faster even for slower networks.

     > True, until the network becomes too slow, then you see "server
     > not responding" messages...

Anybody trying to run multi-user NFS over a 9.6Kbaud link needs their
head examined.
For *real use* situations, the total number of on-the-wire NFS
requests is currently limited to 16. In addition, TCP timeouts are
supposed to be > 1 minute. The above scenario simply will not happen.

For any reasonable setup over a slow network, the aggregation will
happen naturally anyway since the NFS layer both can and will fill the
buffers as soon as they start to empty (provided we have pending
datagrams).

For fast networks, we should at least ensure that single datagrams are
aggregated properly. The way to do that is to use MSG_MORE...

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] SVC sockets don't disable Nagle

[Olof Johansson]

Bogdan Costescu wrote:
> On Tue, 29 Apr 2003, Olof Johansson wrote:
> 
> 
>>There is a chance that small requests and replies (~100 bytes) will not
>>be aggregated into the same segments but instead sent out separately.
> 
> Although I haven't looked at the code recently, I think that what you said 
> is true when this is the only request from the client. But what happens 
> when there are several requests or when GETATTR calls happen at the same 
> time with some data transfer (which is normally the case for a multiuser 
> NFS client) ?

This is exactly what I meant. There is a chance that the GETATTR (or 
other small request and/or reply) will be sent out in an individual 
segment instead of being aggregated. This will give some additional 
network overhead, but give latency improvement. Actually, for a busy NFS 
connection the small requests/replies will still be aggregated without 
Nagle. At least that's what I've been seeing here.

>>In essence, this will result in some additional network overhead due to
>>headers, but the response times will be perceived as faster even for
>>slower networks.
> 
> True, until the network becomes too slow, then you see "server not 
> responding" messages...

I can't see how disabling Nagle would lead to RPC timeouts:

Lower network overhead during congestion is taken care of automatically: 
As the network throughput decreases, TCP segments will be sent out at a 
lower pace and/or smaller size (due to window sizes going down). There 
will be more time to wait for more data to fill the segment with, even 
without having Nagle enabled.

>>I think it would just clutter the documentation and list of options,
> 
> 
> Yes, but Samba has this option and many others and still lots of people 
> are using it :-)

I didn't say that Samba is better than NFS :-)

Seriously: Samba has a configuration file, while NFS needs to stuff all 
the options into a line or two of export and/or mount options. Option 
bloat is not as big a problem when you have a large configuration file.

>>since I don't forsee any practical scenarios in which anyone would have
>>a positive performance impact from having Nagle enabled.
> 
> Well, my previous message included this question: "What is the impact of 
> this change for a slower/congested network ?" So I'll rephrase it: do you 
> have some data to show that disabling Nagle does not have a negative 
> impact ?

No, this would be to prove a negative which is always hard. But the same 
reasoning as above regarding congested networks apply. Actually, it 
applies to any scenario where the machine can process requests faster 
than the network can send the replies.

>>On a side note: As far as I know, most (all?) other NFS implementations 
>>out there already disable Nagle.
> 
> I didn't say that the other NFS implementations are better than the Linux 
> implementation :-)

I'm of the humble opinion that there is still a bit of room for 
improvement in the Linux implementation, but it's continuosly getting 
better. :-)


-Olof


-- 
Olof Johansson                                        Office: 4E002/905
pSeries Linux Development                             IBM Systems Group
Email: olof@austin.ibm.com                          Phone: 512-838-9858
All opinions are my own and not those of IBM



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs