[PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Chuck Lever]

TCP support in the Linux NFS server is stable enough that we can leave it
on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to
"Y" anyway.

A run-time switch might be more appropriate if people feel they would like
to disable NFSD's TCP support.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---

 fs/Kconfig       |   10 ----------
 fs/nfsd/nfssvc.c |    2 --
 2 files changed, 0 insertions(+), 12 deletions(-)

diff --git a/fs/Kconfig b/fs/Kconfig
index e6d4034..5ccff9a 100644
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -1686,7 +1686,6 @@ config NFSD
 	select EXPORTFS
 	select NFSD_V2_ACL if NFSD_V3_ACL
 	select NFS_ACL_SUPPORT if NFSD_V2_ACL
-	select NFSD_TCP if NFSD_V4
 	select CRYPTO_MD5 if NFSD_V4
 	select CRYPTO if NFSD_V4
 	select FS_POSIX_ACL if NFSD_V4
@@ -1763,15 +1762,6 @@ config NFSD_V4
 
 	  If unsure, say N.
 
-config NFSD_TCP
-	bool "Provide NFS server over TCP support"
-	depends on NFSD
-	default y
-	help
-	  If you want your NFS server to support TCP connections, say Y here.
-	  TCP connections usually perform better than the default UDP when
-	  the network is lossy or congested.  If unsure, say Y.
-
 config LOCKD
 	tristate
 
diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 9647b0f..941041f 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -244,7 +244,6 @@ static int nfsd_init_socks(int port)
 	if (error < 0)
 		return error;
 
-#ifdef CONFIG_NFSD_TCP
 	error = lockd_up(IPPROTO_TCP);
 	if (error >= 0) {
 		error = svc_create_xprt(nfsd_serv, "tcp", port,
@@ -254,7 +253,6 @@ static int nfsd_init_socks(int port)
 	}
 	if (error < 0)
 		return error;
-#endif
 	return 0;
 }
 

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Greg Banks]

Chuck Lever wrote:
> TCP support in the Linux NFS server is stable enough that we can leave it
> on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to
> "Y" anyway.
>
> A run-time switch might be more appropriate if people feel they would like
> to disable NFSD's TCP support.
>
>   
Looks good.

Actually, I'd be inclined to go one step further and set UDP support
off by default.

-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Trond Myklebust]

On Tue, 2008-02-05 at 11:19 +1100, Greg Banks wrote:
> Chuck Lever wrote:
> > TCP support in the Linux NFS server is stable enough that we can leave it
> > on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to
> > "Y" anyway.
> >
> > A run-time switch might be more appropriate if people feel they would like
> > to disable NFSD's TCP support.
> >
> >   
> Looks good.
> 
> Actually, I'd be inclined to go one step further and set UDP support
> off by default.

That will break older clients.

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Greg Banks]

Trond Myklebust wrote:
> On Tue, 2008-02-05 at 11:19 +1100, Greg Banks wrote:
>   
>> Chuck Lever wrote:
>>     
>>> TCP support in the Linux NFS server is stable enough that we can leave it
>>> on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to
>>> "Y" anyway.
>>>
>>> A run-time switch might be more appropriate if people feel they would like
>>> to disable NFSD's TCP support.
>>>
>>>   
>>>       
>> Looks good.
>>
>> Actually, I'd be inclined to go one step further and set UDP support
>> off by default.
>>     
>
> That will break older clients.
>
>   
Hence the default, rather than removing the code entirely.

-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Chuck Lever]

On Feb 4, 2008, at 7:29 PM, Greg Banks wrote:
> Trond Myklebust wrote:
>> On Tue, 2008-02-05 at 11:19 +1100, Greg Banks wrote:
>>
>>> Chuck Lever wrote:
>>>
>>>> TCP support in the Linux NFS server is stable enough that we can  
>>>> leave it
>>>> on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and  
>>>> defaults to
>>>> "Y" anyway.
>>>>
>>>> A run-time switch might be more appropriate if people feel they  
>>>> would like
>>>> to disable NFSD's TCP support.
>>>>
>>>>
>>>>
>>> Looks good.
>>>
>>> Actually, I'd be inclined to go one step further and set UDP support
>>> off by default.
>>>
>>
>> That will break older clients.
>>
>>
> Hence the default, rather than removing the code entirely.


What might make sense is to remove NFSD_TCP, but add NFSD_UDP,  
defaulting to Y.

Then in a year or two we can change the default to N.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Greg Banks]

Chuck Lever wrote:
> On Feb 4, 2008, at 7:29 PM, Greg Banks wrote:
>> Trond Myklebust wrote:
>>> On Tue, 2008-02-05 at 11:19 +1100, Greg Banks wrote:
>>>
>>>> Chuck Lever wrote:
>>>>
>>>>> TCP support in the Linux NFS server is stable enough that we can
>>>>> leave it
>>>>> on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and
>>>>> defaults to
>>>>> "Y" anyway.
>>>>>
>>>>> A run-time switch might be more appropriate if people feel they
>>>>> would like
>>>>> to disable NFSD's TCP support.
>>>>>
>>>>>
>>>>>
>>>> Looks good.
>>>>
>>>> Actually, I'd be inclined to go one step further and set UDP support
>>>> off by default.
>>>>
>>>
>>> That will break older clients.
>>>
>>>
>> Hence the default, rather than removing the code entirely.
>
>
> What might make sense is to remove NFSD_TCP, but add NFSD_UDP,
> defaulting to Y.
>
> Then in a year or two we can change the default to N.
>
Fine by me.


-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Neil Brown]

On Tuesday February 5, gnb-cP1dWloDopni96+mSzHFpQC/G2K4zDHf@public.gmane.org wrote:
> >
> > What might make sense is to remove NFSD_TCP, but add NFSD_UDP,
> > defaulting to Y.
> >
> > Then in a year or two we can change the default to N.
> >
> Fine by me.

I, on the other hand, think we should leave support for UDP and TCP in
the kernel indefinitely and rely on the 'rpc.nfsd' tool to choose
what to listen to.


NeilBrown

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Frank van Maarseveen]

On Tue, Feb 05, 2008 at 04:49:39PM +1100, Greg Banks wrote:
> Chuck Lever wrote:
> > On Feb 4, 2008, at 7:29 PM, Greg Banks wrote:
> >> Trond Myklebust wrote:
> >>> On Tue, 2008-02-05 at 11:19 +1100, Greg Banks wrote:
> >>>
> >>>> Chuck Lever wrote:
> >>>>
> >>>>> TCP support in the Linux NFS server is stable enough that we can
> >>>>> leave it
> >>>>> on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and
> >>>>> defaults to
> >>>>> "Y" anyway.
> >>>>>
> >>>>> A run-time switch might be more appropriate if people feel they
> >>>>> would like
> >>>>> to disable NFSD's TCP support.
> >>>>>
> >>>>>
> >>>>>
> >>>> Looks good.
> >>>>
> >>>> Actually, I'd be inclined to go one step further and set UDP support
> >>>> off by default.
> >>>>
> >>>
> >>> That will break older clients.
> >>>
> >>>
> >> Hence the default, rather than removing the code entirely.
> >
> >
> > What might make sense is to remove NFSD_TCP, but add NFSD_UDP,
> > defaulting to Y.
> >
> > Then in a year or two we can change the default to N.
> >
> Fine by me.

Last time I checked (around 2.6.22) writing large files on NFSv3 over
UDP was 20% faster compared to TCP (Gb LAN with one switch connecting
all machines).

TCP and its timeout/retransmission behavior isn't always the best choice.

-- 
Frank

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Trond Myklebust]

On Tue, 2008-02-05 at 16:50 +0100, Frank van Maarseveen wrote:
> Last time I checked (around 2.6.22) writing large files on NFSv3 over
> UDP was 20% faster compared to TCP (Gb LAN with one switch connecting
> all machines).
> 
> TCP and its timeout/retransmission behavior isn't always the best choice.

If your environment has only 1 client working against a dedicated NFS
server on a clean network, then that may indeed be the case, but as soon
as you have more than 1 client, TCP almost always ends up outperforming
UDP.

Cheers
  Trond

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Frank van Maarseveen]

On Tue, Feb 05, 2008 at 12:50:39PM -0500, Trond Myklebust wrote:
> 
> On Tue, 2008-02-05 at 16:50 +0100, Frank van Maarseveen wrote:
> > Last time I checked (around 2.6.22) writing large files on NFSv3 over
> > UDP was 20% faster compared to TCP (Gb LAN with one switch connecting
> > all machines).
> > 
> > TCP and its timeout/retransmission behavior isn't always the best choice.
> 
> If your environment has only 1 client working against a dedicated NFS
> server on a clean network, then that may indeed be the case, but as soon
> as you have more than 1 client, TCP almost always ends up outperforming
> UDP.

There are multiple clients (12 per physical server on average) but the
clients generally don't write (or read) at the same time. Anyway, I'll
keep this in mind.

-- 
Frank

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[Greg Banks]

Frank van Maarseveen wrote:
> On Tue, Feb 05, 2008 at 04:49:39PM +1100, Greg Banks wrote:
>   
>> Chuck Lever wrote:
>>     
>>> On Feb 4, 2008, at 7:29 PM, Greg Banks wrote:
>>>       
>>>> Trond Myklebust wrote:
>>>>         
>>>>> On Tue, 2008-02-05 at 11:19 +1100, Greg Banks wrote:
>>>>>
>>>>>           
>>>>>> Chuck Lever wrote:
>>>>>>
>>>>>>             
>>>>>>> TCP support in the Linux NFS server is stable enough that we can
>>>>>>> leave it
>>>>>>> on always.  CONFIG_NFSD_TCP adds about 10 lines of code, and
>>>>>>> defaults to
>>>>>>> "Y" anyway.
>>>>>>>
>>>>>>> A run-time switch might be more appropriate if people feel they
>>>>>>> would like
>>>>>>> to disable NFSD's TCP support.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> Looks good.
>>>>>>
>>>>>> Actually, I'd be inclined to go one step further and set UDP support
>>>>>> off by default.
>>>>>>
>>>>>>             
>>>>> That will break older clients.
>>>>>
>>>>>
>>>>>           
>>>> Hence the default, rather than removing the code entirely.
>>>>         
>>> What might make sense is to remove NFSD_TCP, but add NFSD_UDP,
>>> defaulting to Y.
>>>
>>> Then in a year or two we can change the default to N.
>>>
>>>       
>> Fine by me.
>>     
>
> Last time I checked (around 2.6.22) writing large files on NFSv3 over
> UDP was 20% faster compared to TCP (Gb LAN with one switch connecting
> all machines).
>   
Did all of your file arrive at the server, and in the same order it left
the client?  NFS on UDP relies on IP fragmentation, which is known to
introduce silent data corruption at high data rates (google for "IPID aliasing").

Also, last time I checked, UDP support in the server uses a single socket
for all traffic, and processes need to serialise on the svc_sock lock to send,
so aggregate UDP throughput is strictly limited compared to TCP.  As in, 145 MB/s
for UDP compared to filling 12 1gige pipes for TCP.  I have a patch to fix this,
but given the inherent data corruption issues of UDP I haven't bothered posting
the most recent version.



> TCP and its timeout/retransmission behavior isn't always the best choice.
>
>   
The timeout & retrans that sunrpc implements on top of UDP is arguably worse,
especially if you use the "soft" mount option.

-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.

Re: [PATCH 4/5] NFSD: Remove NFSD_TCP kernel build option

[J. Bruce Fields]

On Wed, Feb 06, 2008 at 10:05:39AM +1100, Greg Banks wrote:
> Frank van Maarseveen wrote:
> > Last time I checked (around 2.6.22) writing large files on NFSv3 over
> > UDP was 20% faster compared to TCP (Gb LAN with one switch connecting
> > all machines).
> >   
> Did all of your file arrive at the server, and in the same order it left
> the client?  NFS on UDP relies on IP fragmentation, which is known to
> introduce silent data corruption at high data rates (google for "IPID aliasing").

The right query appears to be "IPID aliasing NFS", which (at least for
me) gets you a nice explanation from Olaf's 2006 OLS paper as the first
hit....

--b.

> Also, last time I checked, UDP support in the server uses a single socket
> for all traffic, and processes need to serialise on the svc_sock lock to send,
> so aggregate UDP throughput is strictly limited compared to TCP.  As in, 145 MB/s
> for UDP compared to filling 12 1gige pipes for TCP.  I have a patch to fix this,
> but given the inherent data corruption issues of UDP I haven't bothered posting
> the most recent version.
> 
> 
> 
> > TCP and its timeout/retransmission behavior isn't always the best choice.
> >
> >   
> The timeout & retrans that sunrpc implements on top of UDP is arguably worse,
> especially if you use the "soft" mount option.
> 
> -- 
> Greg Banks, R&D Software Engineer, SGI Australian Software Group.
> The cake is *not* a lie.
> I don't speak for SGI.
>