Text based mount options ignoring the preferred rwsize?

Text based mount options ignoring the preferred rwsize?

[James Pearson]

I've noticed a difference in the rsize used when mounting a file system 
  between using text and binary mount options.

The client is running a CentOS5 based distro with a 2.6.32-rc8 kernel

The server has a preferred rsize of 128kb and maximum rsize of 512kb

When I use mount.nfs from CentOS5/RHEL5 nfs-utils (based on v1.0.9) and 
don't give any rsize option, it mounts the file system with a rsize of 
128kb. This uses binary mount options

But, when using mount.nfs from nfs-utils 1.2.0, the file system is 
mounted with an rsize of 512kb

Looking at the nfs-utils and kernel source, it appears that for binary 
options, rsize is set to 0 if not given by mount.nfs, and the kernel 
eventually, in this case, increases this to preferred size.

But for text mount options, if not set by mount.nfs, the default size is 
  set to NFS_MAX_FILE_IO_SIZE in the kernel, which, in this case, gets 
reduced to the server maximum size.

Should the kernel be setting rsize (and wsize) to 0 by default?

Thanks

James Pearson

Re: Text based mount options ignoring the preferred rwsize?

[Chuck Lever]

On Sep 8, 2009, at 12:47 PM, James Pearson wrote:
> I've noticed a difference in the rsize used when mounting a file  
> system  between using text and binary mount options.
>
> The client is running a CentOS5 based distro with a 2.6.32-rc8 kernel
>
> The server has a preferred rsize of 128kb and maximum rsize of 512kb
>
> When I use mount.nfs from CentOS5/RHEL5 nfs-utils (based on v1.0.9)  
> and don't give any rsize option, it mounts the file system with a  
> rsize of 128kb. This uses binary mount options
>
> But, when using mount.nfs from nfs-utils 1.2.0, the file system is  
> mounted with an rsize of 512kb
>
> Looking at the nfs-utils and kernel source, it appears that for  
> binary options, rsize is set to 0 if not given by mount.nfs, and the  
> kernel eventually, in this case, increases this to preferred size.
>
> But for text mount options, if not set by mount.nfs, the default  
> size is  set to NFS_MAX_FILE_IO_SIZE in the kernel, which, in this  
> case, gets reduced to the server maximum size.
>
> Should the kernel be setting rsize (and wsize) to 0 by default?

nfs(5) says:

"If an [rw]size value is not specified, or if the specified [rw]size  
value is larger than the maximum that either client or server can  
support, the client and server negotiate the largest [rw]size value  
that they can both support."

So the text-based behavior is what is documented now.

Does anyone know of a reason to use the server's "preferred" transfer  
size rather than the largest size supported by both client and  
server?  Usually those are the same.

> Thanks
>
> James Pearson
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"  
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text based mount options ignoring the preferred rwsize?

[James Pearson]

Chuck Lever wrote:
>>
>> Should the kernel be setting rsize (and wsize) to 0 by default?
> 
> 
> nfs(5) says:
> 
> "If an [rw]size value is not specified, or if the specified [rw]size  
> value is larger than the maximum that either client or server can  
> support, the client and server negotiate the largest [rw]size value  
> that they can both support."
> 
> So the text-based behavior is what is documented now.
> 
> Does anyone know of a reason to use the server's "preferred" transfer  
> size rather than the largest size supported by both client and  server?  
> Usually those are the same.

In this case, the manufacturer of the NFS server recommends using 128Kb 
for rsize and 512Kb for wsize - although the maximum rsize it supports 
is 512Kb. I assume in their testing, these values have given optimal 
performance figures.

The default behaviour with binary mount options when no [rw]size is to 
select these preferred values - which to me, makes sense - as by not 
giving a [rw]size, you are leaving it up the server to pick the 'best' 
values for you - which I guess in most (all other?) cases happen to be 
the maximum size.

James Pearson

Re: Text based mount options ignoring the preferred rwsize?

[Chuck Lever]

On Sep 9, 2009, at 5:47 PM, James Pearson wrote:
> Chuck Lever wrote:
>>>
>>> Should the kernel be setting rsize (and wsize) to 0 by default?
>> nfs(5) says:
>> "If an [rw]size value is not specified, or if the specified  
>> [rw]size  value is larger than the maximum that either client or  
>> server can  support, the client and server negotiate the largest  
>> [rw]size value  that they can both support."
>> So the text-based behavior is what is documented now.
>> Does anyone know of a reason to use the server's "preferred"  
>> transfer  size rather than the largest size supported by both  
>> client and  server?  Usually those are the same.
>
> In this case, the manufacturer of the NFS server recommends using  
> 128Kb for rsize and 512Kb for wsize - although the maximum rsize it  
> supports is 512Kb. I assume in their testing, these values have  
> given optimal performance figures.

In this case maybe you should set these explicitly.

> The default behaviour with binary mount options when no [rw]size is  
> to select these preferred values - which to me, makes sense - as by  
> not giving a [rw]size, you are leaving it up the server to pick the  
> 'best' values for you - which I guess in most (all other?) cases  
> happen to be the maximum size.

It might make sense to think of the server's preference as the value  
you should use.  But for the client, the larger the better in nearly  
every case.  So which one should you choose by default?

Someone could still make the case that the old behavior was better in  
some way (and provide benchmark evidence for this, of course, for  
multiple combinations of implementations across different networks).

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text based mount options ignoring the preferred rwsize?

[Trond Myklebust]

On Wed, 2009-09-09 at 22:47 +0100, James Pearson wrote:
> The default behaviour with binary mount options when no [rw]size is to 
> select these preferred values - which to me, makes sense - as by not 
> giving a [rw]size, you are leaving it up the server to pick the 'best' 
> values for you - which I guess in most (all other?) cases happen to be 
> the maximum size.

Right. The above was indeed the guiding principle back when I did the
rsize/wsize negotiation for NFSv3 and NFSv2 for the binary mount code.

The NFS protocol specifies that the maximum values are there to tell you
that the server will do short read/writes if you exceed these. However,
the preferred values may correspond to a different 'sweet spot' for the
server read and write implementations.

Cheers
    Trond

Re: Text based mount options ignoring the preferred rwsize?

[James Pearson]

Trond Myklebust wrote:
> On Wed, 2009-09-09 at 22:47 +0100, James Pearson wrote:
> 
>>The default behaviour with binary mount options when no [rw]size is to 
>>select these preferred values - which to me, makes sense - as by not 
>>giving a [rw]size, you are leaving it up the server to pick the 'best' 
>>values for you - which I guess in most (all other?) cases happen to be 
>>the maximum size.
> 
> 
> Right. The above was indeed the guiding principle back when I did the
> rsize/wsize negotiation for NFSv3 and NFSv2 for the binary mount code.
> 
> The NFS protocol specifies that the maximum values are there to tell you
> that the server will do short read/writes if you exceed these. However,
> the preferred values may correspond to a different 'sweet spot' for the
> server read and write implementations.

So does that mean that the binary mount options are doing the right 
thing, whereas the text mount options are not? Also, just to confirm, 
I'm using NFSv3.

Thanks

James Pearson

Re: Text based mount options ignoring the preferred rwsize?

[Chuck Lever]

On Sep 10, 2009, at 5:08 AM, James Pearson wrote:
> Trond Myklebust wrote:
>> On Wed, 2009-09-09 at 22:47 +0100, James Pearson wrote:
>>> The default behaviour with binary mount options when no [rw]size  
>>> is to select these preferred values - which to me, makes sense -  
>>> as by not giving a [rw]size, you are leaving it up the server to  
>>> pick the 'best' values for you - which I guess in most (all  
>>> other?) cases happen to be the maximum size.
>> Right. The above was indeed the guiding principle back when I did the
>> rsize/wsize negotiation for NFSv3 and NFSv2 for the binary mount  
>> code.
>> The NFS protocol specifies that the maximum values are there to  
>> tell you
>> that the server will do short read/writes if you exceed these.  
>> However,
>> the preferred values may correspond to a different 'sweet spot' for  
>> the
>> server read and write implementations.
>
> So does that mean that the binary mount options are doing the right  
> thing, whereas the text mount options are not? Also, just to  
> confirm, I'm using NFSv3.

I don't think there is a clear right or wrong here.  The binary  
behavior follows protocol guidelines, which is fine.  But, there are  
plenty of cases where using the server's preferred size by default is  
not always optimal.  For example, the server may advertise a  
preference for 32768, but using that setting with UDP on a congested  
network would result in poor performance.

In other words, the client may decide that using a larger or smaller  
setting than the server's preference would be better.  Or, it may use  
the largest possible setting for r/wsize, but advertise the server's  
preference number in stat(2) results.

Fwiw, I don't have any problem switching the text-based defaults to  
zero.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com