nfs-utils 1.0.5 break amd?

nfs-utils 1.0.5 break amd?

[Marc Schmitt]

Hi all,

After upgrading a RedHat 7.3 box with kernel 2.4.18-27.7xsmp from
nfs-utils 1.0.3 to 1.0.5, amd stopped working.

On the client (am-utils-6.0.7-4, nfs-utils-0.3.3-6.73), I issue:

ls /net/<fileserver>

With 1.0.3, I get the correct directory listing under /net/<fileserver>

With 1.0.5, I get:
ls: /net/<fileserver>: Input/output error


On the server, with 1.0.3, I'd see an entry for every export in
/etc/exports in /var/log/messages, mount refused or granted, according
to the export permissions. 
With 1.0.5, I get one single entry:
Jul 31 19:14:34 <fileserver> rpc.mountd: export request from <client>


For 1.0.5, the client reports the following errors in /var/log/messages:
Jul 31 19:24:04 <client> amd[1072]: No fs type specified (key =
"/defaults", map = ""root"")
Jul 31 19:24:04 <client> amd[1072]: get_nfs_version NFS(3,tcp) failed
for <fileserver>: : RPC: Program not registered
Jul 31 19:24:04 <client> amd[1072]: get_nfs_version: returning (3,udp)
on host <fileserver>
Jul 31 19:24:04 <client> amd[1072]: Using NFS version 3, protocol udp on
host <fileserver>
Jul 31 19:24:04 <client> amd[1072]: Flushed /net/<fileserver>; dependent
on <fileserver>
Jul 31 19:24:04 <client> amd[1072]: recompute_portmap: NFS version 3
Jul 31 19:24:04 <client> amd[1072]: Using MOUNT version: 3
Jul 31 19:24:04 <client> amd[1072]: get_nfs_version NFS(3,tcp) failed
for <fileserver>: : RPC: Program not registered
Jul 31 19:24:04 <client> amd[1072]: get_nfs_version: returning (3,udp)
on host <fileserver>
Jul 31 19:24:04 <client> amd[1072]: Using NFS version 3, protocol udp on
host <fileserver>
Jul 31 19:24:04 <client> amd[25799]: amfs_host_fmount: NFS version 3
Jul 31 19:24:04 <client> amd[25799]: host_fmount rpc failed: RPC: Remote
system error
Jul 31 19:24:04 <client> amd[1072]: /net/<fileserver>: mount
(amfs_auto_cont): Input/output error

Here is my /etc/sysconfig/nfs on the server:
# For more information on nfs tuning, please see the NFS-HOWTO
# http://nfs.sourceforge.net/nfs-howto/

# Pass any additional options for mountd.
# MOUNTD_OPTIONS=

# Pin mountd to a given port rather than random one from portmapper
# MOUNTD_PORT=

# Don't advertise TCP for mount.
MOUNTD_TCP=no

# NFS V3
MOUNTD_NFS_V3=yes

# NFS V2
MOUNTD_NFS_V2=yes

# The number of open file descriptors
#MOUNTD_OPEN_FILES=128

# Pass the number of instances of nfsd (8 is default; 16 or more
# might be needed to handle heavy client traffic)
NFSDCOUNT=32

# Increase the memory limits on the socket input queues for
# the nfs processes .. NFS  benchmark SPECsfs demonstrate a
# need for a larger than default size (64kb) .. setting
# TUNE_QUEUE to yes will set the values to 256kb.
TUNE_QUEUE="yes"
NFS_QS=262144


Any idea what could be wrong? TIA

Regards,
    Marc





-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs-utils 1.0.5 break amd?

[Neil Brown]

On  July 31, mschmitt@inf.ethz.ch wrote:
> Hi all,
> 
> After upgrading a RedHat 7.3 box with kernel 2.4.18-27.7xsmp from
> nfs-utils 1.0.3 to 1.0.5, amd stopped working.
> 
> On the client (am-utils-6.0.7-4, nfs-utils-0.3.3-6.73), I issue:
> 
> ls /net/<fileserver>
> 
> With 1.0.3, I get the correct directory listing under /net/<fileserver>
> 
> With 1.0.5, I get:
> ls: /net/<fileserver>: Input/output error

Wierd.

> 
> 
> On the server, with 1.0.3, I'd see an entry for every export in
> /etc/exports in /var/log/messages, mount refused or granted, according
> to the export permissions. 
> With 1.0.5, I get one single entry:
> Jul 31 19:14:34 <fileserver> rpc.mountd: export request from <client>
> 

Can you run
   showmount -e <fileserver>
on <client> and see what it returns?
If it doesn't return the right stuff, maybe strace mountd on the
server while you are running the showmout on the client.

NeilBrown


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs-utils 1.0.5 break amd?

[Marc Schmitt]

Ion is right, it's not the amd and Neil, you're right, too, `showmount
-e` is the way to go. :)

Here is a short cookbook on how to reproduce the problem on a test
server (no client needed to trigger the problem):

1) RedHat 7.3 with kernel 2.4.18-27.7.x
2) nfs-utils 1.0.5-1 (built with rpm -ta)
3) /etc/sysconfig/nfs
# For more information on nfs tuning, please see the NFS-HOWTO
# http://nfs.sourceforge.net/nfs-howto/
  
# Pass any additional options for mountd.
# MOUNTD_OPTIONS=
  
# Pin mountd to a given port rather than random one from portmapper
# MOUNTD_PORT=
  
# Don't advertise TCP for mount.
MOUNTD_TCP=no
  
# NFS V3
MOUNTD_NFS_V3=yes
  
# NFS V2
MOUNTD_NFS_V2=yes
  
# The number of open file descriptors
#MOUNTD_OPEN_FILES=128
  
# Pass the number of instances of nfsd (8 is default; 16 or more
# might be needed to handle heavy client traffic)
NFSDCOUNT=32
  
# Increase the memory limits on the socket input queues for
# the nfs processes .. NFS  benchmark SPECsfs demonstrate a
# need for a larger than default size (64kb) .. setting
# TUNE_QUEUE to yes will set the values to 256kb.
TUNE_QUEUE="yes"
NFS_QS=262144

4) create an exports file with 183 entries (here is exactly what I used,
in case the entries char length matters):
/home2/nfs/1 192.168.1.40(rw)
.
.
.
/home2/nfs/183 192.168.1.40(rw)

5) service nfslock start; service nfs start
6) issue `showmount -e`, it will show:
/home2/nfs/1 192.168.1.41
.
.
.
/home2/nfs/183 192.168.1.41

7) echo "/home2/nfs/184 192.168.1.40(rw)" >> /etc/exports
8) `exportfs -r` (rc 0)
9) `showmount -e` (rc 1: rpc mount export: RPC: Remote system error)

The moment I add the 184th entry to /etc/exports, `showmount -e` fails
with the same kind of error as amd does (host_fmount rpc failed: RPC:
Remote system error).

Now comes the interesting part, changing MOUNTD_TCP in
/etc/sysconfig/nfs from no to yes will make the problem go away. With
MOUNTD_TCP=yes I went up to 1000 exports and `showmount -e` worked fine.

To be honest, this is something I had changed in the same go when I
updated 1.0.3 to 1.0.5. So, for completeness, I ran the same tests with
1.0.3, the result is identical. For more than 183 entries and
MOUNTD_TCP=no, `showmount -e` returns the RPC error under 1.0.3, too.

I'm sorry about this, I should have investigated more carefully from
beginning on.

Greetings,
    Marc

On Fri, 2003-08-01 at 05:12, Neil Brown wrote:
> On  July 31, mschmitt@inf.ethz.ch wrote:
> > Hi all,
> > 
> > After upgrading a RedHat 7.3 box with kernel 2.4.18-27.7xsmp from
> > nfs-utils 1.0.3 to 1.0.5, amd stopped working.
> > 
> > On the client (am-utils-6.0.7-4, nfs-utils-0.3.3-6.73), I issue:
> > 
> > ls /net/<fileserver>
> > 
> > With 1.0.3, I get the correct directory listing under /net/<fileserver>
> > 
> > With 1.0.5, I get:
> > ls: /net/<fileserver>: Input/output error
> 
> Wierd.
> 
> > 
> > 
> > On the server, with 1.0.3, I'd see an entry for every export in
> > /etc/exports in /var/log/messages, mount refused or granted, according
> > to the export permissions. 
> > With 1.0.5, I get one single entry:
> > Jul 31 19:14:34 <fileserver> rpc.mountd: export request from <client>
> > 
> 
> Can you run
>    showmount -e <fileserver>
> on <client> and see what it returns?
> If it doesn't return the right stuff, maybe strace mountd on the
> server while you are running the showmout on the client.
> 
> NeilBrown



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs-utils 1.0.5 break amd?

[Neil Brown]

On  August 4, mschmitt@inf.ethz.ch wrote:
> 
> Now comes the interesting part, changing MOUNTD_TCP in
> /etc/sysconfig/nfs from no to yes will make the problem go away. With
> MOUNTD_TCP=yes I went up to 1000 exports and `showmount -e` worked fine.

Yes,  that makes sense.  UDP has a limited request/reply size.  TCP
doesn't.

I honestly cannot think of any good reason to have MOUNTD_TCP=no  
Maybe I should update the man page
  
       -n  or  --no-tcp
              Don't advertise TCP for mount.

              NOTE: there is no good reason do use this option, and
	      if you do, then things will mysteriously break.

:-)
NeilBrown


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs-utils 1.0.5 break amd?

[Marc Schmitt]

Well, again it's me confusing stuff.
With MOUNTD_TCP=no I thought I could prevent the clients from mounting
the server with NFS over tcp... which is something I already have,
because current RH kernels don't have NFS server support over TCP
enabled in the kernel.

Thanks for your time.

Greetings,
    Marc

On Tue, 2003-08-05 at 04:13, Neil Brown wrote:
> On  August 4, mschmitt@inf.ethz.ch wrote:
> > 
> > Now comes the interesting part, changing MOUNTD_TCP in
> > /etc/sysconfig/nfs from no to yes will make the problem go away. With
> > MOUNTD_TCP=yes I went up to 1000 exports and `showmount -e` worked fine.
> 
> Yes,  that makes sense.  UDP has a limited request/reply size.  TCP
> doesn't.
> 
> I honestly cannot think of any good reason to have MOUNTD_TCP=no  
> Maybe I should update the man page
>   
>        -n  or  --no-tcp
>               Don't advertise TCP for mount.
> 
>               NOTE: there is no good reason do use this option, and
> 	      if you do, then things will mysteriously break.
> 
> :-)
> NeilBrown
> 
> 
> -------------------------------------------------------
> This SF.Net email sponsored by: Free pre-built ASP.NET sites including
> Data Reports, E-commerce, Portals, and Forums are available now.
> Download today and enter to win an XBOX or Visual Studio .NET.
> http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs




-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs-utils 1.0.5 break amd?

[Ion Badulescu]

On 05 Aug 2003 10:22:02 +0200, Marc Schmitt <mschmitt@inf.ethz.ch> wrote:
> Well, again it's me confusing stuff.
> With MOUNTD_TCP=no I thought I could prevent the clients from mounting
> the server with NFS over tcp... which is something I already have,
> because current RH kernels don't have NFS server support over TCP
> enabled in the kernel.

No, the transport protocol used to talk to mountd has no effect
whatsoever on the transport protocol used to talk to nfsd. Only the
version of the mountd protocol matters (and note that while mountd v3
corresponds to nfs v3, mountd v1 and v2 both correspond to nfs v2).

Since you use amd, you can always force it not to use tcp for nfs
mounts.  You can either do it globally for the client by setting
nfs_proto = udp in its amd.conf, or on a per mount entry basis by adding
proto = udp to that entry's mount options.

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs-utils 1.0.5 break amd?

[Ion Badulescu]

On 31 Jul 2003 19:32:22 +0200, Marc Schmitt <mschmitt@inf.ethz.ch> wrote:

> After upgrading a RedHat 7.3 box with kernel 2.4.18-27.7xsmp from
> nfs-utils 1.0.3 to 1.0.5, amd stopped working.
> 
> On the client (am-utils-6.0.7-4, nfs-utils-0.3.3-6.73), I issue:
> 
> ls /net/<fileserver>
> 
> With 1.0.3, I get the correct directory listing under /net/<fileserver>
> 
> With 1.0.5, I get:
> ls: /net/<fileserver>: Input/output error

Can you mount any shares from <fileserver> by hand, using mount? If not, 
as I suspect, then the problem is not amd.

Ion
[amd co-maintainer]

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs-utils 1.0.5 break amd?

[Marc Schmitt]

On Mon, 2003-08-04 at 18:05, Ion Badulescu wrote:
> On 31 Jul 2003 19:32:22 +0200, Marc Schmitt <mschmitt@inf.ethz.ch> wrote:
> 
> > After upgrading a RedHat 7.3 box with kernel 2.4.18-27.7xsmp from
> > nfs-utils 1.0.3 to 1.0.5, amd stopped working.
> > 
> > On the client (am-utils-6.0.7-4, nfs-utils-0.3.3-6.73), I issue:
> > 
> > ls /net/<fileserver>
> > 
> > With 1.0.3, I get the correct directory listing under /net/<fileserver>
> > 
> > With 1.0.5, I get:
> > ls: /net/<fileserver>: Input/output error
> 
> Can you mount any shares from <fileserver> by hand, using mount? If not, 
> as I suspect, then the problem is not amd.

Yes, the machine is currently in production (home and project
directories) and has 174 exports, all of them work (otherwhise I'd have
heard something from my users :).

The client I run 'ls /net/<fileserver>' on has access to 3 of the
exported directories, which used to get mounted with nfs-utils 1.0.3.

I'll try to find more debug infos tonight.

Greetz
    Marc



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs