Intermittent RHE3-u3 client mount failures to Netapp fileserver

Intermittent RHE3-u3 client mount failures to Netapp fileserver

[David Meleedy]

I have been trying to troubleshoot this problem on Redhat Enterprise 3
WS update 3, both iX86 and amd64 versions.  I have already sent this
to the automounter mailing list, and I suspect they don't have a
solution because it is more nfs related.

The problem is that intermittently, a previously mounted directory that
had been mounted with the automounter, and then unmounted becomes
unavailable.  The amount of time of unavailability seems to be related
to the --timeout=60 option specified to the automount program.

I don't believe the routing equipment is at fault because using the
same network ports, we have had Redhat 8 clients working just fine.

client info
===========
Red Hat Enterprise Linux WS release 3 (Taroon Update 3)
kernel: 2.4.21-20.EL
nfs-utils: 1.0.6-31EL
mount: 2.11y-31.1
autofs: 4.1.3-12

server info
===========
aflac> version
NetApp Release 6.4.2: Mon Sep 15 12:35:28 PDT 2003


This is the client rpc info:
alfalfa-56: rpcinfo -p alfalfa
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  32768  status
    100024    1   tcp  32768  status
    100007    2   udp    816  ypbind
    100007    1   udp    816  ypbind
    100007    2   tcp    819  ypbind
    100007    1   tcp    819  ypbind
    391002    2   tcp  32769  sgi_fam
    100011    1   udp    711  rquotad
    100011    2   udp    711  rquotad
    100011    1   tcp    731  rquotad
    100011    2   tcp    731  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100021    1   udp  32770  nlockmgr
    100021    3   udp  32770  nlockmgr
    100021    4   udp  32770  nlockmgr
    100021    1   tcp  32770  nlockmgr
    100021    3   tcp  32770  nlockmgr
    100021    4   tcp  32770  nlockmgr
    100005    1   udp    728  mountd
    100005    1   tcp    744  mountd
    100005    2   udp    728  mountd
    100005    2   tcp    744  mountd
    100005    3   udp    728  mountd
    100005    3   tcp    744  mountd
    100001    3   udp    725  rstatd
    100001    2   udp    725  rstatd
    100001    1   udp    725  rstatd

This is the server rpc info:

alfalfa-57: rpcinfo -p aflac
   program vers proto   port
    100011    1   udp   4049  rquotad
    100021    4   tcp   4045  nlockmgr
    100021    3   tcp   4045  nlockmgr
    100021    1   tcp   4045  nlockmgr
    100021    4   udp   4045  nlockmgr
    100021    3   udp   4045  nlockmgr
    100021    1   udp   4045  nlockmgr
    100024    1   tcp   4047  status
    100024    1   udp   4047  status
    100005    3   tcp   4046  mountd
    100005    2   tcp   4046  mountd
    100005    1   tcp   4046  mountd
    100005    3   udp   4046  mountd
    100005    2   udp   4046  mountd
    100005    1   udp   4046  mountd
    100003    4   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   udp   2049  nfs
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper

The automounter is configured as follows:
/etc/sysconfig/autofs:
LOCALOPTIONS="-DSITE=nwd_ -DARCH=iX86 -DOSREL=redhat3"
DAEMONOPTIONS="--timeout=60 --verbose --debug"

/etc/auto.master

a typical entry:
/cpd            /etc/auto.cpd

I have also tried:
/cpd            /etc/auto.cpd -vers=2
(that didn't seem to fix the problem)

I am now trying:
/cpd		/etc/auto.cpd -nolock

auto.cpd is a script that returns a mount point, e.g.

alfalfa-72: /etc/auto.cpd gnu
aflac,aflac-171:/vol/vol1/cpd/${SITE}${ARCH}_${OSREL}/&


These are the kind of messages I am seeing:
============================================================================
Dec 12 06:00:37 alfalfa kernel: nfs warning: mount version older than kernel
Dec 12 06:00:37 alfalfa kernel: RPC: Can't bind to reserved port (98).
Dec 12 06:00:37 alfalfa kernel: nfs_get_root: getattr error = 5
Dec 12 06:00:37 alfalfa kernel: nfs_read_super: get root inode failed
Dec 12 06:00:37 alfalfa automount[1617]: >> mount: wrong fs type, bad option, 
ba
d superblock on aflac:/vol/vol1/cpd/nwd_local,
Dec 12 06:00:37 alfalfa automount[1617]: >>        or too many mounted file 
syst
ems
Dec 12 06:00:37 alfalfa automount[1617]: mount(nfs): nfs: mount failure 
aflac:/v
ol/vol1/cpd/nwd_local on /cpd/local
Dec 12 06:00:37 alfalfa automount[1617]: failed to mount /cpd/local
Dec 12 06:00:37 alfalfa kernel: RPC: Can't bind to reserved port (98).
Dec 12 06:00:37 alfalfa kernel: nfs_get_root: getattr error = 5
Dec 12 06:00:37 alfalfa kernel: RPC: Can't bind to reserved port (98).
Dec 12 06:00:37 alfalfa kernel: nfs_get_root: getattr error = 5
Dec 12 06:00:37 alfalfa kernel: nfs_read_super: get root inode failed
Dec 12 06:00:40 alfalfa kernel: nfs warning: mount version older than kernel
Dec 12 06:00:40 alfalfa kernel: RPC: Can't bind to reserved port (98).
Dec 12 06:00:40 alfalfa kernel: nfs_get_root: getattr error = 5
Dec 12 06:00:40 alfalfa kernel: nfs_read_super: get root inode failed
Dec 12 06:00:40 alfalfa automount[1625]: >> mount: wrong fs type, bad option, 
ba
d superblock on aflac:/vol/vol1/cpd/nwd_local,
Dec 12 06:00:40 alfalfa automount[1625]: >>        or too many mounted file 
syst
ems
Dec 12 06:00:40 alfalfa automount[1625]: mount(nfs): nfs: mount failure 
aflac:/v
ol/vol1/cpd/nwd_local on /cpd/local
Dec 12 06:00:40 alfalfa automount[1625]: failed to mount /cpd/local
Dec 12 06:00:40 alfalfa kernel: RPC: Can't bind to reserved port (98).
Dec 12 06:00:40 alfalfa kernel: nfs_get_root: getattr error = 5
Dec 12 06:00:40 alfalfa kernel: RPC: Can't bind to reserved port (98).
Dec 12 06:00:40 alfalfa kernel: nfs_get_root: getattr error = 5
Dec 12 06:00:40 alfalfa kernel: nfs_read_super: get root inode failed
Dec 12 06:00:40 alfalfa kernel: nfs warning: mount version older than kernel
Dec 12 06:00:40 alfalfa kernel: RPC: Can't bind to reserved port (98).
Dec 12 06:00:40 alfalfa kernel: nfs_get_root: getattr error = 5
Dec 12 06:00:40 alfalfa kernel: nfs_read_super: get root inode failed
Dec 12 06:00:40 alfalfa automount[1761]: >> mount: wrong fs type, bad option, 
ba
d superblock on aflac:/vol/vol1/cpd/nwd_local,
Dec 12 06:00:40 alfalfa automount[1761]: >>        or too many mounted file 
syst
ems
Dec 12 06:00:40 alfalfa automount[1761]: mount(nfs): nfs: mount failure 
aflac:/v
ol/vol1/cpd/nwd_local on /cpd/local
Dec 12 06:00:40 alfalfa automount[1761]: failed to mount /cpd/local
============================================================================

The reason I am trying "vers=2", and "nolock" nfs options to fix
this is because of this article -- so far vers=2 did not fix the problem:

http://support.trittontechnologies.com/kbarticles/nas_asap/T4133-1015.htm

I was wondering if anyone here has some better ideas of things I could
try, or if anyone knows what is going wrong here.

Thanks,

-Dave

________________________________________________________________________
David Meleedy				Analog Devices, Inc.
David.Meleedy@analog.com		Three Technology Way
Phone: 781 461 3494			Norwood, MA  02062-9106  USA




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Intermittent RHE3-u3 client mount failures to Netapp fileserver

[Lever, Charles]

> I have been trying to troubleshoot this problem on Redhat Enterprise 3
> WS update 3, both iX86 and amd64 versions.  I have already sent this
> to the automounter mailing list, and I suspect they don't have a
> solution because it is more nfs related.
>=20
> The problem is that intermittently, a previously mounted=20
> directory that
> had been mounted with the automounter, and then unmounted becomes
> unavailable.  The amount of time of unavailability seems to be related
> to the --timeout=3D60 option specified to the automount program.
>=20
> I don't believe the routing equipment is at fault because using the
> same network ports, we have had Redhat 8 clients working just fine.

what's running on port 98 on your RHEL 3.0 boxes?  (netstat)

and check the hardware too.  we had an IPMI hardware implementation from
Intel at one point that took a port without notifying the operating
system, and that caused similar failures.

vers=3D2 and nolock are probably not necessary, and will likely cause =
more
problems than they solve.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Intermittent RHE3-u3 client mount failures to Netapp fileserver

[Mike Waychison]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lever, Charles wrote:
> 
> and check the hardware too.  we had an IPMI hardware implementation from
> Intel at one point that took a port without notifying the operating
> system, and that caused similar failures.
> 

I've seen this too with e1000 hardware.  I usually blacklist the ports
by adding the 623 for udp / tcp to my (x)inetd config and pointing them
to /bin/true.

I'm wondering if anybody has figured out a way to 'blacklist' these
ports somehow from within the drivers..    ..probably best left as a
hotplug event.


- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE:  The opinions expressed in this email are held by me,
and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBwGmxdQs4kOxk3/MRApeFAJkBj5Dn2KDsdkmknxUh+fmunZrALQCfTGm6
7N0fuLRQ22ff2MIUtzTS6KM=
=xdDY
-----END PGP SIGNATURE-----


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Intermittent RHE3-u3 client mount failures to Netapp fileserver

[Lever, Charles]

olaf-

as we consider the changes you suggested to the port allocation
algorithm, can we think about the port blacklisting problem too?


> -----Original Message-----
> From: Mike Waychison [mailto:Michael.Waychison@Sun.COM]=20
> Sent: Wednesday, December 15, 2004 11:43 AM
> To: Lever, Charles
> Cc: David Meleedy; nfs@lists.sourceforge.net
> Subject: Re: [NFS] Intermittent RHE3-u3 client mount failures=20
> to Netapp fileserver
>=20
>=20
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>=20
> Lever, Charles wrote:
> >=20
> > and check the hardware too.  we had an IPMI hardware=20
> implementation from
> > Intel at one point that took a port without notifying the operating
> > system, and that caused similar failures.
> >=20
>=20
> I've seen this too with e1000 hardware.  I usually blacklist the ports
> by adding the 623 for udp / tcp to my (x)inetd config and=20
> pointing them
> to /bin/true.
>=20
> I'm wondering if anybody has figured out a way to 'blacklist' these
> ports somehow from within the drivers..    ..probably best left as a
> hotplug event.
>=20
>=20
> - --
> Mike Waychison
> Sun Microsystems, Inc.
> 1 (650) 352-5299 voice
> 1 (416) 202-8336 voice
>=20
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> NOTICE:  The opinions expressed in this email are held by me,
> and may not represent the views of Sun Microsystems, Inc.
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.5 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>=20
> iD8DBQFBwGmxdQs4kOxk3/MRApeFAJkBj5Dn2KDsdkmknxUh+fmunZrALQCfTGm6
> 7N0fuLRQ22ff2MIUtzTS6KM=3D
> =3DxdDY
> -----END PGP SIGNATURE-----
>=20


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs