* Re: Intermittant problems mounting with automounter @ 2004-11-30 20:07 David Meleedy 2004-12-14 14:10 ` Todd Denniston 2004-12-15 0:13 ` Mike Waychison 0 siblings, 2 replies; 5+ messages in thread From: David Meleedy @ 2004-11-30 20:07 UTC (permalink / raw) To: autofs Well, in order to troubleshoot this issue further, autofs on Redhat Enterprise 3, update 3 I removed all options from the /etc/auto.master file: from: /cpd /etc/auto.cpd hard,intr,rsize=8192,wsize=8192 to: /cpd /etc/auto.cpd After this change, the following errors seemed to go away: ==== Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 Nov 19 11:27:38 codered kernel: nfs_read_super: get root inode failed Nov 19 11:27:38 codered kernel: nfs warning: mount version older than kernel Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 Nov 19 11:27:38 codered kernel: nfs_read_super: get root inode failed Nov 19 11:27:38 codered automount[3623]: >> mount: wrong fs type, bad option, bad superblock on aflac:/vol/vol1/cpd/nwd_usr, Nov 19 11:27:38 codered automount[3623]: >> or too many mounted file systems Nov 19 11:27:38 codered automount[3623]: mount(nfs): nfs: mount failure aflac:/vol/vol1/cpd/nwd_usr on /cpd/usr Nov 19 11:27:38 codered automount[3623]: failed to mount /cpd/usr ===== So whatever it is might be caused by the options? -Dave ________________________________________________________________________ David Meleedy Analog Devices, Inc. David.Meleedy@analog.com Three Technology Way Phone: 781 461 3494 Norwood, MA 02062-9106 USA ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Re: Intermittant problems mounting with automounter 2004-11-30 20:07 Intermittant problems mounting with automounter David Meleedy @ 2004-12-14 14:10 ` Todd Denniston 2004-12-14 23:36 ` David Meleedy 2004-12-15 0:13 ` Mike Waychison 1 sibling, 1 reply; 5+ messages in thread From: Todd Denniston @ 2004-12-14 14:10 UTC (permalink / raw) To: David Meleedy; +Cc: autofs David Meleedy wrote: > > Well, in order to troubleshoot this issue further, autofs on Redhat > Enterprise 3, update 3 > > I removed all options from the /etc/auto.master file: > > from: > /cpd /etc/auto.cpd hard,intr,rsize=8192,wsize=8192 > to: > /cpd /etc/auto.cpd > > After this change, the following errors seemed to go away: > > ==== > Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). > Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 > Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). > Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 > Nov 19 11:27:38 codered kernel: nfs_read_super: get root inode failed > Nov 19 11:27:38 codered kernel: nfs warning: mount version older than kernel > Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). > Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 > Nov 19 11:27:38 codered kernel: nfs_read_super: get root inode failed > Nov 19 11:27:38 codered automount[3623]: >> mount: wrong fs type, bad option, > bad superblock on aflac:/vol/vol1/cpd/nwd_usr, > Nov 19 11:27:38 codered automount[3623]: >> or too many mounted file > systems > Nov 19 11:27:38 codered automount[3623]: mount(nfs): nfs: mount failure > aflac:/vol/vol1/cpd/nwd_usr on /cpd/usr > Nov 19 11:27:38 codered automount[3623]: failed to mount /cpd/usr > ===== > > So whatever it is might be caused by the options? Check on your server, the above looks like messages I have seen on some clients lately, and see if you have messages of the form Nov 19 11:27:?? server rpc.mountd: getfh failed: Operation not permitted These happened to me when some clients who were using soft mount option finally switched to hard. I had to fallover from one of my nfs drbd cluster machines to the other to fix it. effectively it looked to the client like a server reboot. shutting down the nfsserver and nfslock services then restarting them might be enough. Note: soft mounts caused many other problems, so for me this switching to hard mount is only a quick annoyance to avoid other more troublesome problems, I just had to learn to recognize it. -- Todd Denniston Crane Division, Naval Surface Warfare Center (NSWC Crane) Harnessing the Power of Technology for the Warfighter ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Re: Intermittant problems mounting with automounter 2004-12-14 14:10 ` Todd Denniston @ 2004-12-14 23:36 ` David Meleedy 0 siblings, 0 replies; 5+ messages in thread From: David Meleedy @ 2004-12-14 23:36 UTC (permalink / raw) To: Todd Denniston; +Cc: autofs > Check on your server, the above looks like messages I have seen on some > clients lately, and see if you have messages of the form > Nov 19 11:27:?? server rpc.mountd: getfh failed: Operation not permitted I haven't seen this error in any of the cases. As far as I know "hard" mounts are the default behavior. I have not specified "soft" mounts as an option. from the nfs man page: hard If an NFS file operation has a major timeout then report "server not responding" on the console and continue retrying indefinitely. This is the default. -Dave ________________________________________________________________________ David Meleedy Analog Devices, Inc. David.Meleedy@analog.com Three Technology Way Phone: 781 461 3494 Norwood, MA 02062-9106 USA ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Re: Intermittant problems mounting with automounter 2004-11-30 20:07 Intermittant problems mounting with automounter David Meleedy 2004-12-14 14:10 ` Todd Denniston @ 2004-12-15 0:13 ` Mike Waychison 2004-12-15 15:02 ` Jeff Moyer 1 sibling, 1 reply; 5+ messages in thread From: Mike Waychison @ 2004-12-15 0:13 UTC (permalink / raw) To: David Meleedy; +Cc: autofs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Meleedy wrote: > Well, in order to troubleshoot this issue further, autofs on Redhat > Enterprise 3, update 3 > > I removed all options from the /etc/auto.master file: > > from: > > /cpd /etc/auto.cpd hard,intr,rsize=8192,wsize=8192 > > to: > > /cpd /etc/auto.cpd > > > After this change, the following errors seemed to go away: > > ==== > Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). > Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 > Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). > Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 > Nov 19 11:27:38 codered kernel: nfs_read_super: get root inode failed > Nov 19 11:27:38 codered kernel: nfs warning: mount version older than kernel > Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). > Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 > Nov 19 11:27:38 codered kernel: nfs_read_super: get root inode failed > Nov 19 11:27:38 codered automount[3623]: >> mount: wrong fs type, bad option, > bad superblock on aflac:/vol/vol1/cpd/nwd_usr, > Nov 19 11:27:38 codered automount[3623]: >> or too many mounted file > systems > Nov 19 11:27:38 codered automount[3623]: mount(nfs): nfs: mount failure > aflac:/vol/vol1/cpd/nwd_usr on /cpd/usr > Nov 19 11:27:38 codered automount[3623]: failed to mount /cpd/usr > ===== > > So whatever it is might be caused by the options? > These messages are due to starvation for reserved ports (< 1024). Specifically, the kernel will only use ports < 800. Currently, the kernel uses one port per nfs filesystem. If you mount filesystems very fast, then you can also run out of reserved ports as the local (mountd iirc?) will close tcp sessions and each must wait 2 minutes before being released. One solution is to try out the patch I posted last week that allows nfs mounts to share tcp/udp connections: http://marc.theaimsgroup.com/?l=linux-nfs&m=110261671705396&w=2 (against 2.6.9, although should apply to other kernels w/o too much difficulty) HTH, - -- Mike Waychison Sun Microsystems, Inc. 1 (650) 352-5299 voice 1 (416) 202-8336 voice ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NOTICE: The opinions expressed in this email are held by me, and may not represent the views of Sun Microsystems, Inc. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBv4HFdQs4kOxk3/MRAvoxAJ0XCN+p94P2SdOkm9GuLPpiA9CsOQCdEkk/ ljNrkssKI9W/O/QxGjFycYc= =GF6k -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Re: Intermittant problems mounting with automounter 2004-12-15 0:13 ` Mike Waychison @ 2004-12-15 15:02 ` Jeff Moyer 0 siblings, 0 replies; 5+ messages in thread From: Jeff Moyer @ 2004-12-15 15:02 UTC (permalink / raw) To: Mike Waychison; +Cc: autofs, David Meleedy ==> Regarding Re: [autofs] Re: Intermittant problems mounting with automounter; Mike Waychison <Michael.Waychison@sun.com> adds: Michael.Waychison> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael.Waychison> David Meleedy wrote: >> Well, in order to troubleshoot this issue further, autofs on Redhat >> Enterprise 3, update 3 >> >> I removed all options from the /etc/auto.master file: >> >> from: >> >> /cpd /etc/auto.cpd hard,intr,rsize=8192,wsize=8192 >> >> to: >> >> /cpd /etc/auto.cpd >> >> >> After this change, the following errors seemed to go away: >> >> ==== Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port >> (98). Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 >> Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). >> Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 Nov 19 >> 11:27:38 codered kernel: nfs_read_super: get root inode failed Nov 19 >> 11:27:38 codered kernel: nfs warning: mount version older than kernel >> Nov 19 11:27:38 codered kernel: RPC: Can't bind to reserved port (98). >> Nov 19 11:27:38 codered kernel: nfs_get_root: getattr error = 5 Nov 19 >> 11:27:38 codered kernel: nfs_read_super: get root inode failed Nov 19 >> 11:27:38 codered automount[3623]: >> mount: wrong fs type, bad option, >> bad superblock on aflac:/vol/vol1/cpd/nwd_usr, Nov 19 11:27:38 codered >> automount[3623]: >> or too many mounted file systems Nov 19 11:27:38 >> codered automount[3623]: mount(nfs): nfs: mount failure >> aflac:/vol/vol1/cpd/nwd_usr on /cpd/usr Nov 19 11:27:38 codered >> automount[3623]: failed to mount /cpd/usr ===== >> >> So whatever it is might be caused by the options? >> Michael.Waychison> These messages are due to starvation for reserved ports Michael.Waychison> (< 1024). Specifically, the kernel will only use ports Michael.Waychison> < 800. Currently, the kernel uses one port per nfs Michael.Waychison> filesystem. If you mount filesystems very fast, then Michael.Waychison> you can also run out of reserved ports as the local Michael.Waychison> (mountd iirc?) will close tcp sessions and each must Michael.Waychison> wait 2 minutes before being released. Michael.Waychison> One solution is to try out the patch I posted last week Michael.Waychison> that allows nfs mounts to share tcp/udp connections: Michael.Waychison> http://marc.theaimsgroup.com/?l=linux-nfs&m=110261671705396&w=2 Michael.Waychison> (against 2.6.9, although should apply to other kernels Michael.Waychison> w/o too much difficulty) Also note that we had a socket descriptor leak that could cause this problem. I also added code to pass a socket into clntudp_bufcreate so that we don't use up additional reserved ports. The individual patches have been posted to the list before, but I've included them below for convenience. -Jeff --- autofs-4.1.3/lib/rpc_subs.c~ 2004-09-14 10:32:30.000000000 -0400 +++ autofs-4.1.3/lib/rpc_subs.c 2004-09-14 10:36:29.766803120 -0400 @@ -48,7 +48,7 @@ struct conn_info { */ static CLIENT* create_udp_client(struct conn_info *info) { - int fd = -1; + int fd = RPC_ANYSOCK; CLIENT *client; struct sockaddr_in addr; struct hostent *hp; @@ -169,6 +169,7 @@ static CLIENT* create_tcp_client(struct if (!client) goto out_close; + clnt_control(client, CLSET_FD_CLOSE, NULL); return client; out_close: --- autofs-4.1.3/lib/rpc_subs.c.orig 2004-11-19 13:05:39.063691232 -0500 +++ autofs-4.1.3/lib/rpc_subs.c 2004-11-19 13:05:41.973248912 -0500 @@ -48,28 +48,50 @@ struct conn_info { */ static CLIENT* create_udp_client(struct conn_info *info) { - int fd = RPC_ANYSOCK; + int fd; CLIENT *client; - struct sockaddr_in addr; + struct sockaddr_in laddr, raddr; struct hostent *hp; if (info->proto->p_proto != IPPROTO_UDP) return NULL; - memset(&addr, 0, sizeof(addr)); + memset(&laddr, 0, sizeof(laddr)); + memset(&raddr, 0, sizeof(raddr)); hp = gethostbyname(info->host); if (!hp) return NULL; - addr.sin_family = AF_INET; - addr.sin_port = htons(info->port); - memcpy(&addr.sin_addr.s_addr, hp->h_addr, hp->h_length); + raddr.sin_family = AF_INET; + raddr.sin_port = htons(info->port); + memcpy(&raddr.sin_addr.s_addr, hp->h_addr, hp->h_length); + + /* + * bind to any unused port. If we left this up to the rpc + * layer, it would bind to a reserved port, which has been shown + * to exhaust the reserved port range in some situations. + */ + fd = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP); + if (fd < 0) + return NULL; + laddr.sin_family = AF_INET; + laddr.sin_port = 0; + laddr.sin_addr.s_addr = htonl(INADDR_ANY); + + if (bind(fd, (struct sockaddr *)&laddr, + sizeof(struct sockaddr_in)) < 0) { + close(fd); + fd = RPC_ANYSOCK; + /* FALLTHROUGH */ + } - client = clntudp_bufcreate(&addr, + client = clntudp_bufcreate(&raddr, info->program, info->version, info->timeout, &fd, info->send_sz, info->recv_sz); + if (client) + clnt_control(client, CLSET_FD_CLOSE, NULL); return client; } ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2004-12-15 15:02 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2004-11-30 20:07 Intermittant problems mounting with automounter David Meleedy 2004-12-14 14:10 ` Todd Denniston 2004-12-14 23:36 ` David Meleedy 2004-12-15 0:13 ` Mike Waychison 2004-12-15 15:02 ` Jeff Moyer
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.