rpc.mountd stops functioning

rpc.mountd stops functioning

[Johan van den Dorpe]

Hello,

For a long time we've been experiencing problems with rpc.mountd 
stopping functioning.

I'm finding it hard to pinpoint the precise circumstances where we 
encounter a problem, but here is what we have observed:

- A mount request, or a showmount -e, to the server will hang

- The server will print a log message authenticating the mount request

- The server doesn't print log messages that it has authenticated 
unmount requests.

- Killing rpc.mountd and then restarting it fixes the problem (I've not 
tried a HUP)

- There is a correlation between the number of mounts being handled 
(i.e. the number of entries in rmtab) and the frequency of the problem.

- The problem only occurs on servers that are mounted by a significant 
number of hosts (500-700).

- The more exports on the server, the more frequently the problem occurs.

- It should be noted that clients are mounting subdirectories of a 
single exported filesystem, so number of rmtab entries > number of xtab 
entries.

- Clearing the rmtab and restarting nfs (service nfs restart) seems to 
provide the longest time between failures.

-- 
Johan van den Dorpe


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[Johan van den Dorpe]

I clicked send a bit fast. Other info:

Running RHEL 3 with 2.4.25 vanilla kernel. nfs-utils-1.0.5

Experienced the problem in the past with Red Hat 7.3 with same 2.4.25 
kernel and 2.4.20 kernel + XFS patches + Trond's all patch for 2.4.20. 
Tried nfs-utils 0.3.3 and 1.0.6

Johan van den Dorpe wrote:
> Hello,
> 
> For a long time we've been experiencing problems with rpc.mountd 
> stopping functioning.
> 
> I'm finding it hard to pinpoint the precise circumstances where we 
> encounter a problem, but here is what we have observed:
> 
> - A mount request, or a showmount -e, to the server will hang
> 
> - The server will print a log message authenticating the mount request
> 
> - The server doesn't print log messages that it has authenticated 
> unmount requests.
> 
> - Killing rpc.mountd and then restarting it fixes the problem (I've not 
> tried a HUP)
> 
> - There is a correlation between the number of mounts being handled 
> (i.e. the number of entries in rmtab) and the frequency of the problem.
> 
> - The problem only occurs on servers that are mounted by a significant 
> number of hosts (500-700).
> 
> - The more exports on the server, the more frequently the problem occurs.
> 
> - It should be noted that clients are mounting subdirectories of a 
> single exported filesystem, so number of rmtab entries > number of xtab 
> entries.
> 
> - Clearing the rmtab and restarting nfs (service nfs restart) seems to 
> provide the longest time between failures.
> 


-- 
Johan van den Dorpe


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[Johan van den Dorpe]

I think I've got this sussed. After reading this thread:

http://sourceforge.net/mailarchive/message.php?msg_id=9891180
http://sourceforge.net/mailarchive/message.php?msg_id=9901077

I thought it might be worth testing the vanilla nfs-utils package from 
kernel.org rather than the redhat supplied versions.

Now, I've had this setup running for 28 days without any issues on one 
of our servers that has been a real problem in the past.

I'm going to try this solution on a wider scale to try and confirm this 
further, but for now I consider this the fix.

My advice to anyone running a vanilla kernel is to run the vanilla 
nfs-utils package and not a redhat nfs-utils package.

Johan van den Dorpe wrote:
> I clicked send a bit fast. Other info:
> 
> Running RHEL 3 with 2.4.25 vanilla kernel. nfs-utils-1.0.5
> 
> Experienced the problem in the past with Red Hat 7.3 with same 2.4.25 
> kernel and 2.4.20 kernel + XFS patches + Trond's all patch for 2.4.20. 
> Tried nfs-utils 0.3.3 and 1.0.6
> 
> Johan van den Dorpe wrote:
> 
>> Hello,
>>
>> For a long time we've been experiencing problems with rpc.mountd 
>> stopping functioning.
>>
>> I'm finding it hard to pinpoint the precise circumstances where we 
>> encounter a problem, but here is what we have observed:
>>
>> - A mount request, or a showmount -e, to the server will hang
>>
>> - The server will print a log message authenticating the mount request
>>
>> - The server doesn't print log messages that it has authenticated 
>> unmount requests.
>>
>> - Killing rpc.mountd and then restarting it fixes the problem (I've 
>> not tried a HUP)
>>
>> - There is a correlation between the number of mounts being handled 
>> (i.e. the number of entries in rmtab) and the frequency of the problem.
>>
>> - The problem only occurs on servers that are mounted by a significant 
>> number of hosts (500-700).
>>
>> - The more exports on the server, the more frequently the problem occurs.
>>
>> - It should be noted that clients are mounting subdirectories of a 
>> single exported filesystem, so number of rmtab entries > number of 
>> xtab entries.
>>
>> - Clearing the rmtab and restarting nfs (service nfs restart) seems to 
>> provide the longest time between failures.
>>
> 
> 


-- 
Johan van den Dorpe


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[J. Bruce Fields]

On Thu, Aug 05, 2004 at 12:19:26PM +0100, Johan van den Dorpe wrote:
> For a long time we've been experiencing problems with rpc.mountd 
> stopping functioning.

Maybe you need the following?

(Neil or someone, could you please apply this?  Last I checked it wasn't
in nfs-utils cvs.)

--Bruce Fields

>From Garrick Staples <garrick@usc.edu>:

After mountd handles a cache upcall, we should clear the relevant bits in the
fd_set.

---

 nfs-utils-1.0.6-bfields/utils/mountd/cache.c |    1 +
 1 files changed, 1 insertion(+)

diff -puN utils/mountd/cache.c~cache_select_bugfix utils/mountd/cache.c
--- nfs-utils-1.0.6/utils/mountd/cache.c~cache_select_bugfix	2004-07-14 12:52:57.000000000 -0400
+++ nfs-utils-1.0.6-bfields/utils/mountd/cache.c	2004-07-14 12:52:57.000000000 -0400
@@ -315,6 +315,7 @@ int cache_process_req(fd_set *readfds) 
 		    FD_ISSET(fileno(cachelist[i].f), readfds)) {
 			cnt++;
 			cachelist[i].cache_handle(cachelist[i].f);
+			FD_CLR(fileno(cachelist[i].f), readfds);
 		}
 	}
 	return cnt;
_


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[Johan van den Dorpe]

J. Bruce Fields wrote:
> On Thu, Aug 05, 2004 at 12:19:26PM +0100, Johan van den Dorpe wrote:
> 
>>For a long time we've been experiencing problems with rpc.mountd 
>>stopping functioning.
> 
> 
> Maybe you need the following?
> 
> (Neil or someone, could you please apply this?  Last I checked it wasn't
> in nfs-utils cvs.)
> 
> --Bruce Fields
> 
> From Garrick Staples <garrick@usc.edu>:
> 
> After mountd handles a cache upcall, we should clear the relevant bits in the
> fd_set.

I've installed this patch and I'm testing it at the moment. It'll be 
some time before I can say whether this has worked.

So far, everythings been fine.

Thanks for your help,

-- 
Johan van den Dorpe


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[Johan van den Dorpe]

J. Bruce Fields wrote:
> On Thu, Aug 05, 2004 at 12:19:26PM +0100, Johan van den Dorpe wrote:
> 
>>For a long time we've been experiencing problems with rpc.mountd 
>>stopping functioning.
> 
> 
> Maybe you need the following?
> 
> (Neil or someone, could you please apply this?  Last I checked it wasn't
> in nfs-utils cvs.)
> 
> --Bruce Fields
> 
> From Garrick Staples <garrick@usc.edu>:
> 
> After mountd handles a cache upcall, we should clear the relevant bits in the
> fd_set.

No joy.

After a touch over 10 days of perfect service the problem reoccured today.

We've been graphing the number of mounts in rmtab and the number 
steadily rose to ~3100 before the problem reoccured.

Are there any other ideas out there?

-- 
Johan van den Dorpe


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[Neil Brown]

On Thursday August 5, bfields@fieldses.org wrote:
> On Thu, Aug 05, 2004 at 12:19:26PM +0100, Johan van den Dorpe wrote:
> > For a long time we've been experiencing problems with rpc.mountd 
> > stopping functioning.
> 
> Maybe you need the following?
> 
> (Neil or someone, could you please apply this?  Last I checked it wasn't
> in nfs-utils cvs.)

I don't think this patch is needed (though it wouldn't actually break anything).
The loop in svc_run.c starts:


		readfds = svc_fdset;
		cache_set_fds(&readfds);

so readfds is completely initialised at the top of the loop, so
clearing something at the bottom (which is essentially what this patch
does) should be a no-op.

Am I wrong?

NeilBrown

> 
> --Bruce Fields
> 
> >From Garrick Staples <garrick@usc.edu>:
> 
> After mountd handles a cache upcall, we should clear the relevant bits in the
> fd_set.
> 
> ---
> 
>  nfs-utils-1.0.6-bfields/utils/mountd/cache.c |    1 +
>  1 files changed, 1 insertion(+)
> 
> diff -puN utils/mountd/cache.c~cache_select_bugfix utils/mountd/cache.c
> --- nfs-utils-1.0.6/utils/mountd/cache.c~cache_select_bugfix	2004-07-14 12:52:57.000000000 -0400
> +++ nfs-utils-1.0.6-bfields/utils/mountd/cache.c	2004-07-14 12:52:57.000000000 -0400
> @@ -315,6 +315,7 @@ int cache_process_req(fd_set *readfds) 
>  		    FD_ISSET(fileno(cachelist[i].f), readfds)) {
>  			cnt++;
>  			cachelist[i].cache_handle(cachelist[i].f);
> +			FD_CLR(fileno(cachelist[i].f), readfds);
>  		}
>  	}
>  	return cnt;
> _
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> one more big change to announce. We are now OSTG- Open Source Technology
> Group. Come see the changes on the new OSTG site. www.ostg.com
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[J. Bruce Fields]

On Fri, Aug 27, 2004 at 01:31:20PM +1000, Neil Brown wrote:
> I don't think this patch is needed (though it wouldn't actually break anything).
> The loop in svc_run.c starts:
> 
> 
> 		readfds = svc_fdset;
> 		cache_set_fds(&readfds);
> 
> so readfds is completely initialised at the top of the loop, so
> clearing something at the bottom (which is essentially what this patch
> does) should be a no-op.
> 
> Am I wrong?

Note that cache_process_req() (which the patch below modifies) is not
all the way at the bottom of the loop; there's still an importan call to
svc_getreqset() after it.

If I understand the problem correctly, what happens is that if select()
returns with a set that includes two file descriptors, the first for a
cache channel and the second for an rpc socket, and if we don't clear
the first file descriptor from that set before passing it to
svc_getreqset(), then the rpc code will try to read an rpc request from
a cache channel file, and will get very confused.

The symptom is that mountd may die occasionally on a server that's
using the new interface (hence mountd is handling upcalls), and that's
getting a lot of mount requests.

I've never seen the bug myself, because I'm almost always using nfsv4,
hence only use mountd to handle upcalls....

--b.


> > >From Garrick Staples <garrick@usc.edu>:
> > 
> > After mountd handles a cache upcall, we should clear the relevant bits in the
> > fd_set.
> > 
> > ---
> > 
> >  nfs-utils-1.0.6-bfields/utils/mountd/cache.c |    1 +
> >  1 files changed, 1 insertion(+)
> > 
> > diff -puN utils/mountd/cache.c~cache_select_bugfix utils/mountd/cache.c
> > --- nfs-utils-1.0.6/utils/mountd/cache.c~cache_select_bugfix	2004-07-14 12:52:57.000000000 -0400
> > +++ nfs-utils-1.0.6-bfields/utils/mountd/cache.c	2004-07-14 12:52:57.000000000 -0400
> > @@ -315,6 +315,7 @@ int cache_process_req(fd_set *readfds) 
> >  		    FD_ISSET(fileno(cachelist[i].f), readfds)) {
> >  			cnt++;
> >  			cachelist[i].cache_handle(cachelist[i].f);
> > +			FD_CLR(fileno(cachelist[i].f), readfds);
> >  		}
> >  	}
> >  	return cnt;
> > _
> > 
> > 
> > -------------------------------------------------------
> > This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> > Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> > one more big change to announce. We are now OSTG- Open Source Technology
> > Group. Come see the changes on the new OSTG site. www.ostg.com
> > _______________________________________________
> > NFS maillist  -  NFS@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: rpc.mountd stops functioning

[Garrick Staples]

[-- Attachment #1: Type: text/plain, Size: 1853 bytes --]

On Fri, Aug 27, 2004 at 07:16:54PM -0400, J. Bruce Fields alleged:
> On Fri, Aug 27, 2004 at 01:31:20PM +1000, Neil Brown wrote:
> > I don't think this patch is needed (though it wouldn't actually break anything).
> > The loop in svc_run.c starts:
> > 
> > 
> > 		readfds = svc_fdset;
> > 		cache_set_fds(&readfds);
> > 
> > so readfds is completely initialised at the top of the loop, so
> > clearing something at the bottom (which is essentially what this patch
> > does) should be a no-op.
> > 
> > Am I wrong?
> 
> Note that cache_process_req() (which the patch below modifies) is not
> all the way at the bottom of the loop; there's still an importan call to
> svc_getreqset() after it.
> 
> If I understand the problem correctly, what happens is that if select()
> returns with a set that includes two file descriptors, the first for a
> cache channel and the second for an rpc socket, and if we don't clear
> the first file descriptor from that set before passing it to
> svc_getreqset(), then the rpc code will try to read an rpc request from
> a cache channel file, and will get very confused.
> 
> The symptom is that mountd may die occasionally on a server that's
> using the new interface (hence mountd is handling upcalls), and that's
> getting a lot of mount requests.

Yes, this is exactly what happens.  The RPC code reading from the cache channel
file segfaults mountd every time.

On my two main fileservers, mountd would die every few hours.  Sometimes,
conditions being optimal to trigger the bug, it would chain-crash.  It hasn't
segfaulted at all since that one line patch.

I'm surprised more people haven't reported this segfault.  I guess not many
people are using the new interface in heavy production.

-- 
Garrick Staples, Linux/HPCC Administrator
University of Southern California

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: rpc.mountd stops functioning

[Neil Brown]

On Friday August 27, bfields@fieldses.org wrote:
> On Fri, Aug 27, 2004 at 01:31:20PM +1000, Neil Brown wrote:
> > I don't think this patch is needed (though it wouldn't actually bre=
ak anything).
> > The loop in svc_run.c starts:
> >=20
> >=20
> > =09=09readfds =3D svc_fdset;
> > =09=09cache_set_fds(&readfds);
> >=20
> > so readfds is completely initialised at the top of the loop, so
> > clearing something at the bottom (which is essentially what this pa=
tch
> > does) should be a no-op.
> >=20
> > Am I wrong?
>=20
> Note that cache_process_req() (which the patch below modifies) is not=

> all the way at the bottom of the loop; there's still an importan call=
 to
> svc_getreqset() after it.
>=20
> If I understand the problem correctly, what happens is that if select=
()
> returns with a set that includes two file descriptors, the first for =
a
> cache channel and the second for an rpc socket, and if we don't clear=

> the first file descriptor from that set before passing it to
> svc_getreqset(), then the rpc code will try to read an rpc request fr=
om
> a cache channel file, and will get very confused.

Ahh, I get it now, thanks.

This is really a bug in glibc.   A relevant piece of code reads:
  xprt =3D xports[fd];
  /* Do we control fd? */
  if (xprt =3D=3D NULL)
     return;

so unknown file descriptors should be ignored.  However xports is
allocated thus:
  if (xports =3D=3D NULL)
    {
      xports =3D (SVCXPRT **) malloc (_rpc_dtablesize () * sizeof (SVCX=
PRT *));
      if (xports =3D=3D NULL) /* Don=B4t add handle */
=09return;
    }

and so is not NULL-filled !!

In any case, you suggested change is good defensive programming and I
will add it.

Thanks.
NeilBrown



-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs