Is order for "exportfs -r" and rpc.mountd important?

Is order for "exportfs -r" and rpc.mountd important?

[Dr. Tilmann Bubeck]

According to manual page "exportfs -r" has two modes: legacy and new. Fedora 
10 seems to use the new mode and mounts /proc/fs/nfsd.

But they use the following order in /etc/init.d/nfs:
  exportfs -r
  rpc.mountd

This results in /proc/fs/nfsd/exports having all obsolete entries 
from /var/lib/nfs/rmtab which causes a lot of trouble with stale NFS file 
handles. This seems to be wrong according to "man exportfs" in new mode.

If I change the order to
  rpc.mountd
  exportfs -r

then everything is OK and /proc/fs/nfsd/exports only contains the info 
from /etc/exports and not all stale clients. This seems to be correct 
according to "man exportfs" in new mode.

Questions: is this behaviour a bug or at least bad? Shouldn't exportfs being 
changed, so that when it detects "new" mode it will never feed rmtab into the 
kernel and rely upon rpc.mountd answering the kernels questions as stated 
in "man exportfs"?

If yes, then the order or the programs is irrelevant.
If not, should the order of the programs be reversed in Fedora?

Thanks
 Till


-- 
+-------+-------------------------------------------------------------+
|       | dr. tilmann bubeck               reinform medien- und       |
|       |                                  informationstechnologie AG |
| rein  | fon  : +49 (711) 7 82 76-52      loeffelstr. 40             |
| form  | fax  : +49 (711) 7 82 76-46      70597 stuttgart / germany  |
|    AG | cell.: +49 (172) 8 84 29 72      fon: +49 (711) 75 86 56-10 |
|       | email: t.bubeck-dx1aEnP5Zukb1SvskN2V4Q@public.gmane.org      http://www.reinform.de     |
|       +-------------------------------------------------------------+
|       | pflichtangaben nach paragraph 80, AktG:                     |
|       | reinform medien- und informationstechnologie AG, stuttgart  |
|       | handelsregister stuttgart, HRB 23001                        |
|       | vorstand:     dr. tilmann bubeck (vorsitz), dr. tommy kuhn  |
|       | aufsichtsrat: frank stege (vorsitz)                         |
+-------+-------------------------------------------------------------+

Re: Is order for "exportfs -r" and rpc.mountd important?

[Dr. Tilmann Bubeck]

To be more precise: the order between "exportfs -r" and "modprobe nfsd" is 
important (and not "exportfs -r" and "rpc.mountd").

Am Freitag, 28. November 2008 15:41:38 schrieb Dr. Tilmann Bubeck:
> According to manual page "exportfs -r" has two modes: legacy and new.
> Fedora 10 seems to use the new mode and mounts /proc/fs/nfsd.
>
> But they use the following order in /etc/init.d/nfs:
>   exportfs -r
>   rpc.mountd
>
> This results in /proc/fs/nfsd/exports having all obsolete entries
> from /var/lib/nfs/rmtab which causes a lot of trouble with stale NFS file
> handles. This seems to be wrong according to "man exportfs" in new mode.
>
> If I change the order to
>   rpc.mountd
>   exportfs -r
>
> then everything is OK and /proc/fs/nfsd/exports only contains the info
> from /etc/exports and not all stale clients. This seems to be correct
> according to "man exportfs" in new mode.
>
> Questions: is this behaviour a bug or at least bad? Shouldn't exportfs
> being changed, so that when it detects "new" mode it will never feed rmtab
> into the kernel and rely upon rpc.mountd answering the kernels questions as
> stated in "man exportfs"?
>
> If yes, then the order or the programs is irrelevant.
> If not, should the order of the programs be reversed in Fedora?
>
> Thanks
>  Till



-- 
+-------+-------------------------------------------------------------+
|       | dr. tilmann bubeck               reinform medien- und       |
|       |                                  informationstechnologie AG |
| rein  | fon  : +49 (711) 7 82 76-52      loeffelstr. 40             |
| form  | fax  : +49 (711) 7 82 76-46      70597 stuttgart / germany  |
|    AG | cell.: +49 (172) 8 84 29 72      fon: +49 (711) 75 86 56-10 |
|       | email: t.bubeck-dx1aEnP5Zukb1SvskN2V4Q@public.gmane.org      http://www.reinform.de     |
|       +-------------------------------------------------------------+
|       | pflichtangaben nach paragraph 80, AktG:                     |
|       | reinform medien- und informationstechnologie AG, stuttgart  |
|       | handelsregister stuttgart, HRB 23001                        |
|       | vorstand:     dr. tilmann bubeck (vorsitz), dr. tommy kuhn  |
|       | aufsichtsrat: frank stege (vorsitz)                         |
+-------+-------------------------------------------------------------+

Re: Is order for "exportfs -r" and rpc.mountd important?

[J. Bruce Fields]

On Fri, Nov 28, 2008 at 05:05:44PM +0100, Dr. Tilmann Bubeck wrote:
> To be more precise: the order between "exportfs -r" and "modprobe nfsd" is 
> important (and not "exportfs -r" and "rpc.mountd").

So, I'm confused: what are the before-and-after orders of all three
(modprobe, exportfs, and rpc.mountd)?

> 
> Am Freitag, 28. November 2008 15:41:38 schrieb Dr. Tilmann Bubeck:
> > According to manual page "exportfs -r" has two modes: legacy and new.
> > Fedora 10 seems to use the new mode and mounts /proc/fs/nfsd.
> >
> > But they use the following order in /etc/init.d/nfs:
> >   exportfs -r
> >   rpc.mountd
> >
> > This results in /proc/fs/nfsd/exports having all obsolete entries
> > from /var/lib/nfs/rmtab which causes a lot of trouble with stale NFS file
> > handles.

What problems exactly?

> > This seems to be wrong according to "man exportfs" in new mode.
> >
> > If I change the order to
> >   rpc.mountd
> >   exportfs -r
> >
> > then everything is OK and /proc/fs/nfsd/exports only contains the info
> > from /etc/exports and not all stale clients. This seems to be correct
> > according to "man exportfs" in new mode.
> >
> > Questions: is this behaviour a bug or at least bad? Shouldn't exportfs
> > being changed, so that when it detects "new" mode it will never feed rmtab
> > into the kernel and rely upon rpc.mountd answering the kernels questions as
> > stated in "man exportfs"?

That does sound wrong.

Perhaps rmtab should also be cleared on exportfs -r?  I don't really
understand how rmtab should work.

--b.

> >
> > If yes, then the order or the programs is irrelevant.
> > If not, should the order of the programs be reversed in Fedora?
> >
> > Thanks
> >  Till
> 
> 
> 
> -- 
> +-------+-------------------------------------------------------------+
> |       | dr. tilmann bubeck               reinform medien- und       |
> |       |                                  informationstechnologie AG |
> | rein  | fon  : +49 (711) 7 82 76-52      loeffelstr. 40             |
> | form  | fax  : +49 (711) 7 82 76-46      70597 stuttgart / germany  |
> |    AG | cell.: +49 (172) 8 84 29 72      fon: +49 (711) 75 86 56-10 |
> |       | email: t.bubeck-dx1aEnP5Zukb1SvskN2V4Q@public.gmane.org      http://www.reinform.de     |
> |       +-------------------------------------------------------------+
> |       | pflichtangaben nach paragraph 80, AktG:                     |
> |       | reinform medien- und informationstechnologie AG, stuttgart  |
> |       | handelsregister stuttgart, HRB 23001                        |
> |       | vorstand:     dr. tilmann bubeck (vorsitz), dr. tommy kuhn  |
> |       | aufsichtsrat: frank stege (vorsitz)                         |
> +-------+-------------------------------------------------------------+
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is order for "exportfs -r" and rpc.mountd important?

[Dr. Tilmann Bubeck]

Sorry for the confusion. Maybe I can make myself clearer this time:

Fedora 10 uses the following simplified /etc/init.d/nfs to start the NFS 
server:

  1. exportfs -r
  2. modprobe nfsd
     This also does (see modprobe.conf.dist): mount -t nfsd nfsd /proc/fs/nfsd

As you can read in "man exportfs" there are two modes: "legacy" and "new"
depending on the existance of /proc/fs/nfsd. The above version of
/etc/init.d/nfs executes exportfs in lecacy mode and therefore feeds
/var/lib/nfs/rmtab into the kernel. After "mount -t nfsd ..." the "new" mode 
is used. In this case the previously fed rmtab is disturbing because we have 
a mix of "legacy" and "new" mode. This results in the problem that some NFS 
clients are receiving a "Stale NFS file handle" after a server reboot.

I tracked the problem down to the kernel based export table which is found 
in /proc/fs/nfsd/exports. The above start order results in:

root# cat /proc/fs/nfsd/exports  | grep dhcp-10-1-1-209
/home   
dhcp-10-1-1-209.loef.reinform.de(rw,root_squash,sync,wdelay,no_subtree_check,fsid=1510)

which results from "exportfs -r" reading rmtab and feeding into the kernel.

By changing the start order to:

  1. modprobe nfsd
     This also does (see modprobe.conf.dist): mount -t nfsd nfsd /proc/fs/nfsd
  2. exportfs -r

then "exportfs -r" starts in "new" mode and does not feed the rmtab into the 
kernel. Therefore the kernel export table is:

root# cat /proc/fs/nfsd/exports 
# Version 1.1
# Path Client(Flags) # IPs
/home   10.0.0.0/8
(rw,root_squash,sync,wdelay,no_subtree_check,fsid=1510,uuid=bd6c73cb:12de4994:8aaba0c5:cc82b618)

This does not give a stale NFS file handle on the client. If the uuid is 
missing, it results in stale NFS file handles.

The problem could be healed by issuing "exportfs -f" as a workaround to clear 
the kernel export tabel. But this is not a good solution, because all NFS 
clients already had a "Stale NFS file handle". But they reconnect correctly 
after "exportfs -f".

First I want to make sure, that the above is indeed correct. I successfully 
verified this on my own systems.

If you think it is correct, then someone should probably fix the "legacy" mode 
of "exportfs -r" to include/generate a uuid or fix the kernel to generate it 
accordingly to the new mode. Maybe something better could be found.

I do think, that the two modes "lecacy" and "new" should be made in a way, 
that they do not interfere with each other. Otherwise the manual pages or 
distributions should be told very clear how to avoid the problem.

I'm glad for any reponse to the above ideas.

Till


Am Mittwoch, 3. Dezember 2008 18:25:18 schrieb J. Bruce Fields:
> On Fri, Nov 28, 2008 at 05:05:44PM +0100, Dr. Tilmann Bubeck wrote:
> > To be more precise: the order between "exportfs -r" and "modprobe nfsd"
> > is important (and not "exportfs -r" and "rpc.mountd").
>
> So, I'm confused: what are the before-and-after orders of all three
> (modprobe, exportfs, and rpc.mountd)?
>
> > Am Freitag, 28. November 2008 15:41:38 schrieb Dr. Tilmann Bubeck:
> > > According to manual page "exportfs -r" has two modes: legacy and new.
> > > Fedora 10 seems to use the new mode and mounts /proc/fs/nfsd.
> > >
> > > But they use the following order in /etc/init.d/nfs:
> > >   exportfs -r
> > >   rpc.mountd
> > >
> > > This results in /proc/fs/nfsd/exports having all obsolete entries
> > > from /var/lib/nfs/rmtab which causes a lot of trouble with stale NFS
> > > file handles.
>
> What problems exactly?
>
> > > This seems to be wrong according to "man exportfs" in new mode.
> > >
> > > If I change the order to
> > >   rpc.mountd
> > >   exportfs -r
> > >
> > > then everything is OK and /proc/fs/nfsd/exports only contains the info
> > > from /etc/exports and not all stale clients. This seems to be correct
> > > according to "man exportfs" in new mode.
> > >
> > > Questions: is this behaviour a bug or at least bad? Shouldn't exportfs
> > > being changed, so that when it detects "new" mode it will never feed
> > > rmtab into the kernel and rely upon rpc.mountd answering the kernels
> > > questions as stated in "man exportfs"?
>
> That does sound wrong.
>
> Perhaps rmtab should also be cleared on exportfs -r?  I don't really
> understand how rmtab should work.
>
> --b.
>
> > > If yes, then the order or the programs is irrelevant.
> > > If not, should the order of the programs be reversed in Fedora?
> > >
> > > Thanks
> > >  Till
> >
> > --
> > +-------+-------------------------------------------------------------+
> >
> > |       | dr. tilmann bubeck               reinform medien- und       |
> > |       |                                  informationstechnologie AG |
> > |
> > | rein  | fon  : +49 (711) 7 82 76-52      loeffelstr. 40             |
> > | form  | fax  : +49 (711) 7 82 76-46      70597 stuttgart / germany  |
> > |    AG | cell.: +49 (172) 8 84 29 72      fon: +49 (711) 75 86 56-10 |
> > |
> > |       | email: t.bubeck-dx1aEnP5Zukb1SvskN2V4Q@public.gmane.org      http://www.reinform.de     |
> > |
> > |       +-------------------------------------------------------------+
> > |
> > |       | pflichtangaben nach paragraph 80, AktG:                     |
> > |       | reinform medien- und informationstechnologie AG, stuttgart  |
> > |       | handelsregister stuttgart, HRB 23001                        |
> > |       | vorstand:     dr. tilmann bubeck (vorsitz), dr. tommy kuhn  |
> > |       | aufsichtsrat: frank stege (vorsitz)                         |
> >
> > +-------+-------------------------------------------------------------+
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
+-------+-------------------------------------------------------------+
|       | dr. tilmann bubeck               reinform medien- und       |
|       |                                  informationstechnologie AG |
| rein  | fon  : +49 (711) 7 82 76-52      loeffelstr. 40             |
| form  | fax  : +49 (711) 7 82 76-46      70597 stuttgart / germany  |
|    AG | cell.: +49 (172) 8 84 29 72      fon: +49 (711) 75 86 56-10 |
|       | email: t.bubeck-dx1aEnP5Zukb1SvskN2V4Q@public.gmane.org      http://www.reinform.de     |
|       +-------------------------------------------------------------+
|       | pflichtangaben nach paragraph 80, AktG:                     |
|       | reinform medien- und informationstechnologie AG, stuttgart  |
|       | handelsregister stuttgart, HRB 23001                        |
|       | vorstand:     dr. tilmann bubeck (vorsitz), dr. tommy kuhn  |
|       | aufsichtsrat: frank stege (vorsitz)                         |
+-------+-------------------------------------------------------------+

Re: Is order for "exportfs -r" and rpc.mountd important?

[Steve Dickson]

Dr. Tilmann Bubeck wrote:
> Sorry for the confusion. Maybe I can make myself clearer this time:
> 
> Fedora 10 uses the following simplified /etc/init.d/nfs to start the NFS 
> server:
> 
>   1. exportfs -r
>   2. modprobe nfsd
>      This also does (see modprobe.conf.dist): mount -t nfsd nfsd /proc/fs/nfsd
> 
> As you can read in "man exportfs" there are two modes: "legacy" and "new"
> depending on the existance of /proc/fs/nfsd. The above version of
> /etc/init.d/nfs executes exportfs in lecacy mode and therefore feeds
> /var/lib/nfs/rmtab into the kernel. After "mount -t nfsd ..." the "new" mode 
> is used. In this case the previously fed rmtab is disturbing because we have 
> a mix of "legacy" and "new" mode. This results in the problem that some NFS 
> clients are receiving a "Stale NFS file handle" after a server reboot.
> 
> I tracked the problem down to the kernel based export table which is found 
> in /proc/fs/nfsd/exports. The above start order results in:
> 
> root# cat /proc/fs/nfsd/exports  | grep dhcp-10-1-1-209
> /home   
> dhcp-10-1-1-209.loef.reinform.de(rw,root_squash,sync,wdelay,no_subtree_check,fsid=1510)
> 
> which results from "exportfs -r" reading rmtab and feeding into the kernel.
> 
> By changing the start order to:
> 
>   1. modprobe nfsd
>      This also does (see modprobe.conf.dist): mount -t nfsd nfsd /proc/fs/nfsd
>   2. exportfs -r
> 
> then "exportfs -r" starts in "new" mode and does not feed the rmtab into the 
> kernel. 
This has been fixed in both Fedora 9 and 10... The update nfs-utils version
should be hitting a yum repository near you soon... 

steved.

Re: Is order for "exportfs -r" and rpc.mountd important?

[Dr. Tilmann Bubeck]

It's very good that it is fixed in Fedora (my preferred distribution), but it 
is still wrong in SuSe (verified) and probably more.

A better way would be to fix NFS and its utilities itself instead of fixing 
the distribution. This means, that the way the kernel and nfs-utils interact 
should be made more robust (see my posting before).

Till



Am Donnerstag, 4. Dezember 2008 14:59:31 schrieb Steve Dickson:
> Dr. Tilmann Bubeck wrote:
> > Sorry for the confusion. Maybe I can make myself clearer this time:
> >
> > Fedora 10 uses the following simplified /etc/init.d/nfs to start the NFS
> > server:
> >
> >   1. exportfs -r
> >   2. modprobe nfsd
> >      This also does (see modprobe.conf.dist): mount -t nfsd nfsd
> > /proc/fs/nfsd
> >
> > As you can read in "man exportfs" there are two modes: "legacy" and "new"
> > depending on the existance of /proc/fs/nfsd. The above version of
> > /etc/init.d/nfs executes exportfs in lecacy mode and therefore feeds
> > /var/lib/nfs/rmtab into the kernel. After "mount -t nfsd ..." the "new"
> > mode is used. In this case the previously fed rmtab is disturbing because
> > we have a mix of "legacy" and "new" mode. This results in the problem
> > that some NFS clients are receiving a "Stale NFS file handle" after a
> > server reboot.
> >
> > I tracked the problem down to the kernel based export table which is
> > found in /proc/fs/nfsd/exports. The above start order results in:
> >
> > root# cat /proc/fs/nfsd/exports  | grep dhcp-10-1-1-209
> > /home
> > dhcp-10-1-1-209.loef.reinform.de(rw,root_squash,sync,wdelay,no_subtree_ch
> >eck,fsid=1510)
> >
> > which results from "exportfs -r" reading rmtab and feeding into the
> > kernel.
> >
> > By changing the start order to:
> >
> >   1. modprobe nfsd
> >      This also does (see modprobe.conf.dist): mount -t nfsd nfsd
> > /proc/fs/nfsd 2. exportfs -r
> >
> > then "exportfs -r" starts in "new" mode and does not feed the rmtab into
> > the kernel.
>
> This has been fixed in both Fedora 9 and 10... The update nfs-utils version
> should be hitting a yum repository near you soon...
>
> steved.



-- 
+-------+-------------------------------------------------------------+
|       | dr. tilmann bubeck               reinform medien- und       |
|       |                                  informationstechnologie AG |
| rein  | fon  : +49 (711) 7 82 76-52      loeffelstr. 40             |
| form  | fax  : +49 (711) 7 82 76-46      70597 stuttgart / germany  |
|    AG | cell.: +49 (172) 8 84 29 72      fon: +49 (711) 75 86 56-10 |
|       | email: t.bubeck-dx1aEnP5Zukb1SvskN2V4Q@public.gmane.org      http://www.reinform.de     |
|       +-------------------------------------------------------------+
|       | pflichtangaben nach paragraph 80, AktG:                     |
|       | reinform medien- und informationstechnologie AG, stuttgart  |
|       | handelsregister stuttgart, HRB 23001                        |
|       | vorstand:     dr. tilmann bubeck (vorsitz), dr. tommy kuhn  |
|       | aufsichtsrat: frank stege (vorsitz)                         |
+-------+-------------------------------------------------------------+

Re: Is order for "exportfs -r" and rpc.mountd important?

[J. Bruce Fields]

On Thu, Dec 04, 2008 at 03:16:13PM +0100, Dr. Tilmann Bubeck wrote:
> It's very good that it is fixed in Fedora (my preferred distribution), but it 
> is still wrong in SuSe (verified) and probably more.
> 
> A better way would be to fix NFS and its utilities itself instead of fixing 
> the distribution. This means, that the way the kernel and nfs-utils interact 
> should be made more robust (see my posting before).

The problem of exportfs ignoring the uuid, at least, should be easy
enough to fix.

I'd start by looking for the relevant code in nfs-utils/utils/exportfs/
that passes the export entry down to the kernel and figure out what
mountd is doing that it's not.

--b.

Re: Is order for "exportfs -r" and rpc.mountd important?

[Steve Dickson]

J. Bruce Fields wrote:
> On Thu, Dec 04, 2008 at 03:16:13PM +0100, Dr. Tilmann Bubeck wrote:
>> It's very good that it is fixed in Fedora (my preferred distribution), but it 
>> is still wrong in SuSe (verified) and probably more.
>>
>> A better way would be to fix NFS and its utilities itself instead of fixing 
>> the distribution. This means, that the way the kernel and nfs-utils interact 
>> should be made more robust (see my posting before).
> 
> The problem of exportfs ignoring the uuid, at least, should be easy
> enough to fix.
> 
> I'd start by looking for the relevant code in nfs-utils/utils/exportfs/
> that passes the export entry down to the kernel and figure out what
> mountd is doing that it's not.

FYI... I created a bz (https://bugzilla.linux-nfs.org/show_bug.cgi?id=171) to 
track this problem

steved.

Re: Is order for "exportfs -r" and rpc.mountd important?

[Neil Brown]

On Thursday December 4, t.bubeck-dx1aEnP5Zukb1SvskN2V4Q@public.gmane.org wrote:
> It's very good that it is fixed in Fedora (my preferred distribution), but it 
> is still wrong in SuSe (verified) and probably more.

Could you be more specific about SuSe please?  I just checked 11.1,
11.0, 10.3, SLES10sp2 and SLES9sp4 and they all look like they have it
in the right order.

> 
> A better way would be to fix NFS and its utilities itself instead of fixing 
> the distribution. This means, that the way the kernel and nfs-utils interact 
> should be made more robust (see my posting before).

Each distro has the its own init scripts.  The best that we as upstream
can do is document what is required.
The fact that "mount -t nfsd /proc/fs/nfsd" should be first on the
server is very clear from the README file distributed with nfs-utils.

Can you be more precise about what you suggest could be fixed in NFS
and its utilities?

You said in the original email:

> Questions: is this behaviour a bug or at least bad? Shouldn't exportfs being 
> changed, so that when it detects "new" mode it will never feed rmtab into the 
> kernel and rely upon rpc.mountd answering the kernels questions as stated 
> in "man exportfs"?

I believe that when exportfs detects "new" mode it will never feed
rmtab into the kernel.  So there is nothing to be fixed there.

Confused?

Thanks,
NeilBrown