nfs-utils crossmnt/fsid bug

nfs-utils crossmnt/fsid bug

[J. Bruce Fields]

With an exports entry like

	/exports *.citi.umich.edu(rw,crossmnt,fsid=0)

and some other stuff mounted under /exports, I get an odd situation
where listing /exports/submnt/ shows me the contents of /exports/.

I looks to me like what's happening is that the new code to
automatically export filesystems under crossmnt exports is exporting
those directories with the fsid option turned on, with the result that
the root of each of those subdirectories ends up with the same
filehandle as the root of /exports, and the hilarity ensues.  But I
haven't tried a patch yet.

Does that make sense?  I assume we should just clear the fsid flag on
submounts, as it's never going to make sense, and leave the logic
otherwise intact.

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[Neil Brown]

On Wednesday September 5, bfields@fieldses.org wrote:
> With an exports entry like
> 
> 	/exports *.citi.umich.edu(rw,crossmnt,fsid=0)
> 
> and some other stuff mounted under /exports, I get an odd situation
> where listing /exports/submnt/ shows me the contents of /exports/.
> 
> I looks to me like what's happening is that the new code to
> automatically export filesystems under crossmnt exports is exporting
> those directories with the fsid option turned on, with the result that
> the root of each of those subdirectories ends up with the same
> filehandle as the root of /exports, and the hilarity ensues.  But I
> haven't tried a patch yet.
> 
> Does that make sense?  I assume we should just clear the fsid flag on
> submounts, as it's never going to make sense, and leave the logic
> otherwise intact.
> 
> --b.

Thanks for noticing and reporting this.

Does this (untested, but it compiles) patch seem right?

NeilBrown

--------
When exported a filesystems with option inherited (by the crossmnt
option) from a higherlevel filesystem, ignore filesystem specific
options like FSID and explicit UUID.

Signed-off-by: NeilBrown <neilb@suse.de>

diff --git a/utils/mountd/cache.c b/utils/mountd/cache.c
index fd317cd..ff56b9f 100644
--- a/utils/mountd/cache.c
+++ b/utils/mountd/cache.c
@@ -565,20 +565,29 @@ static int dump_to_cache(FILE *f, char *domain, char *path, struct exportent *ex
 	qword_print(f, path);
 	qword_printint(f, time(0)+30*60);
 	if (exp) {
-		qword_printint(f, exp->e_flags);
+		int different_fs = 0;
+		struct stat stb1, stb2;
+		if (stat(path, &stb1) == 0 &&
+		    stat(exp->e_path, &stb2) == 0 &&
+		    stb1.st_dev != stb2.st_dev)
+			different_fs = 1;
+		if (different_fs)
+			qword_printint(f, exp->e_flags & ~NFSEXP_FSID);
+		else
+			qword_printint(f, exp->e_flags);
 		qword_printint(f, exp->e_anonuid);
 		qword_printint(f, exp->e_anongid);
 		qword_printint(f, exp->e_fsid);
 		write_fsloc(f, exp, path);
 		write_secinfo(f, exp);
 #if USE_BLKID
- 		if (exp->e_uuid == NULL) {
+ 		if (exp->e_uuid == NULL || different_fs) {
  			char u[16];
  			if (get_uuid(path, NULL, 16, u)) {
  				qword_print(f, "uuid");
  				qword_printhex(f, u, 16);
  			}
- 		} else if (exp->e_uuid) {
+ 		} else {
  			qword_print(f, "uuid");
  			qword_printhex(f, exp->e_uuid, 16);
  		}

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[J. Bruce Fields]

On Fri, Nov 23, 2007 at 01:50:29PM +1100, Neil Brown wrote:
> On Wednesday September 5, bfields@fieldses.org wrote:
> > With an exports entry like
> > 
> > 	/exports *.citi.umich.edu(rw,crossmnt,fsid=0)
> > 
> > and some other stuff mounted under /exports, I get an odd situation
> > where listing /exports/submnt/ shows me the contents of /exports/.
> > 
> > I looks to me like what's happening is that the new code to
> > automatically export filesystems under crossmnt exports is exporting
> > those directories with the fsid option turned on, with the result that
> > the root of each of those subdirectories ends up with the same
> > filehandle as the root of /exports, and the hilarity ensues.  But I
> > haven't tried a patch yet.
> > 
> > Does that make sense?  I assume we should just clear the fsid flag on
> > submounts, as it's never going to make sense, and leave the logic
> > otherwise intact.
> > 
> > --b.
> 
> Thanks for noticing and reporting this.
> 
> Does this (untested, but it compiles) patch seem right?
> 
> NeilBrown
> 
> --------
> When exported a filesystems with option inherited (by the crossmnt
> option) from a higherlevel filesystem, ignore filesystem specific
> options like FSID and explicit UUID.

Looks right to me.  I'm confused about uuid's--how does the non-explicit
UUID case work?

--b.

> 
> Signed-off-by: NeilBrown <neilb@suse.de>
> 
> diff --git a/utils/mountd/cache.c b/utils/mountd/cache.c
> index fd317cd..ff56b9f 100644
> --- a/utils/mountd/cache.c
> +++ b/utils/mountd/cache.c
> @@ -565,20 +565,29 @@ static int dump_to_cache(FILE *f, char *domain, char *path, struct exportent *ex
>  	qword_print(f, path);
>  	qword_printint(f, time(0)+30*60);
>  	if (exp) {
> -		qword_printint(f, exp->e_flags);
> +		int different_fs = 0;
> +		struct stat stb1, stb2;
> +		if (stat(path, &stb1) == 0 &&
> +		    stat(exp->e_path, &stb2) == 0 &&
> +		    stb1.st_dev != stb2.st_dev)
> +			different_fs = 1;
> +		if (different_fs)
> +			qword_printint(f, exp->e_flags & ~NFSEXP_FSID);
> +		else
> +			qword_printint(f, exp->e_flags);
>  		qword_printint(f, exp->e_anonuid);
>  		qword_printint(f, exp->e_anongid);
>  		qword_printint(f, exp->e_fsid);
>  		write_fsloc(f, exp, path);
>  		write_secinfo(f, exp);
>  #if USE_BLKID
> - 		if (exp->e_uuid == NULL) {
> + 		if (exp->e_uuid == NULL || different_fs) {
>   			char u[16];
>   			if (get_uuid(path, NULL, 16, u)) {
>   				qword_print(f, "uuid");
>   				qword_printhex(f, u, 16);
>   			}
> - 		} else if (exp->e_uuid) {
> + 		} else {
>   			qword_print(f, "uuid");
>   			qword_printhex(f, exp->e_uuid, 16);
>   		}

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[Neil Brown]

On Monday November 26, bfields@fieldses.org wrote:
> > When exported a filesystems with option inherited (by the crossmnt
> > option) from a higherlevel filesystem, ignore filesystem specific
> > options like FSID and explicit UUID.
> 
> Looks right to me.  I'm confused about uuid's--how does the non-explicit
> UUID case work?

Thanks, though on reflection I don't think we need to do the
'stat's. Just checking the path names should be sufficient, but I will
double check.

non-explicit UUIDs are derived using libblkid, typically a UUID out of
the filesystem that is being exported.
The only problem is that it appears that libblkid isn't as efficient
as one might like and if you have hundreds of mounts, mountd can slow
down.  I still need to look into that.

NeilBrown

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[Frank Filz]

On Fri, 2007-11-23 at 13:50 +1100, Neil Brown wrote:
> On Wednesday September 5, bfields@fieldses.org wrote:
> > With an exports entry like
> > 
> > 	/exports *.citi.umich.edu(rw,crossmnt,fsid=0)
> > 
> > and some other stuff mounted under /exports, I get an odd situation
> > where listing /exports/submnt/ shows me the contents of /exports/.
> > 
> > I looks to me like what's happening is that the new code to
> > automatically export filesystems under crossmnt exports is exporting
> > those directories with the fsid option turned on, with the result that
> > the root of each of those subdirectories ends up with the same
> > filehandle as the root of /exports, and the hilarity ensues.  But I
> > haven't tried a patch yet.
> > 
> > Does that make sense?  I assume we should just clear the fsid flag on
> > submounts, as it's never going to make sense, and leave the logic
> > otherwise intact.
> > 
> > --b.
> 
> Thanks for noticing and reporting this.
> 
> Does this (untested, but it compiles) patch seem right?

In testing security negotiation, I ran into this problem. I've tried
this patch and it doesn't seem to be working right. My exports file is:

/export/home    gss/krb5i(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
/export/home    gss/krb5p(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
/export         gss/krb5(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)
/export         gss/krb5i(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)

>From the client I execute:

mount -overs=4,sec=krb5 elm3a19.beaverton.ibm.com:/ /mnt 
ls -l /mnt/home

I added some debug logging, and I see the different_fs path being taken
on the /export/home. I'm wondering, could there be a kernel part to this
also?

Thanks

Frank Filz


-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[Neil Brown]

On Monday December 3, ffilzlnx@us.ibm.com wrote:
> On Fri, 2007-11-23 at 13:50 +1100, Neil Brown wrote:
> > Does this (untested, but it compiles) patch seem right?
> 
> In testing security negotiation, I ran into this problem. I've tried
> this patch and it doesn't seem to be working right. My exports file is:
> 
> /export/home    gss/krb5i(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
> /export/home    gss/krb5p(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
> /export         gss/krb5(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)
> /export         gss/krb5i(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)

You don't need the patch for this exports file.  It is only relevant
when the underlying filesystem is not explicitly exported, so the
export flags a deduced implicitly.

So - this is a separate issue.

> 
> From the client I execute:
> 
> mount -overs=4,sec=krb5 elm3a19.beaverton.ibm.com:/ /mnt 
> ls -l /mnt/home

... and what happens?

NeilBrown

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[Frank Filz]

On Tue, 2007-12-04 at 12:09 +1100, Neil Brown wrote:
> On Monday December 3, ffilzlnx@us.ibm.com wrote:
> > On Fri, 2007-11-23 at 13:50 +1100, Neil Brown wrote:
> > > Does this (untested, but it compiles) patch seem right?
> > 
> > In testing security negotiation, I ran into this problem. I've tried
> > this patch and it doesn't seem to be working right. My exports file is:
> > 
> > /export/home    gss/krb5i(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
> > /export/home    gss/krb5p(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
> > /export         gss/krb5(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)
> > /export         gss/krb5i(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)
> 
> You don't need the patch for this exports file.  It is only relevant
> when the underlying filesystem is not explicitly exported, so the
> export flags a deduced implicitly.
> 
> So - this is a separate issue.

Ah, ok.
 
> > From the client I execute:
> > 
> > mount -overs=4,sec=krb5 elm3a19.beaverton.ibm.com:/ /mnt 
> > ls -l /mnt/home
> 
> ... and what happens?

This is what I get:

# mount -overs=4,sec=krb5 elm3a19.beaverton.ibm.com:/ /mnt
# ls -l /mnt
total 32
-rw-r--r--    1 nobody   nobody           32 Nov  7 15:00 foo
drwxr-xr-x   19 nobody   nobody         4096 Dec  3 14:40 home
drwxr-xr-x    2 nobody   nobody         4096 Nov  2 17:51 homw2
# ls -l /mnt/home
total 32
-rw-r--r--    1 nobody   nobody           32 Nov  7 15:00 foo
drwxr-xr-x   19 nobody   nobody         4096 Dec  3 14:40 home
drwxr-xr-x    2 nobody   nobody         4096 Nov  2 17:51 homw2
# ls -l /mnt/homw2  
total 0
-rw-r--r--    1 nobody   nobody            0 Nov  2 17:51 foo
# umount /mnt

This is from an AIX client, however, with the new format exports with
the sec= export option, the AIX client works just fine (given the patch
I submitted previously).

Frank Filz


-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[Frank Filz]

I've done some more exploration, I think the exports are getting set
wrong. First, here is the working case with the new exports format (cut
from an annotated /var/log/messages):

START======================================================
======= cat /etc/exports
/export		*(sec=krb5,fsid=0,insecure,no_subtree_check,no_root_squash,async)
/export/home	*(sec=krb5,nohide,insecure,no_subtree_check,no_root_squash,async)
======= mount -overs=4,sec=krb5 elm3a19.beaverton.ibm.com:/ /mnt
Dec  7 10:47:20 elm3a19 mountd[4691]: cache_export_ent domain=*, exp path=/export, path=/export 
Dec  7 10:47:20 elm3a19 mountd[4691]: dump_to_cache path=/export, exp path=/export 
======= ls -l /mnt
Dec  7 10:47:22 elm3a19 mountd[4691]: nfsd_export domain=*, path=/export/home, exp path=/export/home 
Dec  7 10:47:22 elm3a19 mountd[4691]: dump_to_cache path=/export/home, exp path=/export/home 
Dec  7 10:47:23 elm3a19 mountd[4691]: cache_export_ent domain=*, exp path=/export/home, path=/export/home 
Dec  7 10:47:23 elm3a19 mountd[4691]: dump_to_cache path=/export/home, exp path=/export/home 
======= umount /mnt
==========================
Dec  7 10:47:28 elm3a19 rpc.idmapd[2471]: nfsdcb: authbuf=gss/krb5 authtype=user
Dec  7 10:47:28 elm3a19 rpc.idmapd[2471]: nfsdcb: authbuf=gss/krb5 authtype=group
======= ls -l /mnt/home
======= umount /mnt
END=======================================================


And then here is the non-working form with the old exports format:

START======================================================
======= cat /etc/exports
/export/home	gss/krb5i(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
/export/home	gss/krb5p(nohide,insecure,no_subtree_check,no_root_squash,async,rw)
/export         gss/krb5(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)
/export         gss/krb5i(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)

======= mount -overs=4,sec=krb5 elm3a19.beaverton.ibm.com:/ /mnt
Dec  7 10:46:53 elm3a19 mountd[4691]: cache_export_ent domain=gss/krb5, exp path=/export, path=/export 
Dec  7 10:46:53 elm3a19 mountd[4691]: dump_to_cache path=/export, exp path=/export 
======= ls -l /mnt
Dec  7 10:46:55 elm3a19 mountd[4691]: nfsd_export domain=gss/krb5, path=/export/home, exp path=/export 
Dec  7 10:46:55 elm3a19 mountd[4691]: dump_to_cache path=/export/home, exp path=/export 
Dec  7 10:46:56 elm3a19 mountd[4691]: cache_export_ent domain=gss/krb5, exp path=/export, path=/export/home 
Dec  7 10:46:56 elm3a19 mountd[4691]: dump_to_cache path=/export, exp path=/export 
Dec  7 10:46:56 elm3a19 mountd[4691]: dump_to_cache path=/export/home, exp path=/export 
======= umount /mnt
==========================
======= ls -l /mnt/home
======= umount /mnt
END=======================================================

I see that the way mountd caches the /exports/home export is different
between the two cases, with the non-working old exports format case
caching a path of /export/home but using the /export export.

p.s. I am using nfs-utils-1.1.1 for this, and applied the previously
mentioned crossmount patch.

Oh, and just for completeness, here is implicit exporting via crossmnt,
with the same result:

START======================================================
======= cat /etc/exports
/export
gss/krb5(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)
/export
gss/krb5i(fsid=0,insecure,no_subtree_check,no_root_squash,async,rw,crossmnt)

======= mount -overs=4,sec=krb5 elm3a19.beaverton.ibm.com:/ /mnt
Dec  7 11:00:22 elm3a19 mountd[4691]: cache_export_ent domain=gss/krb5, exp path=/export, path=/export 
Dec  7 11:00:22 elm3a19 mountd[4691]: dump_to_cache path=/export, exp path=/export 
Dec  7 11:00:22 elm3a19 rpc.idmapd[2471]: nfsdcb: authbuf=gss/krb5 authtype=user
Dec  7 11:00:22 elm3a19 rpc.idmapd[2471]: nfsdcb: authbuf=gss/krb5 authtype=group
======= ls -l /mnt
Dec  7 11:00:24 elm3a19 mountd[4691]: nfsd_export domain=gss/krb5, path=/export/home, exp path=/export 
Dec  7 11:00:24 elm3a19 mountd[4691]: dump_to_cache path=/export/home, exp path=/export 
Dec  7 11:00:25 elm3a19 mountd[4691]: cache_export_ent domain=gss/krb5, exp path=/export, path=/export/home 
Dec  7 11:00:25 elm3a19 mountd[4691]: dump_to_cache path=/export, exp path=/export 
Dec  7 11:00:25 elm3a19 mountd[4691]: dump_to_cache path=/export/home, exp path=/export 
======= umount /mnt
==========================
======= ls -l /mnt/home
======= umount /mnt
END=======================================================

Looks the same as with the explicit exporting.

Any suggestions or thoughts?

Thanks

Frank Filz



-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-utils crossmnt/fsid bug

[Frank Filz]

Ok, I've looked at this issue some more. The crossmnt and nohide options
work correctly so long as security negotiation isn't required to cross
the mount point (whether the mount command provides the correct set of
security flavors or not). If the crossmnt export option is provided in
this case (whether or not nohide is specified on the child export), an
ls of the /mnt/home ends up with the results from an ls of /mnt. If the
crossmnt option is not specified on the parent but nohide is specified
on the child, an ls -l of /mnt results in slightly different contents
while an ls -l of /mnt/home results in the contents of the hidden
directory on the parent export.

These behaviors seem undesirable, but perhaps not worth eliminating
since they do represent something unsupported by the old format which is
deprecated in favor of the new format where everything works just fine.

Frank Filz



-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs