FS-specified FSID for non-device based filesystems?

FS-specified FSID for non-device based filesystems?

[Oleg Drokin]

Hello!

   Right now NFSD depends on FS_REQUIRES_DEV to be set in sb->s_type->fs_flags
   or manually set fsid= in /etc/exports in order to export filesystem.
   But there are non-device based filesystems (Lustre for example ;) ) that
   would like to be exported via NFS without guiding users through this
   fsid= business. Of course it is easy to cheat through by setting
   FS_REQUIRES_DEV, but this does not solve entire problem. While allowing
   for single-node export of filesystem to work ok, more generic problem of
   exporting same filesystem from several NFS servers in clustered manner
   won't work because sb->s_dev might be different on different nodes.
   This is not only Lustre problem, I believe same problem will happen
   with GFS or OCFS2 or whatever other SAN-based fs if different nodes
   have different device names (/dev/sda | /dev/sdb) for same SAN device,
   in such a case fsid calculated would be different on different nodes too.

   I wonder if e.g. another export op could be introduced to ask filesystem
   for its unique id (if supported) and use that instead of sb->s_dev in
   fsid calculations. Does this look sane?
   I hope I am not missing anything?
   Or are there better ideas?

   Thanks.

Bye,
    Oleg

Re: FS-specified FSID for non-device based filesystems?

[Bryan Henderson]

>Right now NFSD depends on FS_REQUIRES_DEV to be set in 
sb->s_type->fs_flags
>or manually set fsid= in /etc/exports in order to export filesystem.
>But there are non-device based filesystems (Lustre for example ;) ) that
>would like to be exported via NFS without guiding users through this
>fsid= business. 
>...
>I wonder if e.g. another export op could be introduced to ask filesystem
>for its unique id

It would also be quite valuable with device-based filesystems, for the 
opposite reason.  That a device-based filesystem can be uniquely 
identified by the device number of its device is a fiction that is getting 
harder to entertain every year.  Device numbers of devices change and a 
device can contain multiple filesystems over time.

When Linux added the export ID, at a time when other operating systems 
were addressing the problem with the kind of fsid you're proposing, I 
believe the point was that in Linux you can have multiple incompatible 
exports of the same filesystem -- to give a client the right data, you'd 
need to know from which export, not just which filesystem, a given 
filehandle came.

Nonetheless, I agree that it would be good to have the option of using a 
filesystem identifier instead of an export identifier.  In a system that 
only does simple uniform exports of whole filesystems, I think filesystem 
ID works perfectly, and it is much easier to use.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: FS-specified FSID for non-device based filesystems?

[Oleg Drokin]

Hello!

On Fri, Apr 14, 2006 at 01:22:14PM -0700, Bryan Henderson wrote:
> When Linux added the export ID, at a time when other operating systems 
> were addressing the problem with the kind of fsid you're proposing, I 
> believe the point was that in Linux you can have multiple incompatible 
> exports of the same filesystem -- to give a client the right data, you'd 
> need to know from which export, not just which filesystem, a given 
> filehandle came.

Well, I do not see how there is a difference.
>From my reading of the code it seems that fsid is composed of device's
minor/major and export point's inode ino.

I propose to replace (if supported by fs) device's minor/major by
some unique number returned by fs itself.
Should be no worse than what we have now.

Bye,
    Oleg

Re: FS-specified FSID for non-device based filesystems?

[J. Bruce Fields]

On Fri, Apr 14, 2006 at 11:45:43PM +0300, Oleg Drokin wrote:
> Well, I do not see how there is a difference.  From my reading of the
> code it seems that fsid is composed of device's minor/major and export
> point's inode ino.
> 
> I propose to replace (if supported by fs) device's minor/major by some
> unique number returned by fs itself.  Should be no worse than what we
> have now.

Wouldn't some ways of identifying filesystems (ext2/3 labels?) more
easily be done from userspace?  Would it be possible to do what you want
from userspace using the fsid export option?  There's no reason the
fsid= option has to be set manually; you could generate /etc/exports
lines (or exportfs commands) automatically.

--b.

Re: FS-specified FSID for non-device based filesystems?

[Bryan Henderson]

>From my reading of the code it seems that fsid is composed of device's
>minor/major and export point's inode ino.
>
>I propose to replace (if supported by fs) device's minor/major by
>some unique number returned by fs itself.
>Should be no worse than what we have now.

That's for a device-based filesystem.  But isn't your proposal primarily 
about non-device-based filesystems, and replacing a user-supplied value 
with this filesystem identifier?  It isn't so clear that that can't make 
things worse.

But I don't actually remember the specific cases in which the fsid was 
considered insufficient; only that someone (probably Neil) did find that 
it was.  (Before Linux had export IDs, you simply couldn't export a 
non-device-based filesystem at all, so we had to make a fake device on 
which the filesystem could "reside."  The user managed the device number 
of that device as the filesystem ID).

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: FS-specified FSID for non-device based filesystems?

[Oleg Drokin]

Hello!

On Fri, Apr 14, 2006 at 02:32:44PM -0700, Bryan Henderson wrote:
> >From my reading of the code it seems that fsid is composed of device's
> >minor/major and export point's inode ino.
> >I propose to replace (if supported by fs) device's minor/major by
> >some unique number returned by fs itself.
> >Should be no worse than what we have now.
> That's for a device-based filesystem.  But isn't your proposal primarily 

Yes. Device-based fs has root->s_dev. For non-device based (and for
device-based too, if they would like) - method to get some sort of
unique id to be used in the same way root->s_dev is used.

> about non-device-based filesystems, and replacing a user-supplied value 
> with this filesystem identifier?  It isn't so clear that that can't make 
> things worse.

No, replacing whole fsid= with fs-supplied value would have implications like
that you mentioned - we might have two different points in filesystem
exported and we do not want those to have same id.

> But I don't actually remember the specific cases in which the fsid was 
> considered insufficient; only that someone (probably Neil) did find that 
> it was.  (Before Linux had export IDs, you simply couldn't export a 
> non-device-based filesystem at all, so we had to make a fake device on 
> which the filesystem could "reside."  The user managed the device number 
> of that device as the filesystem ID).

We still have those "fake devices". In fact non-device based fs can set
REQUIRES_FS_DEV in fs_flags and it can be exported after that without
manually specifying fsid. (yeah, I tried). But distributed filesystems would
like to have same device id on different nodes.

Bye,
    Oleg

Re: FS-specified FSID for non-device based filesystems?

[Neil Brown]

On Wednesday April 12, green@linuxhacker.ru wrote:
> 
>    I wonder if e.g. another export op could be introduced to ask filesystem
>    for its unique id (if supported) and use that instead of sb->s_dev in
>    fsid calculations. Does this look sane?
>    I hope I am not missing anything?
>    Or are there better ideas?
> 

Yes.  fsid= is a problem.  Getting information from the filesystem
would be nice.  But there are issues of practicality.

The NFS filehandle is of limited size.
For NFSv2 we have 32 bytes and we cannot really afford more than 4 to
identify the filesystem.
4 bytes isn't really enough for a globally unique ID.  Many
filesystems have UUIDs but I would expect them to be more like 16
bytes.
Such a number could be hashed down to 4bytes, but you would greatly
increase the risk of collision.

Ofcourse, NFSv2 is pretty old now, and it is probably reasonable to
have features that are only available to v3 or later.
v3 has space in the filehandle for up to 64 bytes.  It would probably
be OK to allow upto 16 - or maybe 20 - bytes to identify the
filesystem.

There is still the issue of a filesystem having multiple export
points, though that isn't really a big issue as it is rarely needed so
we can impose extra constraints (you must give fsid=) in that case.

So there remains the question of whether this should be handled in the
kernel or in user-space.  There already exists libblkid which can
produce UUIDs for various filesystems (I believe).  But that reads the
block device, and the whole point of this is to work with filesystems
that don't have a simple block device.

I'd rather not make the 'get uuid' be an export op as it could well be
useful to other clients than just nfsd.

Unfortunately there is no unanimity of how to present filesystem
metadata to userspace.
 - ioctls are frowned upon
 - extended attribute have been suggested, they are about files,
    not filesystems
 - filesystems don't have a place in /sys
 - /proc is frowned upon

Here is an idea:  How about we make a magic filesystem mounted on e.g.
/metafs, which examines the current directory of the accessing process
in order to report metadata.
So
   cd /my/mount/point
   cat /metafa/uuid
will show the uuid of the fs at /my/mount/point...
OK, so that's really weird.  But I think there is a real need to be
able to expose filesystem metadata in a uniform way and I would like
the uuid to be made available through that.
With that in place, I would be happy to look at including a uuid in
the filehandle for v3.

Any ideas?

NeilBrown

Re: FS-specified FSID for non-device based filesystems?

[Greg KH]

On Mon, Apr 17, 2006 at 10:54:06AM +1000, Neil Brown wrote:
>  - filesystems don't have a place in /sys

Then what is /sys/fs/ for?  :)

thanks,

greg k-h

Re: FS-specified FSID for non-device based filesystems?

[Neil Brown]

On Sunday April 16, greg@kroah.com wrote:
> On Mon, Apr 17, 2006 at 10:54:06AM +1000, Neil Brown wrote:
> >  - filesystems don't have a place in /sys
> 
> Then what is /sys/fs/ for?  :)

good question!

adams # uname -a 
Linux adams 2.6.17-rc1-mm1 #298 SMP Tue Apr 11 14:19:52 EST 2006 i686 GNU/Linux
adams # ls -la /sys/fs
total 0
drwxr-xr-x   2 root root 0 Apr 17 14:23 .
drwxr-xr-x  11 root root 0 Apr 17 14:23 ..

Doesn't look very useful yet, but I'm hopeful.

What naming scheme is expected?  i.e. If I want to look at the nfs
client rpc stats for /import/warehouse,  where would I look?
  /sys/fs/import/warehouse/nfs/stats
seems unlikely.

The idea of putting filesystem stuff in /sys seem to come up on
linux-kernel every so often and it never seems to get anywhere, often
because Al Viro pokes obvious holes in the scheme.

I cannot find anything in Documentation/filesystems/sysfs.txt, so:
How is /sys/fs supposed to work?

Thanks,
NeilBrown

Re: FS-specified FSID for non-device based filesystems?

[Anton Altaparmakov]

On Mon, 17 Apr 2006, Neil Brown wrote:
> On Wednesday April 12, green@linuxhacker.ru wrote:
> > 
> >    I wonder if e.g. another export op could be introduced to ask filesystem
> >    for its unique id (if supported) and use that instead of sb->s_dev in
> >    fsid calculations. Does this look sane?
> >    I hope I am not missing anything?
> >    Or are there better ideas?
> 
> Yes.  fsid= is a problem.  Getting information from the filesystem
> would be nice.  But there are issues of practicality.
> 
> The NFS filehandle is of limited size.
> For NFSv2 we have 32 bytes and we cannot really afford more than 4 to
> identify the filesystem.
> 4 bytes isn't really enough for a globally unique ID.  Many
> filesystems have UUIDs but I would expect them to be more like 16
> bytes.
> Such a number could be hashed down to 4bytes, but you would greatly
> increase the risk of collision.
> 
> Ofcourse, NFSv2 is pretty old now, and it is probably reasonable to
> have features that are only available to v3 or later.
> v3 has space in the filehandle for up to 64 bytes.  It would probably
> be OK to allow upto 16 - or maybe 20 - bytes to identify the
> filesystem.
> 
> There is still the issue of a filesystem having multiple export
> points, though that isn't really a big issue as it is rarely needed so
> we can impose extra constraints (you must give fsid=) in that case.
> 
> So there remains the question of whether this should be handled in the
> kernel or in user-space.  There already exists libblkid which can
> produce UUIDs for various filesystems (I believe).  But that reads the
> block device, and the whole point of this is to work with filesystems
> that don't have a simple block device.
> 
> I'd rather not make the 'get uuid' be an export op as it could well be
> useful to other clients than just nfsd.
> 
> Unfortunately there is no unanimity of how to present filesystem
> metadata to userspace.
>  - ioctls are frowned upon
>  - extended attribute have been suggested, they are about files,
>     not filesystems
>  - filesystems don't have a place in /sys
>  - /proc is frowned upon
> 
> Here is an idea:  How about we make a magic filesystem mounted on e.g.
> /metafs, which examines the current directory of the accessing process
> in order to report metadata.
> So
>    cd /my/mount/point
>    cat /metafa/uuid
> will show the uuid of the fs at /my/mount/point...
> OK, so that's really weird.  But I think there is a real need to be
> able to expose filesystem metadata in a uniform way and I would like
> the uuid to be made available through that.
> With that in place, I would be happy to look at including a uuid in
> the filehandle for v3.
> 
> Any ideas?

Something like getattrlist() system call on OSX would do very nicely.  It 
even allows for filesystems to communicate which attributes are supported 
and which are not.  (And there is an equivalent setattrlist(), too.)

http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/getattrlist.2.html

On the kernel VFS/FS interface side this works by the VFS sending a 
bitmask to the FS, with each bit being one of the attributes requested and 
for each set bit, the FS returns the appropriate information in the 
"getattr structure" and sets a flag in a second bitmask to say "I have 
returned this attribute".  The VFS then converts the returned attribute 
values as signified by set bits in the second bitmask from the structure 
returned by the FS into the user supplied buffer in a packed form (see the 
above man page for details on the packed form).

This works very well and is very extensible.

Linux already has a per FS getattr() inode operation which could easily be 
extended to be more powerful and a system call could be added to return 
information obtained from it...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: FS-specified FSID for non-device based filesystems?

[Greg KH]

On Mon, Apr 17, 2006 at 02:32:04PM +1000, Neil Brown wrote:
> On Sunday April 16, greg@kroah.com wrote:
> > On Mon, Apr 17, 2006 at 10:54:06AM +1000, Neil Brown wrote:
> > >  - filesystems don't have a place in /sys
> > 
> > Then what is /sys/fs/ for?  :)
> 
> good question!
> 
> adams # uname -a 
> Linux adams 2.6.17-rc1-mm1 #298 SMP Tue Apr 11 14:19:52 EST 2006 i686 GNU/Linux
> adams # ls -la /sys/fs
> total 0
> drwxr-xr-x   2 root root 0 Apr 17 14:23 .
> drwxr-xr-x  11 root root 0 Apr 17 14:23 ..
> 
> Doesn't look very useful yet, but I'm hopeful.
> 
> What naming scheme is expected?  i.e. If I want to look at the nfs
> client rpc stats for /import/warehouse,  where would I look?
>   /sys/fs/import/warehouse/nfs/stats
> seems unlikely.
> 
> The idea of putting filesystem stuff in /sys seem to come up on
> linux-kernel every so often and it never seems to get anywhere, often
> because Al Viro pokes obvious holes in the scheme.
> 
> I cannot find anything in Documentation/filesystems/sysfs.txt, so:
> How is /sys/fs supposed to work?

I don't know, as I didn't create it.  I think one of the FUSE developers
did.

thanks,

greg k-h

Re: FS-specified FSID for non-device based filesystems?

[Miklos Szeredi]

> > because Al Viro pokes obvious holes in the scheme.
> > 
> > I cannot find anything in Documentation/filesystems/sysfs.txt, so:
> > How is /sys/fs supposed to work?
> 
> I don't know, as I didn't create it.  I think one of the FUSE developers
> did.

Yeah.  There are a couple of FUSE specific attributes under
/sys/fs/fuse (documented in Documentation/filesystems/fuse.txt).

I'll put a pointer into sysfs.txt.

Miklos

Re: FS-specified FSID for non-device based filesystems?

[Al Viro]

On Wed, Apr 19, 2006 at 02:15:41PM +0200, Miklos Szeredi wrote:
> > > because Al Viro pokes obvious holes in the scheme.
> > > 
> > > I cannot find anything in Documentation/filesystems/sysfs.txt, so:
> > > How is /sys/fs supposed to work?
> > 
> > I don't know, as I didn't create it.  I think one of the FUSE developers
> > did.
> 
> Yeah.  There are a couple of FUSE specific attributes under
> /sys/fs/fuse (documented in Documentation/filesystems/fuse.txt).
> 
> I'll put a pointer into sysfs.txt.

Speaking of obvious holes, what is
        /* Setting file->private_data can't race with other mount()
           instances, since BKL is held for ->get_sb() */
        if (file->private_data)
                return -EINVAL;
doing, seeing that the next thing you do is GFP_KERNEL allocations, which
would kill all hope to use BKL-based protection here?

Re: FS-specified FSID for non-device based filesystems?

[Miklos Szeredi]

> Speaking of obvious holes, what is
>         /* Setting file->private_data can't race with other mount()
>            instances, since BKL is held for ->get_sb() */
>         if (file->private_data)
>                 return -EINVAL;
> doing, seeing that the next thing you do is GFP_KERNEL allocations, which
> would kill all hope to use BKL-based protection here?

Why?  I didn't know BKL is incompatible with allocations.

Miklos

Re: FS-specified FSID for non-device based filesystems?

[Al Viro]

On Thu, Apr 20, 2006 at 01:58:43PM +0200, Miklos Szeredi wrote:
> > Speaking of obvious holes, what is
> >         /* Setting file->private_data can't race with other mount()
> >            instances, since BKL is held for ->get_sb() */
> >         if (file->private_data)
> >                 return -EINVAL;
> > doing, seeing that the next thing you do is GFP_KERNEL allocations, which
> > would kill all hope to use BKL-based protection here?
> 
> Why?  I didn't know BKL is incompatible with allocations.

BKL is dropped when you give up CPU.  It is reacquired when you get scheduled
again.  It's not a mutex; it's not a spinlock.  The right way to look at it
is the following:
	* task has dumbness counter; lock_kernel() increments it,
unlock_kernel() decrements.
	* at most one dumb task can have CPU at any given time
	* dumb tasks are not preempted
That's it.  You are only protected against other dumb tasks and *only* as long
as you don't block.  Blocking won't make you any less dumb, so the same
warranties resume once you do get CPU again.  However, anything might have
happened while you were blocked.

On UP boxen with PREEMPT disabled it's simply a no-op - it's all about
emulating that situation.

Re: FS-specified FSID for non-device based filesystems?

[Miklos Szeredi]

> BKL is dropped when you give up CPU.  It is reacquired when you get scheduled
> again.  It's not a mutex; it's not a spinlock.  The right way to look at it
> is the following:
> 	* task has dumbness counter; lock_kernel() increments it,
> unlock_kernel() decrements.
> 	* at most one dumb task can have CPU at any given time
> 	* dumb tasks are not preempted
> That's it.  You are only protected against other dumb tasks and *only* as long
> as you don't block.  Blocking won't make you any less dumb, so the same
> warranties resume once you do get CPU again.  However, anything might have
> happened while you were blocked.
> 
> On UP boxen with PREEMPT disabled it's simply a no-op - it's all about
> emulating that situation.

I see, thanks.  I'll fix the race.

Miklos