nfs exporting my stacked fs

nfs exporting my stacked fs

[Andrew Sharp]

I could use some help getting my stacked (I guess that's the term d' jour)
file system to be nfs exported/mounted.  From reading the code, it seems
that my file system either has to require a device, it doesn't, or it
has to have some sort of FSID (the call to nfsctl_export() is failing),
only I can't seem to figure out how to accomplish that.  I tried to go
through the mountd source, but came up empty.  What did I miss?

TIA,

a

Re: nfs exporting my stacked fs

[Neil Brown]

On Wednesday May 28, andy@ccpu.com wrote:
> I could use some help getting my stacked (I guess that's the term d' jour)
> file system to be nfs exported/mounted.  From reading the code, it seems
> that my file system either has to require a device, it doesn't, or it
> has to have some sort of FSID (the call to nfsctl_export() is failing),
> only I can't seem to figure out how to accomplish that.  I tried to go
> through the mountd source, but came up empty.  What did I miss?

If your filesystem isn't device based (which seems likely for a
stacked filesystem), the nfsd cannot use the device number as a short,
fixed length identifier for the filesystem, which it needs for storing
in the filehandle.  So, you need to provide one.
This is done in the /etc/exportss file with the fsid= option.
e.g.

 /my/file/system myclient(rw,no_root_squash,sync,fsid=27)

NeilBrown

Re: nfs exporting my stacked fs

[Andrew Sharp]

On Thu, May 29, 2003 at 09:05:37AM +1000, Neil Brown wrote:
> On Wednesday May 28, andy@ccpu.com wrote:
> > I could use some help getting my stacked (I guess that's the term d' jour)
> > file system to be nfs exported/mounted.  From reading the code, it seems
> > that my file system either has to require a device, it doesn't, or it
> > has to have some sort of FSID (the call to nfsctl_export() is failing),
> > only I can't seem to figure out how to accomplish that.  I tried to go
> > through the mountd source, but came up empty.  What did I miss?
> 
> If your filesystem isn't device based (which seems likely for a
> stacked filesystem), the nfsd cannot use the device number as a short,
> fixed length identifier for the filesystem, which it needs for storing
> in the filehandle.  So, you need to provide one.
> This is done in the /etc/exportss file with the fsid= option.
> e.g.
> 
>  /my/file/system myclient(rw,no_root_squash,sync,fsid=27)

Thanks for that info, I was unaware of that option.  Soooo, this brings
up some other questions for me.

What is the relation between the fsid option mentioned above and the
fsid returned by statfs op?  Which I discovered shortly after sending my
first email.  Should they match?  The i_dev is apparently used on device
datasets for the filehandle, so do all datasets with the same i_dev
somehow guarantee that i_dev/inode # 'filehandles' are somehow unique?

a

Re: nfs exporting my stacked fs

[Neil Brown]

On Wednesday May 28, andy@ccpu.com wrote:
> On Thu, May 29, 2003 at 09:05:37AM +1000, Neil Brown wrote:
> > 
> >  /my/file/system myclient(rw,no_root_squash,sync,fsid=27)
> 
> Thanks for that info, I was unaware of that option.  Soooo, this brings
> up some other questions for me.
> 
> What is the relation between the fsid option mentioned above and the
> fsid returned by statfs op?  Which I discovered shortly after sending my
> first email.  Should they match?  The i_dev is apparently used on device
> datasets for the filehandle, so do all datasets with the same i_dev
> somehow guarantee that i_dev/inode # 'filehandles' are somehow unique?
> 
> a

What should the 'stat' fsid be ?  Good question.  The NFS protocol
specs don't really say much about it.
In NFSv3, the spec explicitly says it must always be the same, so the
client is free to ignore it.  However the linux client doesn't ignore
it but uses it as part of it's handle to identify an object (I think).
So I guess we should really use the fsid..

I'm not sure what you mean when you are talking about datasets...

An inode number is provided by a filesystem, and each device can only
have one filesystem mounted, so as long as the filesystem ensures
inode numbers are unique, then dev+inode will always be unique.

NeilBrown

Re: nfs exporting my stacked fs

[Trond Myklebust]

>>>>> " " == Neil Brown <neilb@cse.unsw.edu.au> writes:

     > is free to ignore it.  However the linux client doesn't ignore
     > it but uses it as part of it's handle to identify an object (I
     > think).  So I guess we should really use the fsid..

Note: that behaviour has changed in recent kernels. We no longer look
at the fsid.

However we will have to reintroduce it for NFSv4. Unlike earlier
versions of NFS, v4 allows us to cross mountpoints on the server. When
this happens, the client is supposed to detect the crossing by means
of the fsid, and then to automount the new partition.
This behaviour would also make sense under NFSv3 for 'nohide'...

Cheers,
  Trond

Re: nfs exporting my stacked fs

[Bryan Henderson]

>An inode number is provided by a filesystem, and each device can only
>have one filesystem mounted, so as long as the filesystem ensures
>inode numbers are unique, then dev+inode will always be unique.

This is true in any instant, but for most NFS purposes, we also care about
identifying a file uniquely over time, even when that time might involve
sea changes such as a reboot.  E.g. one would like to make sure that a file
handle that referred to file A an hour ago still refers to file A now.  Or
at least it doesn't refer to file B.  So it is worth mentioning here the
(often discussed) ways in which a single dev+inode might refer to two
different files.

If you delete file A and create file B, A might have the same inode number
as B.  Inode generation numbers are sometimes used to disambiguate.  So are
stronger definitions of "filesystem ensures inode numbers are unique."  If
you delete the entire filesystem and create a new one on the same device,
dev+inode+inode_generation may refer to file A today and file B tomorrow.
If you change the device number of the device (all too easy in Linux,
especially with a reboot), same thing.

The fundamental idea of the fsid (filesystem ID) is that it uniquely
identifies a filesystem.  Putting a new filesystem onto a disk generates a
new fsid.  Changing the device number (or moving the filesystem to another
disk) does not change the fsid.

I think we've discovered recently that the fundamental idea of the fsid is
inadequate for NFS purposes, though, and that an export ID serves our
purpose better.

(btw - I know Neil knows all this well -- my response to his post is meant
to pick up where that post left off, not contradict it).

--
Bryan Henderson                       IBM Almaden Research Center
San Jose CA                           Filesystems

Re: nfs exporting my stacked fs

[Trond Myklebust]

>>>>> " " == Bryan Henderson <hbryan@us.ibm.com> writes:

     > This is true in any instant, but for most NFS purposes, we also
     > care about identifying a file uniquely over time, even when
     > that time might involve sea changes such as a reboot.  E.g. one
     > would like to make sure that a file handle that referred to
     > file A an hour ago still refers to file A now.

According to RFC1813 (which clarifies common practices for RFC1094)
file handles are *unique*. More specifically:

  "If two file handles from the same server are equal, they must refer
  to the same file, but if they are not equal, no conclusions can be
  drawn."

RFC1813 makes no exceptions to this rule. Including for time....

The one thing that is allowed, is that the server may revoke a
filehandle if (and only if) the file no longer exists, or if access to
the file has been revoked.


IOW: the condition you describe above is trivially observed...

Cheers,
  Trond

Re: nfs exporting my stacked fs

[Andrew Sharp]

On Sat, May 31, 2003 at 01:10:09AM +0200, Trond Myklebust wrote:
> >>>>> " " == Bryan Henderson <hbryan@us.ibm.com> writes:
> 
>      > This is true in any instant, but for most NFS purposes, we also
>      > care about identifying a file uniquely over time, even when
>      > that time might involve sea changes such as a reboot.  E.g. one
>      > would like to make sure that a file handle that referred to
>      > file A an hour ago still refers to file A now.
> 
> According to RFC1813 (which clarifies common practices for RFC1094)
> file handles are *unique*. More specifically:
> 
>   "If two file handles from the same server are equal, they must refer
>   to the same file, but if they are not equal, no conclusions can be
>   drawn."
> 
> RFC1813 makes no exceptions to this rule. Including for time....

Yes.  At first I was annoyed that the Linux NFS client peeks at the
file handle, because sometimes I want to change it for the same file,
but then it turns out that the solution to another problem of mine
fixes that as well.  And now it sounds like the v4 file handle won't
be considered opaque for the client any more.  As far as being able to
find the right file from a file handle, no matter how dusty, my fs has
the same problem as NFS in this regard.  A bit of challenge on Linux if
one hopes to maintain a proper dcache.

> The one thing that is allowed, is that the server may revoke a
> filehandle if (and only if) the file no longer exists, or if access to
> the file has been revoked.
> 
> 
> IOW: the condition you describe above is trivially observed...

I take it then that the fsid returned by ->statfs is largely not used
right now?  My version of nfs-utils (1.0) doesn't support the fsid option
in config files, so I've just hacked the kernel to get me through this
part of the development phase.

I'm guessing that all or part of this stuff is handled a bit differently
in 2.5?

a

Re: nfs exporting my stacked fs

[Trond Myklebust]

>>>>> " " == Andrew Sharp <andy@ccpu.com> writes:

     > Yes.  At first I was annoyed that the Linux NFS client peeks at
     > the file handle, because sometimes I want to change it for the
     > same file, but then it turns out that the solution to another
     > problem of mine fixes that as well.  And now it sounds like the
     > v4 file handle won't be considered opaque for the client any
     > more.

All v2/v3/v4 file handles are opaque as far as the client is
concerned. The only issue is that of uniqueness. Whereas NFSv2/v3 do
not support the concept of a filehandle that is not unique over time,
NFSv4 does (in theory) by means of the concept of "volatile
filehandles".

Note though that NFSv4 volatile filehandles have a number of unsolved
race conditions associated to them. I would recommend that people stay
away from them if at all possible.

     > I take it then that the fsid returned by ->statfs is largely
     > not used right now?  My version of nfs-utils (1.0) doesn't
     > support the fsid option in config files, so I've just hacked
     > the kernel to get me through this part of the development
     > phase.

The fsid is unused for NFSv2/v3. It is very important for NFSv4, as
the latter allows the server to cross mount points (rather like the
somewhat ill-defined "nohide" option did for NFSv2/v3).

In NFSv4, the client is expected to automount the new partition if we
cross a mount point on the server. Crossing a mount point is detected
by means of the fsid change...

     > I'm guessing that all or part of this stuff is handled a bit
     > differently in 2.5?

Not yet. However it will be once we fix the NFSv4 behaviour.

Cheers,
  Trond

Re: nfs exporting my stacked fs

[Bryan Henderson]

> > This is true in any instant, but for most NFS purposes, we also
> > care about identifying a file uniquely over time, even when
> > that time might involve sea changes such as a reboot.  E.g. one
> > would like to make sure that a file handle that referred to
> > file A an hour ago still refers to file A now.
>...
>the condition you describe above is trivially observed...

I don't exactly follow the language here, but from context I think you're
saying:  RFC 1813 requires file handles to be absolutely unique, which is a
condition that I said one "would like."

That's interesting (thanks for the info), but not as interesting as whether
the file handles in any real world NFS implementation are so unique.  And
of course they aren't.  Because they don't incorporate ideal fsids.  In a
typical NFS 2/3 implementation, a device number fills the role of fsid.  I
think we can agree that absent some simplifying assumptions, that means two
equal file handles from a server could refer to two different files.

I think we're all still looking for a practical implementation that
approaches the ideals of the RFC.

--
Bryan Henderson                       IBM Almaden Research Center
San Jose                              Filesystems

Re: nfs exporting my stacked fs

[Trond Myklebust]

>>>>> " " == Bryan Henderson <hbryan@us.ibm.com> writes:

     > In a typical NFS 2/3 implementation, a device number fills the
     > role of fsid.  I think we can agree that absent some
     > simplifying assumptions, that means two equal file handles from
     > a server could refer to two different files.

Then server admins are deliberately putting themselves in a situation
where their users may see data corruption. The above problem is
trivially solved by means of the 'fsid' export option.

For most setups, this should not be necessary (since device numbers
will suffice). Those that don't have permanently allocated device
numbers (e.g. fibre channel), should use the fsid export option in
order to provide a permanent value...

     > I think we're all still looking for a practical implementation
     > that approaches the ideals of the RFC.

I hope not.

Cheers,
  Trond

Re: nfs exporting my stacked fs

[Bryan Henderson]

>Then server admins are deliberately putting themselves in a situation
>where their users may see data corruption. The above problem is
>trivially solved by means of the 'fsid' export option.

Agreed, but note that you're talking specifically about a very recent Linux
system with a careful system administrator.  I was talking about the world
of NFS in general -- the practical world.

>For most setups, this should not be necessary (since device numbers
>will suffice).  Those that don't have permanently allocated device
>numbers (e.g. fibre channel), should use the fsid export option in
>order to provide a permanent value...

But now you're talking about sufficiency, whereas I was talking about
actually implementing unique fsids.  Device number is clearly sufficient,
since people have been using it for decades.  But it does not come close to
being a real fsid.  No device numbers in Linux (on architectures I know
about) are permanently associated with physical devices.  Lots of things,
including recabling, can change them.  Moreover, filesystems are not
permanently associated with physical devices.  You can move a filesystem
from one device to another.

The fsid export option, while a leap ahead of what we had before, is
clearly not the ultimate solution to the fsid problem, because it requires
too much human participation.  System administrators don't normally have to
administrate at that level.  It's like choosing device addresses and IRQ
levels when you install network cards.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: nfs exporting my stacked fs

[David Chow]

>Agreed, but note that you're talking specifically about a very recent Linux
>system with a careful system administrator.  I was talking about the world
>of NFS in general -- the practical world.
>
>  
>
>>For most setups, this should not be necessary (since device numbers
>>will suffice).  Those that don't have permanently allocated device
>>numbers (e.g. fibre channel), should use the fsid export option in
>>order to provide a permanent value...
>>    
>>
OK, I think I'd ask Trond a similar question long ago about fsids. One 
good example of the need of fsid
is high availability NFS . When a shared volume over fibre-channel, SAN 
or shared SCSI will definitely alter the device numbers without having 
to recabling . IBM's SAN unit does support virtual LUNs (I had a go with 
FastT 200 and Bryan should know better than me do) which device number 
has no sense at all. One or more machines can share the same physical 
volume with different LUNs (seen by the OS) and of course, device 
numbers are just junk. I think SAN is a subsystem designed for server 
farms, it is a bit difficult to make cluster of servers to maintain 
universal global device numbers, otherwise failover doesn't work though. 
May be for apps but unfortunately not NFS without fsid .

>But now you're talking about sufficiency, whereas I was talking about
>actually implementing unique fsids.  Device number is clearly sufficient,
>since people have been using it for decades.  But it does not come close to
>being a real fsid.  No device numbers in Linux (on architectures I know
>about) are permanently associated with physical devices.  Lots of things,
>including recabling, can change them.  Moreover, filesystems are not
>permanently associated with physical devices.  You can move a filesystem
>from one device to another.
>
>The fsid export option, while a leap ahead of what we had before, is
>clearly not the ultimate solution to the fsid problem, because it requires
>too much human participation.  System administrators don't normally have to
>administrate at that level.  It's like choosing device addresses and IRQ
>levels when you install network cards.
>  
>
Why not? We did that in the old days didn't we? A unversity graduate 
from computer course should well understand what is an address or IRQ. 
If I am a careful sys admin or settting up a HA cluster failover 
solution, I'll need it. I strongly support the fsid option as there is 
no point to remove it. For other users (admin who don't care or 
standalone nfs servers), nfsd can still use the device numbers to 
generate file handles.

Referring the subject of this mail, I did wrote a nfsd exportable 
consistent stacked fs . I simply register a dummy block device in my fs 
module to fool the nfsd I had one (in sense I don't use it at all). 
However, a change in fstab can change everything. Whether you guys want 
to rely on fsid in nfsd export identification, its up to you as I am not 
the maintainer of it. However, I do find it useful and especially for 
stacked fs and in many situation disregarding your fs is stacked or not. 
Or fsid should have save some device numbers (as they will run out soon) 
at least I don't have to waste a couple of them.

regards,
David Chow

Re: nfs exporting my stacked fs

[Bryan Henderson]

>>The fsid export option, while a leap ahead of what we had before, is
>>clearly not the ultimate solution to the fsid problem, because it
requires
>>too much human participation.  System administrators don't normally have
to
>>administrate at that level.  It's like choosing device addresses and IRQ
>>levels when you install network cards.
>>
>>
>Why not? We did that in the old days didn't we?

Because it isn't the old days.  I agree that in the old days, having the
system administrator manually manage export IDs would be perfectly
reasonable, as was having the system administrator manage IRQs and device
addresses.  But today, people generally expect more automation from
computers.

>I strongly support the fsid option as there is
>no point to remove it. For other users (admin who don't care or
>standalone nfs servers), nfsd can still use the device numbers to
>generate file handles.

Me too.  In fact, I'm not aware of any dissent on this point.  As I
mentioned much earlier, the "fsid" option is better than even a true fsid
in some cases.  With it, you can distinguish between two exports of a
single filesystem.

But we should all bear in mind that an fsid problem continues to exist.
Non-Linux systems don't have an export option to set the fsid.  Old Linux
systems don't either.  Device numbers do not map permanently to
filesystems, even with the simplest disk storage.  System administrators
don't want to manage the NFS file handle namespace.  And finally, there is
a problem of confusion, because what the RFC and common sense call an fsid
isn't exactly the same as what any existing system, including Linux with
the export option, implements by that name.  And that makes conversations
like this one more difficult.

--
Bryan Henderson                       IBM Almaden Research Center
San Jose CA                           Filesystems