RE: [nfsv4] RE: Finding hardlinks

RE: [nfsv4] RE: Finding hardlinks

[Noveck, Dave]

For now, I'm not going to address the controversial issues here,
mainly because I haven't decided how I feel about them yet.

     Whether allowing multiple filehandles per object is a good
     or even reasonably acceptable idea.

     What the fact that RFC3530 talks about implies about what
     clients should do about the issue.

One thing that I hope is not controversial is that the v4.1 spec
should either get rid of this or make it clear and implementable.
I expect plenty of controversy about which of those to choose, but
hope that there isn't any about the proposition that we have to 
choose one of those two.

> SECINFO information is, for instance, given
> out on a per-filehandle basis, does that mean that the server will
have
> different security policies? 

Well yes, RFC3530 does say "The new SECINFO operation will allow the 
client to determine, on a per filehandle basis", but I think that
just has to be considered as an error rather than indicating that if
you have two different filehandles for the same object, they can have 
different security policies.  SECINFO in RFC3530 takes a directory fh
and a name, so if there are multiple filehandles for the object with
that name, there is no way for SECINFO to associate different policies
with different filehandles.  All it has is the name to go by.  I think
this should be corrected to "on a per-object basis" in the new spec no 
matter what we do on other issues.

I think the principle here has to be that if we do allow multiple 
fh's to map to the same object, we require that they designate the 
same object, and thus it is not allowed for the server to act as if 
you have multiple different object with different characteristics.

Similarly as to:

> In some places, people haven't even started
> to think about the consequences: 
>
>     If GETATTR directed to the two filehandles does not return the
>     fileid attribute for both of the handles, then it cannot be
>     determined whether the two objects are the same.  Therefore,
>     operations which depend on that knowledge (e.g., client side data
>     caching) cannot be done reliably.

I think they (and maybe "they" includes me, I haven't checked the
history
here) started to think about them, but went in a bad direction.

The implication here that you can have a different set of attributes
supported for the same object based on which filehandle is used to 
access the attributes is totally bogus.

The definition of supp_attr says "The bit vector which would retrieve
all mandatory and recommended attributes that are supported for this 
object.  The scope of this attribute applies to all objects with a
matching fsid."  So having the same object have different attributes
supported based on the filehandle used or even two objects in the same
fs having different attributes supported, in particular having fileid
supported for one and not the other just isn't valid.

> The fact is that RFC3530 contains masses of rope with which
> to allow server and client vendors to hang themselves. 

If that means simply making poor choices, then OK.  But if there are 
other cases where you feel that the specification of a feature is simply

incoherent and the consequences not really thought out, then I think 
we need to discuss them and not propagate that state of affairs to v4.1.

-----Original Message-----
From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no] 
Sent: Friday, January 05, 2007 5:29 AM
To: Benny Halevy
Cc: Jan Harkes; Miklos Szeredi; nfsv4@ietf.org;
linux-kernel@vger.kernel.org; Mikulas Patocka;
linux-fsdevel@vger.kernel.org; Jeff Layton; Arjan van de Ven
Subject: Re: [nfsv4] RE: Finding hardlinks


On Fri, 2007-01-05 at 10:28 +0200, Benny Halevy wrote:
> Trond Myklebust wrote:
> > Exactly where do you see us violating the close-to-open cache
> > consistency guarantees?
> > 
> 
> I haven't seen that. What I did see is cache inconsistency when
opening
> the same file with different file descriptors when the filehandle
changes.
> My testing shows that at least fsync and close fail with EIO when the
filehandle
> changed while there was dirty data in the cache and that's good.
Still,
> not sharing the cache while the file is opened (even on a different
file
> descriptors by the same process) seems impractical.

Tough. I'm not going to commit to adding support for multiple
filehandles. The fact is that RFC3530 contains masses of rope with which
to allow server and client vendors to hang themselves. The fact that the
protocol claims support for servers that use multiple filehandles per
inode does not mean it is necessarily a good idea. It adds unnecessary
code complexity, it screws with server scalability (extra GETATTR calls
just in order to probe existing filehandles), and it is insufficiently
well documented in the RFC: SECINFO information is, for instance, given
out on a per-filehandle basis, does that mean that the server will have
different security policies? In some places, people haven't even started
to think about the consequences:

      If GETATTR directed to the two filehandles does not return the
      fileid attribute for both of the handles, then it cannot be
      determined whether the two objects are the same.  Therefore,
      operations which depend on that knowledge (e.g., client side data
      caching) cannot be done reliably.

This implies the combination is legal, but offers no indication as to
how you would match OPEN/CLOSE requests via different paths. AFAICS you
would have to do non-cached I/O with no share modes (i.e. NFSv3-style
"special" stateids). There is no way in hell we will ever support
non-cached I/O in NFS other than the special case of O_DIRECT.


...and no, I'm certainly not interested in "fixing" the RFC on this
point in any way other than getting this crap dropped from the spec. I
see no use for it at all.

Trond


_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

Re: RE: Finding hardlinks

[Benny Halevy]

How about discussing this topic in the upcoming Connectathon?

Benny

Noveck, Dave wrote:
> For now, I'm not going to address the controversial issues here,
> mainly because I haven't decided how I feel about them yet.
> 
>      Whether allowing multiple filehandles per object is a good
>      or even reasonably acceptable idea.
> 
>      What the fact that RFC3530 talks about implies about what
>      clients should do about the issue.
> 
> One thing that I hope is not controversial is that the v4.1 spec
> should either get rid of this or make it clear and implementable.
> I expect plenty of controversy about which of those to choose, but
> hope that there isn't any about the proposition that we have to 
> choose one of those two.
> 
>> SECINFO information is, for instance, given
>> out on a per-filehandle basis, does that mean that the server will
> have
>> different security policies? 
> 
> Well yes, RFC3530 does say "The new SECINFO operation will allow the 
> client to determine, on a per filehandle basis", but I think that
> just has to be considered as an error rather than indicating that if
> you have two different filehandles for the same object, they can have 
> different security policies.  SECINFO in RFC3530 takes a directory fh
> and a name, so if there are multiple filehandles for the object with
> that name, there is no way for SECINFO to associate different policies
> with different filehandles.  All it has is the name to go by.  I think
> this should be corrected to "on a per-object basis" in the new spec no 
> matter what we do on other issues.
> 
> I think the principle here has to be that if we do allow multiple 
> fh's to map to the same object, we require that they designate the 
> same object, and thus it is not allowed for the server to act as if 
> you have multiple different object with different characteristics.
> 
> Similarly as to:
> 
>> In some places, people haven't even started
>> to think about the consequences: 
>>
>>     If GETATTR directed to the two filehandles does not return the
>>     fileid attribute for both of the handles, then it cannot be
>>     determined whether the two objects are the same.  Therefore,
>>     operations which depend on that knowledge (e.g., client side data
>>     caching) cannot be done reliably.
> 
> I think they (and maybe "they" includes me, I haven't checked the
> history
> here) started to think about them, but went in a bad direction.
> 
> The implication here that you can have a different set of attributes
> supported for the same object based on which filehandle is used to 
> access the attributes is totally bogus.
> 
> The definition of supp_attr says "The bit vector which would retrieve
> all mandatory and recommended attributes that are supported for this 
> object.  The scope of this attribute applies to all objects with a
> matching fsid."  So having the same object have different attributes
> supported based on the filehandle used or even two objects in the same
> fs having different attributes supported, in particular having fileid
> supported for one and not the other just isn't valid.
> 
>> The fact is that RFC3530 contains masses of rope with which
>> to allow server and client vendors to hang themselves. 
> 
> If that means simply making poor choices, then OK.  But if there are 
> other cases where you feel that the specification of a feature is simply
> 
> incoherent and the consequences not really thought out, then I think 
> we need to discuss them and not propagate that state of affairs to v4.1.
> 
> -----Original Message-----
> From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no] 
> Sent: Friday, January 05, 2007 5:29 AM
> To: Benny Halevy
> Cc: Jan Harkes; Miklos Szeredi; nfsv4@ietf.org;
> linux-kernel@vger.kernel.org; Mikulas Patocka;
> linux-fsdevel@vger.kernel.org; Jeff Layton; Arjan van de Ven
> Subject: Re: [nfsv4] RE: Finding hardlinks
> 
> 
> On Fri, 2007-01-05 at 10:28 +0200, Benny Halevy wrote:
>> Trond Myklebust wrote:
>>> Exactly where do you see us violating the close-to-open cache
>>> consistency guarantees?
>>>
>> I haven't seen that. What I did see is cache inconsistency when
> opening
>> the same file with different file descriptors when the filehandle
> changes.
>> My testing shows that at least fsync and close fail with EIO when the
> filehandle
>> changed while there was dirty data in the cache and that's good.
> Still,
>> not sharing the cache while the file is opened (even on a different
> file
>> descriptors by the same process) seems impractical.
> 
> Tough. I'm not going to commit to adding support for multiple
> filehandles. The fact is that RFC3530 contains masses of rope with which
> to allow server and client vendors to hang themselves. The fact that the
> protocol claims support for servers that use multiple filehandles per
> inode does not mean it is necessarily a good idea. It adds unnecessary
> code complexity, it screws with server scalability (extra GETATTR calls
> just in order to probe existing filehandles), and it is insufficiently
> well documented in the RFC: SECINFO information is, for instance, given
> out on a per-filehandle basis, does that mean that the server will have
> different security policies? In some places, people haven't even started
> to think about the consequences:
> 
>       If GETATTR directed to the two filehandles does not return the
>       fileid attribute for both of the handles, then it cannot be
>       determined whether the two objects are the same.  Therefore,
>       operations which depend on that knowledge (e.g., client side data
>       caching) cannot be done reliably.
> 
> This implies the combination is legal, but offers no indication as to
> how you would match OPEN/CLOSE requests via different paths. AFAICS you
> would have to do non-cached I/O with no share modes (i.e. NFSv3-style
> "special" stateids). There is no way in hell we will ever support
> non-cached I/O in NFS other than the special case of O_DIRECT.
> 
> 
> ...and no, I'm certainly not interested in "fixing" the RFC on this
> point in any way other than getting this crap dropped from the spec. I
> see no use for it at all.
> 
> Trond
> 
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www1.ietf.org/mailman/listinfo/nfsv4


_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

RE: RE: Finding hardlinks

[Noveck, Dave]

I'm not going to be at Connectathon, but I could call in for a
discussion.=20

-----Original Message-----
From: Benny Halevy [mailto:bhalevy@panasas.com]=20
Sent: Monday, January 15, 2007 3:45 AM
To: Noveck, Dave; Trond Myklebust
Cc: Spencer Shepler; nfsv4@ietf.org; nfs@lists.sourceforge.net
Subject: Re: [nfsv4] RE: Finding hardlinks

How about discussing this topic in the upcoming Connectathon?

Benny

Noveck, Dave wrote:
> For now, I'm not going to address the controversial issues here,=20
> mainly because I haven't decided how I feel about them yet.
>=20
>      Whether allowing multiple filehandles per object is a good
>      or even reasonably acceptable idea.
>=20
>      What the fact that RFC3530 talks about implies about what
>      clients should do about the issue.
>=20
> One thing that I hope is not controversial is that the v4.1 spec=20
> should either get rid of this or make it clear and implementable.
> I expect plenty of controversy about which of those to choose, but=20
> hope that there isn't any about the proposition that we have to choose

> one of those two.
>=20
>> SECINFO information is, for instance, given out on a per-filehandle=20
>> basis, does that mean that the server will
> have
>> different security policies?=20
>=20
> Well yes, RFC3530 does say "The new SECINFO operation will allow the=20
> client to determine, on a per filehandle basis", but I think that just

> has to be considered as an error rather than indicating that if you=20
> have two different filehandles for the same object, they can have=20
> different security policies.  SECINFO in RFC3530 takes a directory fh=20
> and a name, so if there are multiple filehandles for the object with=20
> that name, there is no way for SECINFO to associate different policies

> with different filehandles.  All it has is the name to go by.  I think

> this should be corrected to "on a per-object basis" in the new spec no

> matter what we do on other issues.
>=20
> I think the principle here has to be that if we do allow multiple fh's

> to map to the same object, we require that they designate the same=20
> object, and thus it is not allowed for the server to act as if you=20
> have multiple different object with different characteristics.
>=20
> Similarly as to:
>=20
>> In some places, people haven't even started to think about the=20
>> consequences:
>>
>>     If GETATTR directed to the two filehandles does not return the
>>     fileid attribute for both of the handles, then it cannot be
>>     determined whether the two objects are the same.  Therefore,
>>     operations which depend on that knowledge (e.g., client side data
>>     caching) cannot be done reliably.
>=20
> I think they (and maybe "they" includes me, I haven't checked the=20
> history
> here) started to think about them, but went in a bad direction.
>=20
> The implication here that you can have a different set of attributes=20
> supported for the same object based on which filehandle is used to=20
> access the attributes is totally bogus.
>=20
> The definition of supp_attr says "The bit vector which would retrieve=20
> all mandatory and recommended attributes that are supported for this=20
> object.  The scope of this attribute applies to all objects with a=20
> matching fsid."  So having the same object have different attributes=20
> supported based on the filehandle used or even two objects in the same

> fs having different attributes supported, in particular having fileid=20
> supported for one and not the other just isn't valid.
>=20
>> The fact is that RFC3530 contains masses of rope with which to allow=20
>> server and client vendors to hang themselves.
>=20
> If that means simply making poor choices, then OK.  But if there are=20
> other cases where you feel that the specification of a feature is=20
> simply
>=20
> incoherent and the consequences not really thought out, then I think=20
> we need to discuss them and not propagate that state of affairs to
v4.1.
>=20
> -----Original Message-----
> From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no]
> Sent: Friday, January 05, 2007 5:29 AM
> To: Benny Halevy
> Cc: Jan Harkes; Miklos Szeredi; nfsv4@ietf.org;=20
> linux-kernel@vger.kernel.org; Mikulas Patocka;=20
> linux-fsdevel@vger.kernel.org; Jeff Layton; Arjan van de Ven
> Subject: Re: [nfsv4] RE: Finding hardlinks
>=20
>=20
> On Fri, 2007-01-05 at 10:28 +0200, Benny Halevy wrote:
>> Trond Myklebust wrote:
>>> Exactly where do you see us violating the close-to-open cache=20
>>> consistency guarantees?
>>>
>> I haven't seen that. What I did see is cache inconsistency when
> opening
>> the same file with different file descriptors when the filehandle
> changes.
>> My testing shows that at least fsync and close fail with EIO when the
> filehandle
>> changed while there was dirty data in the cache and that's good.
> Still,
>> not sharing the cache while the file is opened (even on a different
> file
>> descriptors by the same process) seems impractical.
>=20
> Tough. I'm not going to commit to adding support for multiple=20
> filehandles. The fact is that RFC3530 contains masses of rope with=20
> which to allow server and client vendors to hang themselves. The fact=20
> that the protocol claims support for servers that use multiple=20
> filehandles per inode does not mean it is necessarily a good idea. It=20
> adds unnecessary code complexity, it screws with server scalability=20
> (extra GETATTR calls just in order to probe existing filehandles), and

> it is insufficiently well documented in the RFC: SECINFO information=20
> is, for instance, given out on a per-filehandle basis, does that mean=20
> that the server will have different security policies? In some places,

> people haven't even started to think about the consequences:
>=20
>       If GETATTR directed to the two filehandles does not return the
>       fileid attribute for both of the handles, then it cannot be
>       determined whether the two objects are the same.  Therefore,
>       operations which depend on that knowledge (e.g., client side
data
>       caching) cannot be done reliably.
>=20
> This implies the combination is legal, but offers no indication as to=20
> how you would match OPEN/CLOSE requests via different paths. AFAICS=20
> you would have to do non-cached I/O with no share modes (i.e.=20
> NFSv3-style "special" stateids). There is no way in hell we will ever=20
> support non-cached I/O in NFS other than the special case of O_DIRECT.
>=20
>=20
> ...and no, I'm certainly not interested in "fixing" the RFC on this=20
> point in any way other than getting this crap dropped from the spec. I

> see no use for it at all.
>=20
> Trond
>=20
>=20
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

Finding hardlinks

[Mikulas Patocka]

Hi

I've came across this problem: how can a userspace program (such as for 
example "cp -a") tell that two files form a hardlink? Comparing inode 
number will break on filesystems that can have more than 2^32 files (NFS3, 
OCFS, SpadFS; kernel developers already implemented iget5_locked for the 
case of colliding inode numbers). Other possibilities:

--- compare not only ino, but all stat entries and make sure that
 	i_nlink > 1?
 	--- is not 100% reliable either, only lowers failure probability
--- create a hardlink and watch if i_nlink is increased on both files?
 	--- doesn't work on read-only filesystems
--- compare file content?
 	--- "cp -a" won't then corrupt data at least, but will create
 	hardlinks where they shouldn't be.

Is there some reliable way how should "cp -a" command determine that? 
Finding in kernel whether two dentries point to the same inode is trivial 
but I am not sure how to let userspace know ... am I missing something?

Mikulas

Re: Finding hardlinks

[Miklos Szeredi]

> I've came across this problem: how can a userspace program (such as for 
> example "cp -a") tell that two files form a hardlink? Comparing inode 
> number will break on filesystems that can have more than 2^32 files (NFS3, 
> OCFS, SpadFS; kernel developers already implemented iget5_locked for the 
> case of colliding inode numbers). Other possibilities:
> 
> --- compare not only ino, but all stat entries and make sure that
>  	i_nlink > 1?
>  	--- is not 100% reliable either, only lowers failure probability
> --- create a hardlink and watch if i_nlink is increased on both files?
>  	--- doesn't work on read-only filesystems
> --- compare file content?
>  	--- "cp -a" won't then corrupt data at least, but will create
>  	hardlinks where they shouldn't be.
> 
> Is there some reliable way how should "cp -a" command determine that? 
> Finding in kernel whether two dentries point to the same inode is trivial 
> but I am not sure how to let userspace know ... am I missing something?

The stat64.st_ino field is 64bit, so AFAICS you'd only need to extend
the kstat.ino field to 64bit and fix those filesystems to fill in
kstat correctly.

SUSv3 requires st_ino/st_dev to be unique within a system so the
application shouldn't need to bend over backwards.

Miklos

Re: Finding hardlinks

[Jan Harkes]

On Wed, Dec 20, 2006 at 12:44:42PM +0100, Miklos Szeredi wrote:
> The stat64.st_ino field is 64bit, so AFAICS you'd only need to extend
> the kstat.ino field to 64bit and fix those filesystems to fill in
> kstat correctly.

Coda actually uses 128-bit file identifiers internally, so 64-bits
really doesn't cut it. Since the 128-bit space is used pretty sparsely
there is a hash which avoids most collistions in 32-bit i_ino space, but
not completely. I can also imagine that at some point someone wants to
implement a git-based filesystem where it would be more natural to use
160-bit SHA1 hashes as unique object identifiers.

But Coda only allow hardlinks within a single directory and if someone
renames a hardlinked file and one of the names ends up in a different
directory we implicitly create a copy of the object. This actually
leverages off of the way we handle volume snapshots and the fact that we
use whole file caching and writes, so we only copy the metadata while
the data is 'copy-on-write'.

I'm considering changing the way we handle hardlinks by having link(2)
always create a new object with copy-on-write semantics (i.e. replacing
link with some sort of a copyfile operation). This way we can get rid of
several special cases like the cross-directory rename. It also avoids
problems when the various replicas of an object are found to be
inconsistent and we allow the user to expand the file. On expansion a
file becomes a directory that contains all the objects on individual
replicas. Handling the expansion in a dcache friendly way is nasty
enough as is and complicated by the fact that we really don't want such
an expansion to result in hard-linked directories, so we are forced to
inventing new unique object identifiers, etc. Again, not having
hardlinks would simplify things somewhat here.

Any application that tries to be smart enough to keep track of which
files are hardlinked should (in my opinion) also have a way to disable
this behaviour.

Jan

Re: Finding hardlinks

[Mikulas Patocka]

On Thu, 21 Dec 2006, Jan Harkes wrote:

> On Wed, Dec 20, 2006 at 12:44:42PM +0100, Miklos Szeredi wrote:
>> The stat64.st_ino field is 64bit, so AFAICS you'd only need to extend
>> the kstat.ino field to 64bit and fix those filesystems to fill in
>> kstat correctly.
>
> Coda actually uses 128-bit file identifiers internally, so 64-bits
> really doesn't cut it. Since the 128-bit space is used pretty sparsely
> there is a hash which avoids most collistions in 32-bit i_ino space, but
> not completely. I can also imagine that at some point someone wants to
> implement a git-based filesystem where it would be more natural to use
> 160-bit SHA1 hashes as unique object identifiers.
>
> But Coda only allow hardlinks within a single directory and if someone
> renames a hardlinked file and one of the names ends up in a different
> directory we implicitly create a copy of the object. This actually
> leverages off of the way we handle volume snapshots and the fact that we
> use whole file caching and writes, so we only copy the metadata while
> the data is 'copy-on-write'.

The problem is that if inode number collision happens occasionally, you 
get data corruption with cp -a command --- it will just copy one file and 
hardlink the other.

> Any application that tries to be smart enough to keep track of which
> files are hardlinked should (in my opinion) also have a way to disable
> this behaviour.

If user (or script) doesn't specify that flag, it doesn't help. I think 
the best solution for these filesystems would be either to add new syscall
 	int is_hardlink(char *filename1, char *filename2)
(but I know adding syscall bloat may be objectionable)
or add new field in statvfs ST_HAS_BROKEN_INO_T, that applications can 
test and disable hardlink processing.

Mikulas

> Jan
>

Re: Finding hardlinks

[Arjan van de Ven]

> 
> If user (or script) doesn't specify that flag, it doesn't help. I think 
> the best solution for these filesystems would be either to add new syscall
>  	int is_hardlink(char *filename1, char *filename2)
> (but I know adding syscall bloat may be objectionable)

it's also the wrong api; the filenames may have been changed under you
just as you return from this call, so it really is a
"was_hardlink_at_some_point()" as you specify it.
If you make it work on fd's.. it has a chance at least.

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: Finding hardlinks

[Mikulas Patocka]

>> If user (or script) doesn't specify that flag, it doesn't help. I think
>> the best solution for these filesystems would be either to add new syscall
>>  	int is_hardlink(char *filename1, char *filename2)
>> (but I know adding syscall bloat may be objectionable)
>
> it's also the wrong api; the filenames may have been changed under you
> just as you return from this call, so it really is a
> "was_hardlink_at_some_point()" as you specify it.
> If you make it work on fd's.. it has a chance at least.

Yes, but it doesn't matter --- if the tree changes under "cp -a" command, 
no one guarantees you what you get.
 	int fis_hardlink(int handle1, int handle 2);
Is another possibility but it can't detect hardlinked symlinks.

Mikulas

Re: Finding hardlinks

[Benny Halevy]

Mikulas Patocka wrote:
>>> If user (or script) doesn't specify that flag, it doesn't help. I think
>>> the best solution for these filesystems would be either to add new syscall
>>>  	int is_hardlink(char *filename1, char *filename2)
>>> (but I know adding syscall bloat may be objectionable)
>> it's also the wrong api; the filenames may have been changed under you
>> just as you return from this call, so it really is a
>> "was_hardlink_at_some_point()" as you specify it.
>> If you make it work on fd's.. it has a chance at least.
> 
> Yes, but it doesn't matter --- if the tree changes under "cp -a" command, 
> no one guarantees you what you get.
>  	int fis_hardlink(int handle1, int handle 2);
> Is another possibility but it can't detect hardlinked symlinks.

It seems like the posix idea of unique <st_dev, st_ino> doesn't
hold water for modern file systems and that creates real problems for
backup apps which rely on that to detect hard links.

Adding a vfs call to check for file equivalence seems like a good idea to me.
A syscall exposing it to user mode apps can look like what you sketched above,
and another variant of it can maybe take two paths and possibly a flags field
(for e.g. don't follow symlinks).

I'm cross-posting this also to nfsv4@ietf. NFS has exactly the same problem
with <fsid, fileid> as fileid is 64 bit wide. Although the nfs client can
determine that two filesystem objects are hard linked if they have the same
filehandle but there are cases where two distinct filehandles can still refer to
the same filesystem object.  Letting the nfs client determine file equivalency
based on filehandles will probably satisfy most users but if the exported
fs supports the new call discussed above, exporting it over NFS makes a
lot of sense to me... What do you guys think about adding such an operation
to NFS?

Benny

> 
> Mikulas
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Finding hardlinks

[Jeff Layton]

Benny Halevy wrote:
> 
> It seems like the posix idea of unique <st_dev, st_ino> doesn't
> hold water for modern file systems and that creates real problems for
> backup apps which rely on that to detect hard links.
> 

Why not? Granted, many of the filesystems in the Linux kernel don't enforce that 
they have unique st_ino values, but I'm working on a set of patches to try and 
fix that.

> Adding a vfs call to check for file equivalence seems like a good idea to me.
> A syscall exposing it to user mode apps can look like what you sketched above,
> and another variant of it can maybe take two paths and possibly a flags field
> (for e.g. don't follow symlinks).
> 
> I'm cross-posting this also to nfsv4@ietf. NFS has exactly the same problem
> with <fsid, fileid> as fileid is 64 bit wide. Although the nfs client can
> determine that two filesystem objects are hard linked if they have the same
> filehandle but there are cases where two distinct filehandles can still refer to
> the same filesystem object.  Letting the nfs client determine file equivalency
> based on filehandles will probably satisfy most users but if the exported
> fs supports the new call discussed above, exporting it over NFS makes a
> lot of sense to me... What do you guys think about adding such an operation
> to NFS?
> 

This sounds like a bug to me. It seems like we should have a one to one 
correspondence of filehandle -> inode. In what situations would this not be the 
case?

-- Jeff

Re: Finding hardlinks

[Benny Halevy]

Jeff Layton wrote:
> Benny Halevy wrote:
>> It seems like the posix idea of unique <st_dev, st_ino> doesn't
>> hold water for modern file systems and that creates real problems for
>> backup apps which rely on that to detect hard links.
>>
> 
> Why not? Granted, many of the filesystems in the Linux kernel don't enforce that 
> they have unique st_ino values, but I'm working on a set of patches to try and 
> fix that.

That's great and will surely help most file systems (apparently not Coda as
Jan says they use 128 bit internal file identifiers).

What about 32 bit architectures? Is ino_t going to be 64 bit
there too?

> 
>> Adding a vfs call to check for file equivalence seems like a good idea to me.
>> A syscall exposing it to user mode apps can look like what you sketched above,
>> and another variant of it can maybe take two paths and possibly a flags field
>> (for e.g. don't follow symlinks).
>>
>> I'm cross-posting this also to nfsv4@ietf. NFS has exactly the same problem
>> with <fsid, fileid> as fileid is 64 bit wide. Although the nfs client can
>> determine that two filesystem objects are hard linked if they have the same
>> filehandle but there are cases where two distinct filehandles can still refer to
>> the same filesystem object.  Letting the nfs client determine file equivalency
>> based on filehandles will probably satisfy most users but if the exported
>> fs supports the new call discussed above, exporting it over NFS makes a
>> lot of sense to me... What do you guys think about adding such an operation
>> to NFS?
>>
> 
> This sounds like a bug to me. It seems like we should have a one to one 
> correspondence of filehandle -> inode. In what situations would this not be the 
> case?

Well, the NFS protocol allows that [see rfc1813, p. 21: "If two file handles from
the same server are equal, they must refer to the same file, but if they are not
equal, no conclusions can be drawn."]

As an example, some file systems encode hint information into the filehandle
and the hints may change over time, another example is encoding parent
information into the filehandle and then handles representing hard links
to the same file from different directories will differ.

> 
> -- Jeff
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Finding hardlinks

[Mikulas Patocka]

>> This sounds like a bug to me. It seems like we should have a one to one
>> correspondence of filehandle -> inode. In what situations would this not be the
>> case?
>
> Well, the NFS protocol allows that [see rfc1813, p. 21: "If two file handles from
> the same server are equal, they must refer to the same file, but if they are not
> equal, no conclusions can be drawn."]
>
> As an example, some file systems encode hint information into the filehandle
> and the hints may change over time, another example is encoding parent
> information into the filehandle and then handles representing hard links
> to the same file from different directories will differ.

BTW. how does (or how should?) NFS client deal with cache coherency if 
filehandles for the same file differ?

Mikulas

RE: Finding hardlinks

[Halevy, Benny]

Mikulas Patocka wrote:
> 
>>> This sounds like a bug to me. It seems like we should have a one to one
>>> correspondence of filehandle -> inode. In what situations would this not be the
>>> case?
>>
>> Well, the NFS protocol allows that [see rfc1813, p. 21: "If two file handles from
>> the same server are equal, they must refer to the same file, but if they are not
>> equal, no conclusions can be drawn."]
>>
>> As an example, some file systems encode hint information into the filehandle
>> and the hints may change over time, another example is encoding parent
>> information into the filehandle and then handles representing hard links
>> to the same file from different directories will differ.
>
>BTW. how does (or how should?) NFS client deal with cache coherency if 
>filehandles for the same file differ?
>

Trond can probably answer this better than me...
As I read it, currently the nfs client matches both the fileid and the
filehandle (in nfs_find_actor). This means that different filehandles
for the same file would result in different inodes :(.
Strictly following the nfs protocol, comparing only the fileid should
be enough IF fileids are indeed unique within the filesystem.
Comparing the filehandle works as a workaround when the exported filesystem
(or the nfs server) violates that.  From a user stand point I think that
this should be configurable, probably per mount point.

>Mikulas
>

Re: [nfsv4] RE: Finding hardlinks

[Trond Myklebust]

On Thu, 2006-12-28 at 15:07 -0500, Halevy, Benny wrote:
> Mikulas Patocka wrote:

> >BTW. how does (or how should?) NFS client deal with cache coherency if 
> >filehandles for the same file differ?
> >
> 
> Trond can probably answer this better than me...
> As I read it, currently the nfs client matches both the fileid and the
> filehandle (in nfs_find_actor). This means that different filehandles
> for the same file would result in different inodes :(.
> Strictly following the nfs protocol, comparing only the fileid should
> be enough IF fileids are indeed unique within the filesystem.
> Comparing the filehandle works as a workaround when the exported filesystem
> (or the nfs server) violates that.  From a user stand point I think that
> this should be configurable, probably per mount point.

Matching files by fileid instead of filehandle is a lot more trouble
since fileids may be reused after a file has been deleted. Every time
you look up a file, and get a new filehandle for the same fileid, you
would at the very least have to do another GETATTR using one of the
'old' filehandles in order to ensure that the file is the same object as
the one you have cached. Then there is the issue of what to do when you
open(), read() or write() to the file: which filehandle do you use, are
the access permissions the same for all filehandles, ...

All in all, much pain for little or no gain.

Most servers therefore take great pains to ensure that clients can use
filehandles to identify inodes. The exceptions tend to be broken in
other ways (Note: knfsd without the no_subtree_check option is one of
these exceptions - it can break in the case of cross-directory renames).

Cheers,
  Trond

RE: RE: Finding hardlinks

[Halevy, Benny]

Trond Myklebust wrote:
>  
> On Thu, 2006-12-28 at 15:07 -0500, Halevy, Benny wrote:
> > Mikulas Patocka wrote:
> 
> > >BTW. how does (or how should?) NFS client deal with cache coherency if 
> > >filehandles for the same file differ?
> > >
> > 
> > Trond can probably answer this better than me...
> > As I read it, currently the nfs client matches both the fileid and the
> > filehandle (in nfs_find_actor). This means that different filehandles
> > for the same file would result in different inodes :(.
> > Strictly following the nfs protocol, comparing only the fileid should
> > be enough IF fileids are indeed unique within the filesystem.
> > Comparing the filehandle works as a workaround when the exported filesystem
> > (or the nfs server) violates that.  From a user stand point I think that
> > this should be configurable, probably per mount point.
> 
> Matching files by fileid instead of filehandle is a lot more trouble
> since fileids may be reused after a file has been deleted. Every time
> you look up a file, and get a new filehandle for the same fileid, you
> would at the very least have to do another GETATTR using one of the
> 'old' filehandles in order to ensure that the file is the same object as
> the one you have cached. Then there is the issue of what to do when you
> open(), read() or write() to the file: which filehandle do you use, are
> the access permissions the same for all filehandles, ...
> 
> All in all, much pain for little or no gain.

See my answer to your previous reply.  It seems like the current
implementation is in violation of the nfs protocol and the extra pain
is required.

> 
> Most servers therefore take great pains to ensure that clients can use
> filehandles to identify inodes. The exceptions tend to be broken in
> other ways

This is true maybe in linux, but not necessarily in non-linux based nfs
servers.

> (Note: knfsd without the no_subtree_check option is one of
> these exceptions - it can break in the case of cross-directory renames).
> 
> Cheers,
>   Trond



_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

RE: RE: Finding hardlinks

[Trond Myklebust]

On Sun, 2006-12-31 at 16:25 -0500, Halevy, Benny wrote:
> Trond Myklebust wrote:
> >  
> > On Thu, 2006-12-28 at 15:07 -0500, Halevy, Benny wrote:
> > > Mikulas Patocka wrote:
> > 
> > > >BTW. how does (or how should?) NFS client deal with cache coherency if 
> > > >filehandles for the same file differ?
> > > >
> > > 
> > > Trond can probably answer this better than me...
> > > As I read it, currently the nfs client matches both the fileid and the
> > > filehandle (in nfs_find_actor). This means that different filehandles
> > > for the same file would result in different inodes :(.
> > > Strictly following the nfs protocol, comparing only the fileid should
> > > be enough IF fileids are indeed unique within the filesystem.
> > > Comparing the filehandle works as a workaround when the exported filesystem
> > > (or the nfs server) violates that.  From a user stand point I think that
> > > this should be configurable, probably per mount point.
> > 
> > Matching files by fileid instead of filehandle is a lot more trouble
> > since fileids may be reused after a file has been deleted. Every time
> > you look up a file, and get a new filehandle for the same fileid, you
> > would at the very least have to do another GETATTR using one of the
> > 'old' filehandles in order to ensure that the file is the same object as
> > the one you have cached. Then there is the issue of what to do when you
> > open(), read() or write() to the file: which filehandle do you use, are
> > the access permissions the same for all filehandles, ...
> > 
> > All in all, much pain for little or no gain.
> 
> See my answer to your previous reply.  It seems like the current
> implementation is in violation of the nfs protocol and the extra pain
> is required.

...and we should care because...?

Trond


_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

Re: RE: Finding hardlinks

[Benny Halevy]

Trond Myklebust wrote:
> On Sun, 2006-12-31 at 16:25 -0500, Halevy, Benny wrote:
>> Trond Myklebust wrote:
>>>  
>>> On Thu, 2006-12-28 at 15:07 -0500, Halevy, Benny wrote:
>>>> Mikulas Patocka wrote:
>>>>> BTW. how does (or how should?) NFS client deal with cache coherency if 
>>>>> filehandles for the same file differ?
>>>>>
>>>> Trond can probably answer this better than me...
>>>> As I read it, currently the nfs client matches both the fileid and the
>>>> filehandle (in nfs_find_actor). This means that different filehandles
>>>> for the same file would result in different inodes :(.
>>>> Strictly following the nfs protocol, comparing only the fileid should
>>>> be enough IF fileids are indeed unique within the filesystem.
>>>> Comparing the filehandle works as a workaround when the exported filesystem
>>>> (or the nfs server) violates that.  From a user stand point I think that
>>>> this should be configurable, probably per mount point.
>>> Matching files by fileid instead of filehandle is a lot more trouble
>>> since fileids may be reused after a file has been deleted. Every time
>>> you look up a file, and get a new filehandle for the same fileid, you
>>> would at the very least have to do another GETATTR using one of the
>>> 'old' filehandles in order to ensure that the file is the same object as
>>> the one you have cached. Then there is the issue of what to do when you
>>> open(), read() or write() to the file: which filehandle do you use, are
>>> the access permissions the same for all filehandles, ...
>>>
>>> All in all, much pain for little or no gain.
>> See my answer to your previous reply.  It seems like the current
>> implementation is in violation of the nfs protocol and the extra pain
>> is required.
> 
> ...and we should care because...?
> 
> Trond
> 

Believe it or not, but server companies like Panasas try to follow the standard
when designing and implementing their products while relying on client vendors
to do the same.

I sincerely expect you or anybody else for this matter to try to provide
feedback and object to the protocol specification in case they disagree
with it (or think it's ambiguous or self contradicting) rather than ignoring
it and implementing something else. I think we're shooting ourselves in the
foot when doing so and it is in our common interest to strive to reach a
realistic standard we can all comply with and interoperate with each other.

Benny


_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

Re: [nfsv4] RE: Finding hardlinks

[Trond Myklebust]

On Wed, 2007-01-03 at 14:35 +0200, Benny Halevy wrote:
> I sincerely expect you or anybody else for this matter to try to provide
> feedback and object to the protocol specification in case they disagree
> with it (or think it's ambiguous or self contradicting) rather than ignoring
> it and implementing something else. I think we're shooting ourselves in the
> foot when doing so and it is in our common interest to strive to reach a
> realistic standard we can all comply with and interoperate with each other.

You are reading the protocol wrong in this case.

While the protocol does allow the server to implement the behaviour that
you've been advocating, it in no way mandates it. Nor does it mandate
that the client should gather files with the same (fsid,fileid) and
cache them together. Those are issues to do with _implementation_, and
are thus beyond the scope of the IETF.

In our case, the client will ignore the unique_handles attribute. It
will use filehandles as our inode cache identifier. It will not jump
through hoops to provide caching semantics that go beyond close-to-open
for servers that set unique_handles to "false".

Trond

Re: RE: Finding hardlinks

[Benny Halevy]

Trond Myklebust wrote:
> On Wed, 2007-01-03 at 14:35 +0200, Benny Halevy wrote:
>> I sincerely expect you or anybody else for this matter to try to provide
>> feedback and object to the protocol specification in case they disagree
>> with it (or think it's ambiguous or self contradicting) rather than ignoring
>> it and implementing something else. I think we're shooting ourselves in the
>> foot when doing so and it is in our common interest to strive to reach a
>> realistic standard we can all comply with and interoperate with each other.
> 
> You are reading the protocol wrong in this case.

Obviously we interpret it differently and that by itself calls for considering
clarification of the text :)

> 
> While the protocol does allow the server to implement the behaviour that
> you've been advocating, it in no way mandates it. Nor does it mandate
> that the client should gather files with the same (fsid,fileid) and
> cache them together. Those are issues to do with _implementation_, and
> are thus beyond the scope of the IETF.
> 
> In our case, the client will ignore the unique_handles attribute. It
> will use filehandles as our inode cache identifier. It will not jump
> through hoops to provide caching semantics that go beyond close-to-open
> for servers that set unique_handles to "false".

I agree that the way the client implements its cache is out of the protocol
scope. But how do you interpret "correct behavior" in section 4.2.1?
 "Clients MUST use filehandle comparisons only to improve performance, not for correct behavior. All clients need to be prepared for situations in which it cannot be determined whether two filehandles denote the same object and in such cases, avoid making invalid assumptions which might cause incorrect behavior."
Don't you consider data corruption due to cache inconsistency an incorrect behavior?

Benny

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4