XATTRs in NFS?

XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

Hi.

I just wondered about the current situation (respectively plans)
regarding XATTRs in NFS (I personally don't care much about the
version,... of course v4 would be nice, but v3 should do either).


I'm aware about the patches for v3 that James Morris has submitted some
time ago, but AFAICS these were never accepted/merged.

Asking him (off-list) he mentioned that any such work would need to take
place in NFS4.


I've seen some comments from the past, saying that it should never
happen since it would cause performance issues - but I guess one could
make it optional.


I have some use case where I need a network filesystem with XATTR
support providing the features of most standard filesystems (locking,
etc.).
And I'm sure there are plenty more use-cases.


So are there any plans to get this into NFS?


Cheers,
Chris.

btw: please keep me CCed.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 23, 2013, at 9:37 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> Hi.
> 
> I just wondered about the current situation (respectively plans)
> regarding XATTRs in NFS (I personally don't care much about the
> version,... of course v4 would be nice, but v3 should do either).
> 
> 
> I'm aware about the patches for v3 that James Morris has submitted some
> time ago, but AFAICS these were never accepted/merged.
> 
> Asking him (off-list) he mentioned that any such work would need to take
> place in NFS4.
> 
> 
> I've seen some comments from the past, saying that it should never
> happen since it would cause performance issues - but I guess one could
> make it optional.
> 
> 
> I have some use case where I need a network filesystem with XATTR
> support providing the features of most standard filesystems (locking,
> etc.).
> And I'm sure there are plenty more use-cases.
> 
> 
> So are there any plans to get this into NFS?
> 


labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.

There are no plans to merge general purpose xattrs. Please just use an application-specific database.

Cheers
   Trond

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.
Sure, but that doesn't really help me.


> There are no plans to merge general purpose xattrs.
Why not? Is it a big deal?


>  Please just use an application-specific database.
Well that won't work,... since that wouldn't be updated if e.g.
pathnames are changed by any program (cp, mv)


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
>> labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.
> Sure, but that doesn't really help me.
> 
> 
>> There are no plans to merge general purpose xattrs.
> Why not? Is it a big deal?
> 

Linux xattrs are a rabid mess.

The whole "system" namespace is something that cannot and should not ever be exposed on a network.
The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?

> 
>> Please just use an application-specific database.
> Well that won't work,... since that wouldn't be updated if e.g.
> pathnames are changed by any program (cp, mv)

If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?

Cheers
  Trond

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi,

Presumably Linux client and server implementations would restrict use of "system."
Perhaps would be more to talk about, if xattrs were implemented experimentally.

I'm interested in knowing about patches that might be floating around for the Linux client, for NFSv4.x.  (I've seen NFSv3 work.)

Thanks,

Matt

----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:

> On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer
> <calestyo@scientia.net> wrote:
> 
> > On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> >> labeled NFS (i.e. security labels for NFS) is already supported in
> Linux 3.10 and newer.
> > Sure, but that doesn't really help me.
> > 
> > 
> >> There are no plans to merge general purpose xattrs.
> > Why not? Is it a big deal?
> > 
> 
> Linux xattrs are a rabid mess.
> 
> The whole "system" namespace is something that cannot and should not
> ever be exposed on a network.
> The "trusted" and "user" namespaces just offer specialised storage.
> Why are they needed?
> 
> > 
> >> Please just use an application-specific database.
> > Well that won't work,... since that wouldn't be updated if e.g.
> > pathnames are changed by any program (cp, mv)
> 
> If the data needs to follow the file, then store it in the file. Why
> do you need the filesystem to manage that for you?
> 
> Cheers
>   Trond--
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[Simo Sorce]

On Thu, 2013-10-24 at 14:32 +0000, Myklebust, Trond wrote:
> On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:
> 
> > On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> >> labeled NFS (i.e. security labels for NFS) is already supported in Linux 3.10 and newer.
> > Sure, but that doesn't really help me.
> > 
> > 
> >> There are no plans to merge general purpose xattrs.
> > Why not? Is it a big deal?
> > 
> 
> Linux xattrs are a rabid mess.

First time I hear this :)

> The whole "system" namespace is something that cannot and should not ever be exposed on a network.
> The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?

Samba for example stores various metadata bits of information there (DOS
bits, original ACLs before posix translation, etc..).
Loosing that data on an mv via NFS breaks stuff.
That said samba and NFS have other synchronization issues, so it is not
the best example, but you can't just discount xattrs. They exist, they
are used, and if you don't have support for them in NFS you are not
transparent to applications.
 
> >> Please just use an application-specific database.
> > Well that won't work,... since that wouldn't be updated if e.g.
> > pathnames are changed by any program (cp, mv)
> 
> If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?

Because the filesystem can do that when multiple applications are
involved without having to change them all to talk to each other and
invent custom protocol all the time just to keep some additional
metadata associated to a file..

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

Re: XATTRs in NFS?

[Myklebust, Trond]

On Thu, 2013-10-24 at 11:05 -0400, Matt W. Benjamin wrote:
> Hi,
> 
> Presumably Linux client and server implementations would restrict use of "system."
> Perhaps would be more to talk about, if xattrs were implemented experimentally.
> 
> I'm interested in knowing about patches that might be floating around for the Linux client, for NFSv4.x.  (I've seen NFSv3 work.)
> 
> Thanks,
> 
> Matt

We've already talked about this, Matt. The answer is still NO.

Trond

> ----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:
> 
> > On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer
> > <calestyo@scientia.net> wrote:
> > 
> > > On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> > >> labeled NFS (i.e. security labels for NFS) is already supported in
> > Linux 3.10 and newer.
> > > Sure, but that doesn't really help me.
> > > 
> > > 
> > >> There are no plans to merge general purpose xattrs.
> > > Why not? Is it a big deal?
> > > 
> > 
> > Linux xattrs are a rabid mess.
> > 
> > The whole "system" namespace is something that cannot and should not
> > ever be exposed on a network.
> > The "trusted" and "user" namespaces just offer specialised storage.
> > Why are they needed?
> > 
> > > 
> > >> Please just use an application-specific database.
> > > Well that won't work,... since that wouldn't be updated if e.g.
> > > pathnames are changed by any program (cp, mv)
> > 
> > If the data needs to follow the file, then store it in the file. Why
> > do you need the filesystem to manage that for you?
> > 
> > Cheers
> >   Trond--
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi,

Sorry I was not clear.  I was not re-stating support for this banned feature.

I'm interested in implementation code I can use privately under GPLv2.

Thanks!

Matt

----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:

> On Thu, 2013-10-24 at 11:05 -0400, Matt W. Benjamin wrote:
> > Hi,
> > 
> > Presumably Linux client and server implementations would restrict
> use of "system."
> > Perhaps would be more to talk about, if xattrs were implemented
> experimentally.
> > 
> > I'm interested in knowing about patches that might be floating
> around for the Linux client, for NFSv4.x.  (I've seen NFSv3 work.)
> > 
> > Thanks,
> > 
> > Matt
> 
> We've already talked about this, Matt. The answer is still NO.
> 
> Trond
> 
> > ----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:
> > 
> > > On Oct 24, 2013, at 3:13 PM, Christoph Anton Mitterer
> > > <calestyo@scientia.net> wrote:
> > > 
> > > > On Thu, 2013-10-24 at 08:45 +0000, Myklebust, Trond wrote:
> > > >> labeled NFS (i.e. security labels for NFS) is already supported
> in
> > > Linux 3.10 and newer.
> > > > Sure, but that doesn't really help me.
> > > > 
> > > > 
> > > >> There are no plans to merge general purpose xattrs.
> > > > Why not? Is it a big deal?
> > > > 
> > > 
> > > Linux xattrs are a rabid mess.
> > > 
> > > The whole "system" namespace is something that cannot and should
> not
> > > ever be exposed on a network.
> > > The "trusted" and "user" namespaces just offer specialised
> storage.
> > > Why are they needed?
> > > 
> > > > 
> > > >> Please just use an application-specific database.
> > > > Well that won't work,... since that wouldn't be updated if e.g.
> > > > pathnames are changed by any program (cp, mv)
> > > 
> > > If the data needs to follow the file, then store it in the file.
> Why
> > > do you need the filesystem to manage that for you?
> > > 
> > > Cheers
> > >   Trond--
> > > To unsubscribe from this list: send the line "unsubscribe
> linux-nfs"
> > > in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> > 
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[Myklebust, Trond]

On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:

> Because the filesystem can do that when multiple applications are
> involved without having to change them all to talk to each other and
> invent custom protocol all the time just to keep some additional
> metadata associated to a file..
> 
It's still a custom protocol. The applications need to agree on a data
format and store it somewhere. The portable way to do this is to write
an application library that they can link to.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: XATTRs in NFS?

[Simo Sorce]

On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
> On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
> 
> > Because the filesystem can do that when multiple applications are
> > involved without having to change them all to talk to each other and
> > invent custom protocol all the time just to keep some additional
> > metadata associated to a file..
> > 
> It's still a custom protocol. The applications need to agree on a data
> format and store it somewhere. The portable way to do this is to write
> an application library that they can link to.

Perhaps I was unclear, you are never going to see that custom library
linked into the 'mv' command.

So your approach makes little sense if the object is to maintain data
coherent when people need to handle files from random applications and
scripts and general system maintenance.

The data may be relevant only to a specific application.

I am not saying you *have* to implement xattrs support, just saying that
it is not a mere 'applications should synchronize data themselves'
problem.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

Re: XATTRs in NFS?

[Jeff Layton]

On Thu, 24 Oct 2013 11:16:10 -0400
Simo Sorce <simo@redhat.com> wrote:

> On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
> > On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
> > 
> > > Because the filesystem can do that when multiple applications are
> > > involved without having to change them all to talk to each other and
> > > invent custom protocol all the time just to keep some additional
> > > metadata associated to a file..
> > > 
> > It's still a custom protocol. The applications need to agree on a data
> > format and store it somewhere. The portable way to do this is to write
> > an application library that they can link to.
> 
> Perhaps I was unclear, you are never going to see that custom library
> linked into the 'mv' command.
> 
> So your approach makes little sense if the object is to maintain data
> coherent when people need to handle files from random applications and
> scripts and general system maintenance.
> 
> The data may be relevant only to a specific application.
> 
> I am not saying you *have* to implement xattrs support, just saying that
> it is not a mere 'applications should synchronize data themselves'
> problem.
> 

I think the real solution if people need this is to lead an effort to
put xattrs into the spec. I think there is still time to get new
features into v4.3 if someone wants to champion it...

-- 
Jeff Layton <jlayton@redhat.com>

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 24, 2013, at 4:16 PM, Simo Sorce <simo@redhat.com>
 wrote:

> On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
>> On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
>> 
>>> Because the filesystem can do that when multiple applications are
>>> involved without having to change them all to talk to each other and
>>> invent custom protocol all the time just to keep some additional
>>> metadata associated to a file..
>>> 
>> It's still a custom protocol. The applications need to agree on a data
>> format and store it somewhere. The portable way to do this is to write
>> an application library that they can link to.
> 
> Perhaps I was unclear, you are never going to see that custom library
> linked into the 'mv' command.
> 

Why should my mv need to link into such a library?

> So your approach makes little sense if the object is to maintain data
> coherent when people need to handle files from random applications and
> scripts and general system maintenance.
> 

See the earlier admonition: store data that needs to be kept together either in the same file, or in the same directory. Use a library when different applications need to access the same data.

> The data may be relevant only to a specific application.
> 
> I am not saying you *have* to implement xattrs support, just saying that
> it is not a mere 'applications should synchronize data themselves'
> problem.

_portable_ applications do not use xattrs. They are a Linuxism that is not described by either POSIX or any other similar standard.

Trond

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi,

It appears that there could be spec implications, but I have not so far been convinced that
nfsv4.x named attributes are not an appropriate vehicle for (some set of) extended attributes.

The fact that nfsv4.x named attributes are suitable to represent large objects, does not make
them unsuitable to represent key value properties.

Matt

----- "Jeff Layton" <jlayton@redhat.com> wrote:

> On Thu, 24 Oct 2013 11:16:10 -0400
> Simo Sorce <simo@redhat.com> wrote:
> 
> > On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
> > > On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
> > > 
> > > > Because the filesystem can do that when multiple applications
> are
> > > > involved without having to change them all to talk to each other
> and
> > > > invent custom protocol all the time just to keep some
> additional
> > > > metadata associated to a file..
> > > > 
> > > It's still a custom protocol. The applications need to agree on a
> data
> > > format and store it somewhere. The portable way to do this is to
> write
> > > an application library that they can link to.
> > 
> > Perhaps I was unclear, you are never going to see that custom
> library
> > linked into the 'mv' command.
> > 
> > So your approach makes little sense if the object is to maintain
> data
> > coherent when people need to handle files from random applications
> and
> > scripts and general system maintenance.
> > 
> > The data may be relevant only to a specific application.
> > 
> > I am not saying you *have* to implement xattrs support, just saying
> that
> > it is not a mere 'applications should synchronize data themselves'
> > problem.
> > 
> 
> I think the real solution if people need this is to lead an effort to
> put xattrs into the spec. I think there is still time to get new
> features into v4.3 if someone wants to champion it...
> 
> -- 
> Jeff Layton <jlayton@redhat.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[Myklebust, Trond]

On Thu, 2013-10-24 at 11:23 -0400, Jeff Layton wrote:
> On Thu, 24 Oct 2013 11:16:10 -0400
> Simo Sorce <simo@redhat.com> wrote:
> 
> > On Thu, 2013-10-24 at 15:11 +0000, Myklebust, Trond wrote:
> > > On Thu, 2013-10-24 at 11:07 -0400, Simo Sorce wrote:
> > > 
> > > > Because the filesystem can do that when multiple applications are
> > > > involved without having to change them all to talk to each other and
> > > > invent custom protocol all the time just to keep some additional
> > > > metadata associated to a file..
> > > > 
> > > It's still a custom protocol. The applications need to agree on a data
> > > format and store it somewhere. The portable way to do this is to write
> > > an application library that they can link to.
> > 
> > Perhaps I was unclear, you are never going to see that custom library
> > linked into the 'mv' command.
> > 
> > So your approach makes little sense if the object is to maintain data
> > coherent when people need to handle files from random applications and
> > scripts and general system maintenance.
> > 
> > The data may be relevant only to a specific application.
> > 
> > I am not saying you *have* to implement xattrs support, just saying that
> > it is not a mere 'applications should synchronize data themselves'
> > problem.
> > 
> 
> I think the real solution if people need this is to lead an effort to
> put xattrs into the spec. I think there is still time to get new
> features into v4.3 if someone wants to champion it...
> 

How would that help? Witness Oracle's success with named attributes,
which are _also_ a non-standard filesystem feature that was hastily
pushed into the NFSv4 spec.

If you really need this for use by applications (as opposed to by
sysadmins - see labeled NFS), then get the functionality into POSIX
first, then add it to the NFS spec.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

On Thu, 2013-10-24 at 14:32 +0000, Myklebust, Trond wrote:
> Linux xattrs are a rabid mess.
Well... might be from a technical POV, but for users they're quite
useful in some scenarios.


> The whole "system" namespace is something that cannot and should not ever be exposed on a network.
> The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?
Well what I do is attaching integrity information to files.

You may say now that this is similar to what btrfs will provide
anyway... but the problem with that is, that checksums are always
updated when something in the system does valid changes to the file.

What I however want is that I really manually have to set this, so that
I notice "accidental" changes, e.g. by myself or by buggy software...


> If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?
... and since this applies to arbitrary files, from text-files over
pictures, videos to binaries,... it's neither possible to store this in
the file, nor can I really track this with an database,... since
literally any program that uses such files, from the picture editor to
the file-manager would need to use such DB.


Cheers,
Chris.

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

On Thu, 2013-10-24 at 11:23 -0400, Jeff Layton wrote:
> I think the real solution if people need this is to lead an effort to
I'm often reading this argument,... "if people need this", respectively
the argument "XATTRs/ACLs" aren't widely used so we're not going to
support them.

IMHO that's no really an argument.... the simple reason that this isn't
widely used, is likely that there are still many places that don't
support it.

All the major Linux (disk) filesystems support it, right? ext*/btrfs/XFS
But many (even standard clients) do either not (e.g. normal GNU tar
format) or behave in a way that somehow break or lose XATTRs/ACLs (e.g.
vim),... NFS is another example - a very important IMHO since it is THE
network filesystem.

So sure,... not many people are using XATTRs, ACLs, but the reason is
simply, that there are still many obstacles in the way - in many cases
upstream refusing to solve these, for the reason that allegedly noone
uses XATTRs/ACLs.
(Which is btw. not fully true,... many modernish stuff like uses e.g.
ACLs like in /dev/).



I do agree that especially the ACLs from POSIX are not the best thing
(NFS4 ACLs are much better),... but I don't see the big problem with
key/value pairs from XATTRs.


Cheers,
Chris.

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 24, 2013, at 5:01 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> On Thu, 2013-10-24 at 14:32 +0000, Myklebust, Trond wrote:
>> Linux xattrs are a rabid mess.
> Well... might be from a technical POV, but for users they're quite
> useful in some scenarios.
> 
> 
>> The whole "system" namespace is something that cannot and should not ever be exposed on a network.
>> The "trusted" and "user" namespaces just offer specialised storage. Why are they needed?
> Well what I do is attaching integrity information to files.
> 
> You may say now that this is similar to what btrfs will provide
> anyway... but the problem with that is, that checksums are always
> updated when something in the system does valid changes to the file.
> 
> What I however want is that I really manually have to set this, so that
> I notice "accidental" changes, e.g. by myself or by buggy software...
> 
> 
>> If the data needs to follow the file, then store it in the file. Why do you need the filesystem to manage that for you?
> ... and since this applies to arbitrary files, from text-files over
> pictures, videos to binaries,... it's neither possible to store this in
> the file, nor can I really track this with an database,... since
> literally any program that uses such files, from the picture editor to
> the file-manager would need to use such DB.

Those programs need to recompute the checksum data anyway in order to verify and/or update it. Checksums that are computed by some third party application have exactly zero value for integrity checking.

Trond

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
> Those programs need to recompute the checksum data anyway in order to
> verify and/or update it. Checksums that are computed by some third
> party application have exactly zero value for integrity checking.
No that's exactly the point,... the applications should _NOT_ set those
checksums, especially not automagically (since then you'd never notice
when just some application is buggy or writes/modifies when you don't
expect it to do so).
The idea is that there is on application (in my case it's just a
script), which sets the integrity data and verifies it.

This works very well for e.g. large data archives, where you most of the
time (but not always) only read files, write new files or move around
existing ones - but only rarely modify existing file's contents.


I do this already like that on local filesystems, which works very
nicely with XATTRs... but now I want to move this on a central data
cluster (where clients connect to via NFS)... and here the problems
start... when I add new data to the archive (from the clients) I cannot
have XATTRs attached, nor can I verify them form the clients.



Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[J. Bruce Fields]

On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
> > Those programs need to recompute the checksum data anyway in order to
> > verify and/or update it. Checksums that are computed by some third
> > party application have exactly zero value for integrity checking.
> No that's exactly the point,... the applications should _NOT_ set those
> checksums, especially not automagically (since then you'd never notice
> when just some application is buggy or writes/modifies when you don't
> expect it to do so).
> The idea is that there is on application (in my case it's just a
> script), which sets the integrity data and verifies it.
> 
> This works very well for e.g. large data archives, where you most of the
> time (but not always) only read files, write new files or move around
> existing ones - but only rarely modify existing file's contents.
> 
> 
> I do this already like that on local filesystems, which works very
> nicely with XATTRs... but now I want to move this on a central data
> cluster (where clients connect to via NFS)... and here the problems
> start... when I add new data to the archive (from the clients) I cannot
> have XATTRs attached, nor can I verify them form the clients.

Can you give any more details about your use case?  (E.g. is there an
article describing this system somewhere?)

Just curious.--b.

Re: XATTRs in NFS?

[Ric Wheeler]

On 10/25/2013 10:08 AM, J. Bruce Fields wrote:
> On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
>> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
>>> Those programs need to recompute the checksum data anyway in order to
>>> verify and/or update it. Checksums that are computed by some third
>>> party application have exactly zero value for integrity checking.
>> No that's exactly the point,... the applications should _NOT_ set those
>> checksums, especially not automagically (since then you'd never notice
>> when just some application is buggy or writes/modifies when you don't
>> expect it to do so).
>> The idea is that there is on application (in my case it's just a
>> script), which sets the integrity data and verifies it.
>>
>> This works very well for e.g. large data archives, where you most of the
>> time (but not always) only read files, write new files or move around
>> existing ones - but only rarely modify existing file's contents.
>>
>>
>> I do this already like that on local filesystems, which works very
>> nicely with XATTRs... but now I want to move this on a central data
>> cluster (where clients connect to via NFS)... and here the problems
>> start... when I add new data to the archive (from the clients) I cannot
>> have XATTRs attached, nor can I verify them form the clients.
> Can you give any more details about your use case?  (E.g. is there an
> article describing this system somewhere?)
>
> Just curious.--b.
>

I think that having xattrs in NFS would be very useful over time. A lot of user 
space tools have been made aware of them, we are certainly lagging all (almost 
all?) local file system here and it can cause a data loss when you copy a file 
from a local file system to an NFS server.

It certainly violates the principle of least surprise that the xattrs are lost 
when move through NFS!

Typical use cases I have seen include storing things like metadata that tracks 
what application created the file, tags to let you know when the last time the 
file was verified by a data integrity scrubber, tags that hold a checksum/crypto 
has of the file contents along with the date of that signature.

Doing a file system service does not mean that we have to be personally 
interested in all of the odd things that our users might use features for, but 
at this point, xattrs are water under the bridge and NFS not supporting them 
makes us look bad :)

Ric

Re: XATTRs in NFS?

[Chuck Lever]

On Oct 25, 2013, at 11:26 AM, Ric Wheeler <ricwheeler@gmail.com> wrote:

> On 10/25/2013 10:08 AM, J. Bruce Fields wrote:
>> On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
>>> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
>>>> Those programs need to recompute the checksum data anyway in order to
>>>> verify and/or update it. Checksums that are computed by some third
>>>> party application have exactly zero value for integrity checking.
>>> No that's exactly the point,... the applications should _NOT_ set those
>>> checksums, especially not automagically (since then you'd never notice
>>> when just some application is buggy or writes/modifies when you don't
>>> expect it to do so).
>>> The idea is that there is on application (in my case it's just a
>>> script), which sets the integrity data and verifies it.
>>> 
>>> This works very well for e.g. large data archives, where you most of the
>>> time (but not always) only read files, write new files or move around
>>> existing ones - but only rarely modify existing file's contents.
>>> 
>>> 
>>> I do this already like that on local filesystems, which works very
>>> nicely with XATTRs... but now I want to move this on a central data
>>> cluster (where clients connect to via NFS)... and here the problems
>>> start... when I add new data to the archive (from the clients) I cannot
>>> have XATTRs attached, nor can I verify them form the clients.
>> Can you give any more details about your use case?  (E.g. is there an
>> article describing this system somewhere?)
>> 
>> Just curious.--b.
>> 
> 
> I think that having xattrs in NFS would be very useful over time. A lot of user space tools have been made aware of them, we are certainly lagging all (almost all?) local file system here and it can cause a data loss when you copy a file from a local file system to an NFS server.
> 
> It certainly violates the principle of least surprise that the xattrs are lost when move through NFS!
> 
> Typical use cases I have seen include storing things like metadata that tracks what application created the file, tags to let you know when the last time the file was verified by a data integrity scrubber, tags that hold a checksum/crypto has of the file contents along with the date of that signature.
> 
> Doing a file system service does not mean that we have to be personally interested in all of the odd things that our users might use features for, but at this point, xattrs are water under the bridge and NFS not supporting them makes us look bad :)

Trond's point is that it still would be a very heavy lift for us.

Local file systems can support whatever they like, since they are Linux-only.

NFS has to interoperate with other operating systems that may or may not support xattrs, or may have something that looks like an xattr, but really is subtly incompatible.

Therefore we must have a standard for storing and accessing xattrs locally (POSIX) and a standard for expressing their name and content on the wire (IETF NFS) before we can consider an implementation.

So we would like to see some very clear evidence that it's worth the effort.  Until then, Linux distributors are free to implement NFS xattr support outside the standard NFS protocol (like the Solaris NFSv3 ACL protocol) and support this themselves, perhaps as proof that "if you build it, they will come."

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi Chuck,

>From a standards perspective, could you please elaborate on how local xattrs are
"subtly incompatible" with NFSv4 named attributes?

It appears to me that the interesting issues (if any) are strictly related to possible lack of well-understood, or recommended, mappings of nfsv4 named attributes to and from whatever file attribute concept(s) exist on the client and server, not with the named attribute interface.

I did find slides in which James Morris asserts that "NFSv4 includes Named Attributes, which are based on the Solaris subfile model and incompatible with name/value schemes"
(http://www.slideshare.net/jamesmorris/adding-extended-attribute-support-to-nfs).
However, named attributes as actually specified in RFC 5661, at least, are not in fact a "fully general" subfile mechanism as found in Solaris.  The various restrictions in section 5.3 of RFC 5661 restrict implementations to a single-level namespace of attributes.  With the non-recursion restriction in place, nfsv4 named attributes appear to form a 1-to-1 mapping with Windows alternate data streams and Mac resource forks, and since nfsv4 named attribute data may be of arbitrary length, they also appear able to represent Linux or FreeBSD named attribute, should the an nfsv4 server on one of those platforms wish to do so.  (I presume that is by design.)

There seems to be a long history of mapping precursors of "posix" named attributes into alternate data streams (http://en.wikipedia.org/wiki/Extended_file_attributes#Windows_NT), and even the reverse (e.g., the ntfs3g file system driver for Linux, http://www.tuxera.com/community/ntfs-3g-advanced/extended-attributes/):

"""The extended attributes in user name space are stored on NTFS as alternate data streams whose name is the unprefixed name of the attribute, and whose contents is the value of the attribute. They can be read and modified in Windows by using the standard file access functions, with a colon and the stream name (which is the unprefixed extended attribute name) appended to the file name."""

Finally, we have existing implementation experience.  I'll leave out Solaris, and mention the Windows NFSv4.1 client.  The Windows client implements nfsv4 named attributes as a direct mapping to Windows fs alternate data streams.

Presumably, there could be other differences between a platform attribute
interface an NFSv4 named attributes, e.g., perhaps update atomicity.  It seems
though that any issues which might arise from this would also be ones the standard
could expect the client and/or server implementation to deal appropriately with.

Matt

----- "Chuck Lever" <chuck.lever@oracle.com> wrote:

> 
> NFS has to interoperate with other operating systems that may or may
> not support xattrs, or may have something that looks like an xattr,
> but really is subtly incompatible.
> 
> Therefore we must have a standard for storing and accessing xattrs
> locally (POSIX) and a standard for expressing their name and content
> on the wire (IETF NFS) before we can consider an implementation.
> 
> So we would like to see some very clear evidence that it's worth the
> effort.  Until then, Linux distributors are free to implement NFS
> xattr support outside the standard NFS protocol (like the Solaris
> NFSv3 ACL protocol) and support this themselves, perhaps as proof that
> "if you build it, they will come."
> 
> -- 
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[J. Bruce Fields]

On Fri, Oct 25, 2013 at 07:52:34PM -0400, Matt W. Benjamin wrote:
> Hi Chuck,
> 
> >From a standards perspective, could you please elaborate on how local xattrs are
> "subtly incompatible" with NFSv4 named attributes?
> 
> It appears to me that the interesting issues (if any) are strictly related to possible lack of well-understood, or recommended, mappings of nfsv4 named attributes to and from whatever file attribute concept(s) exist on the client and server, not with the named attribute interface.
> 
> I did find slides in which James Morris asserts that "NFSv4 includes Named Attributes, which are based on the Solaris subfile model and incompatible with name/value schemes"
> (http://www.slideshare.net/jamesmorris/adding-extended-attribute-support-to-nfs).
> However, named attributes as actually specified in RFC 5661, at least, are not in fact a "fully general" subfile mechanism as found in Solaris.  The various restrictions in section 5.3 of RFC 5661 restrict implementations to a single-level namespace of attributes.  With the non-recursion restriction in place, nfsv4 named attributes appear to form a 1-to-1 mapping with Windows alternate data streams and Mac resource forks, and since nfsv4 named attribute data may be of arbitrary length, they also appear able to represent Linux or FreeBSD named attribute, should the an nfsv4 server on one of those platforms wish to do so.  (I presume that is by design.)
> 
> There seems to be a long history of mapping precursors of "posix" named attributes into alternate data streams (http://en.wikipedia.org/wiki/Extended_file_attributes#Windows_NT), and even the reverse (e.g., the ntfs3g file system driver for Linux, http://www.tuxera.com/community/ntfs-3g-advanced/extended-attributes/):
> 
> """The extended attributes in user name space are stored on NTFS as alternate data streams whose name is the unprefixed name of the attribute, and whose contents is the value of the attribute. They can be read and modified in Windows by using the standard file access functions, with a colon and the stream name (which is the unprefixed extended attribute name) appended to the file name."""
> 
> Finally, we have existing implementation experience.  I'll leave out Solaris, and mention the Windows NFSv4.1 client.  The Windows client implements nfsv4 named attributes as a direct mapping to Windows fs alternate data streams.
> 
> Presumably, there could be other differences between a platform attribute
> interface an NFSv4 named attributes, e.g., perhaps update atomicity.

Right, xattrs are expected to be small-ish and are set and queried
atomically.  A server that implements named attributes using xattrs has
to for example implement write as a read-modify-write of the attribute.
You normally only address xattrs by name, so I guess you'd have to stick
a hash of the name in the filehandle.

We could probably get *something* working, but I don't know whether the
semantics would be close enough to not break an application that was
written to a named-attribute api.

Personally what I'd like to see is a really thorough description of a
use case or two.  All I've ever heard is either "selinux", or some vague
ideas about things people think might be cool to try.

It seemed various Linux desktop projects (Beagle?) were using them for a
while, but I haven't noticed that happening lately.  Did they give up?

Looking at my machines at home....  A "find . -print0 | xargs -0
getfattr" doesn't find anything on my Fedora machines, but it does find
"user.xdg.origin.url" and "user.xdg.referrer.url" xattrs set on a few
downloads created by chromium on a Ubuntu desktop.  Those xattrs are
documented at

	http://freedesktop.org/wiki/CommonExtendedAttributes/

Can't see anyplace they're used, except that the file manager does have
a "user attributes" tab in the "properties" dialog that you can pull up
from a right-click.

The tool I use to keep my home directories in sync (unison) doesn't
appear to propagate those.

Ho-hum.

--b.

> It seems
> though that any issues which might arise from this would also be ones the standard
> could expect the client and/or server implementation to deal appropriately with.
> 
> Matt
> 
> ----- "Chuck Lever" <chuck.lever@oracle.com> wrote:
> 
> > 
> > NFS has to interoperate with other operating systems that may or may
> > not support xattrs, or may have something that looks like an xattr,
> > but really is subtly incompatible.
> > 
> > Therefore we must have a standard for storing and accessing xattrs
> > locally (POSIX) and a standard for expressing their name and content
> > on the wire (IETF NFS) before we can consider an implementation.
> > 
> > So we would like to see some very clear evidence that it's worth the
> > effort.  Until then, Linux distributors are free to implement NFS
> > xattr support outside the standard NFS protocol (like the Solaris
> > NFSv3 ACL protocol) and support this themselves, perhaps as proof that
> > "if you build it, they will come."
> > 
> > -- 
> > Chuck Lever
> > chuck[dot]lever[at]oracle[dot]com
> > 
> > 
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -- 
> Matt Benjamin
> The Linux Box
> 206 South Fifth Ave. Suite 150
> Ann Arbor, MI  48104
> 
> http://linuxbox.com
> 
> tel.  734-761-4689 
> fax.  734-769-8938 
> cel.  734-216-5309 

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi,

Thanks, Bruce.

----- "J. Bruce Fields" <bfields@fieldses.org> wrote:
attributes, e.g., perhaps update
> atomicity.
> 
> Right, xattrs are expected to be small-ish and are set and queried
> atomically.  A server that implements named attributes using xattrs
> has
> to for example implement write as a read-modify-write of the
> attribute.

Ok, helpful.

> You normally only address xattrs by name, so I guess you'd have to
> stick
> a hash of the name in the filehandle.

That's how we're (re) implementing it for G.

> 
> We could probably get *something* working, but I don't know whether
> the
> semantics would be close enough to not break an application that was
> written to a named-attribute api.

I agree, there seems to be a missing piece.

> 
> Personally what I'd like to see is a really thorough description of a
> use case or two.  All I've ever heard is either "selinux", or some
> vague
> ideas about things people think might be cool to try.

I think the Christoph have made the best case that can be made, but I have
nothing more for the Linux debate.  I'd like to see us solve the design 
problems, though.

Matt

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 25, 2013, at 5:26 PM, Ric Wheeler <ricwheeler@gmail.com> wrote:

> On 10/25/2013 10:08 AM, J. Bruce Fields wrote:
>> On Thu, Oct 24, 2013 at 07:22:30PM +0200, Christoph Anton Mitterer wrote:
>>> On Thu, 2013-10-24 at 16:30 +0000, Myklebust, Trond wrote:
>>>> Those programs need to recompute the checksum data anyway in order to
>>>> verify and/or update it. Checksums that are computed by some third
>>>> party application have exactly zero value for integrity checking.
>>> No that's exactly the point,... the applications should _NOT_ set those
>>> checksums, especially not automagically (since then you'd never notice
>>> when just some application is buggy or writes/modifies when you don't
>>> expect it to do so).
>>> The idea is that there is on application (in my case it's just a
>>> script), which sets the integrity data and verifies it.
>>> 
>>> This works very well for e.g. large data archives, where you most of the
>>> time (but not always) only read files, write new files or move around
>>> existing ones - but only rarely modify existing file's contents.
>>> 
>>> 
>>> I do this already like that on local filesystems, which works very
>>> nicely with XATTRs... but now I want to move this on a central data
>>> cluster (where clients connect to via NFS)... and here the problems
>>> start... when I add new data to the archive (from the clients) I cannot
>>> have XATTRs attached, nor can I verify them form the clients.
>> Can you give any more details about your use case?  (E.g. is there an
>> article describing this system somewhere?)
>> 
>> Just curious.--b.
>> 
> 
> I think that having xattrs in NFS would be very useful over time. A lot of user space tools have been made aware of them, we are certainly lagging all (almost all?) local file system here and it can cause a data loss when you copy a file from a local file system to an NFS server.
> 
> It certainly violates the principle of least surprise that the xattrs are lost when move through NFS!

The _problem_ is that xattrs are a system call interface. Setting the xattrs in the "system", "security", or even the "trusted" namespace have undocumented side effects. There are very good security reasons why NFS doesn't have an IOCTL rpc call for executing arbitrary ioctls on the NFS server. Those same reasons apply to xattr.

> Typical use cases I have seen include storing things like metadata that tracks what application created the file, tags to let you know when the last time the file was verified by a data integrity scrubber, tags that hold a checksum/crypto has of the file contents along with the date of that signature.

Standardise those xattrs, and we can export them specifically as part of the NFSv4 spec.

> Doing a file system service does not mean that we have to be personally interested in all of the odd things that our users might use features for, but at this point, xattrs are water under the bridge and NFS not supporting them makes us look bad :)

I'd rather look bad for not supporting a broken filesystem feature than have to deal with the abiove mentioned side-effects. Until the xattr interface is properly documented and standardised so that it can be exported safely, it should be treated as a unexportable, in the same way we treat procfs and sysfs.

Cheers
  Trond

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi,

Surely you don't need NFSv4 to standardize the the (empty?) set of system or magic attributes Linux
should export?

Besides, as you're well aware, most people who ask for xattrs are looking for an ability to associate
arbitrary specific data, not a back door ioctl interface.  That's clearly what the NFSv4 named attributes as standardized were intended for.

I'm well aware of other uses and plans that someone would want to standardize, but it seems
irrelevant to the discussion.

Matt


----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:

> 
> Standardise those xattrs, and we can export them specifically as part
> of the NFSv4 spec.
> 

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2541 bytes --]

On Fri, 2013-10-25 at 10:08 -0400, J. Bruce Fields wrote:
> Can you give any more details about your use case?  (E.g. is there an
> article describing this system somewhere?)
Okay let me try to explain the motivation more elaborately.

The general idea is getting data integrity, i.e. being able to see
whether some files are in a valid state.

This is similar to what e.g. btrfs/zfs provide at (IIRC block/extent
level) with checksuming.
But a) btrfs is not yet fully production ready (IMHO) and zfs in Linux
has it's (license) issues and more important b) the checksuming there
happens always and automatically as soon as some valid filesystem
operations occur on the files.

So what you basically notice are errors on the physical medium (or at
least on block layers below the filesystem).


You won't notice many other cases of file "corruptions":

- when you have programs which do subtly manipulate the files and you
don't notice,...e.g. some image organisation programs store their
meta-data crap in the Exif/XMP fields, even when you don't actively tell
them to do so (and sometimes even when the files are read-only)

- when there are e.g. kernel bugs (in some other places of the kernel)
and you copy the files around. Some years ago I found a bug (not the
solution) where single bit errors happened more or less randomly every
40-100 GB of writing data. In the end it was found to be a IOMMU related
bug and a single one liner of flushing some buffers cleared the problem.
Since such errors would happen already at earlier stages, when writing
such files btrfs/zfs would simply write the checksums of the corrupted
data.

So the idea of my integrity data is, that I really manually say "now the
data is in the state where I consider it to be consistent and I want to
have checksums stored and attached to the files, for exactly that
state", e.g. after I have read out some images from the SD card (perhaps
even twice with the cache cleared and the results diffed) and placed in
my archive.
Afterwards I can regularly verify the whole archive and if at some stage
corruptions as the above would have happened, I can simply take the
respective files from backups.


Obviously that cannot be stored *in* the files,... and a database
solution fails since, it wouldn't track the changes when I move files
around within the archive.


So IMHO the best solution were XATTRs, which do work fine for that
purpose... just the problem that I can't have it via NFS ;)


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Marc Eshel]

linux-nfs-owner@vger.kernel.org wrote on 10/26/2013 06:20:12 AM:

> I'd rather look bad for not supporting a broken filesystem feature 
> than have to deal with the abiove mentioned side-effects. Until the 
> xattr interface is properly documented and standardised so that it 
> can be exported safely, it should be treated as a unexportable, in 
> the same way we treat procfs and sysfs.
> 

This sounds more like an NFS server problem than a client side NFS. Can we 
add support to the client side NFS and let server that think that they can 
handle XATTR implement it?

We should probably start a discussion in the IETF to resolve the issues 
you see with XATTR at least for NFSv4.3. 

Marc.

> Cheers
>   Trond--

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

On Fri, 2013-10-25 at 11:32 -0400, Chuck Lever wrote: 
> Local file systems can support whatever they like, since they are
> Linux-only.
> NFS has to interoperate with other operating systems that may or may
> not support xattrs, or may have something that looks like an xattr,
> but really is subtly incompatible.
Well but that alone can apply to most other things that NFS does support
as well.
Take NFS4 ACLs,... not (fully) supported by I'd guess every Linux
filesystem (or was there some proposal for btrfs to get NFS4 ACLs in
addition?)
Or take very old filesystem who have no concept of different owners.
etc. pp.

In any case (as with Linux filesystems as well), I think XATTRs
shouldn't be enabled by default, if NFS was to support them... and if
someone activates it, he probably knows whether the backend fs does
support it or not.


> So we would like to see some very clear evidence that it's worth the
> effort.  Until then, Linux distributors are free to implement NFS
> xattr support outside the standard NFS protocol (like the Solaris
> NFSv3 ACL protocol) and support this themselves, perhaps as proof that
> "if you build it, they will come."

This always bears the danger that it will become a de facto standard
that one cannot get rid of anymore.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 26, 2013, at 10:01 AM, Matt W. Benjamin <matt@linuxbox.com> wrote:

> Hi,
> 
> Surely you don't need NFSv4 to standardize the the (empty?) set of system or magic attributes Linux
> should export?

The NFSv4 working group has no authority to do so. POSIX would be the right address.

> Besides, as you're well aware, most people who ask for xattrs are looking for an ability to associate
> arbitrary specific data, not a back door ioctl interface.  That's clearly what the NFSv4 named attributes as standardized were intended for.

No. I'm not aware of that.

> I'm well aware of other uses and plans that someone would want to standardize, but it seems
> irrelevant to the discussion.

It's very relevant to the discussion as it defines what namespace applications can expect to work.

Trond

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 26, 2013, at 1:46 PM, Marc Eshel <eshel@us.ibm.com> wrote:

> linux-nfs-owner@vger.kernel.org wrote on 10/26/2013 06:20:12 AM:
> 
>> I'd rather look bad for not supporting a broken filesystem feature 
>> than have to deal with the abiove mentioned side-effects. Until the 
>> xattr interface is properly documented and standardised so that it 
>> can be exported safely, it should be treated as a unexportable, in 
>> the same way we treat procfs and sysfs.
>> 
> 
> This sounds more like an NFS server problem than a client side NFS. Can we 
> add support to the client side NFS and let server that think that they can 
> handle XATTR implement it?

No. It's a real problem for clients too if the NFS server starts exporting ACLs and security labels via this interface. Everything from integer endianness problems when getfacl attempts to read raw binary acls from the server, to breakage of caching models when we allow setfacl and don't invalidate our file access caches...

> We should probably start a discussion in the IETF to resolve the issues 
> you see with XATTR at least for NFSv4.3. 

Let's get the basic discussion of motivation right first.

Trond

Re: XATTRs in NFS?

[Christoph Hellwig]

On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
> The NFSv4 working group has no authority to do so. POSIX would be the right address.

The complete lack of clue, authority or even defined semantics didn't
stop them from defining their ACL model.  User extended attributes are
harmless compared to that, even if everyone has subtly different
flavours.

Re: XATTRs in NFS?

[Simo Sorce]

On Sun, 2013-10-27 at 09:56 -0700, Christoph Hellwig wrote:
> On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
> > The NFSv4 working group has no authority to do so. POSIX would be the right address.
> 
> The complete lack of clue, authority or even defined semantics didn't
> stop them from defining their ACL model.  User extended attributes are
> harmless compared to that, even if everyone has subtly different
> flavours.

+1

-- 
Simo Sorce * Red Hat, Inc * New York

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 27, 2013, at 12:56 PM, Christoph Hellwig <hch@infradead.org> wrote:

> On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
>> The NFSv4 working group has no authority to do so. POSIX would be the right address.
> 
> The complete lack of clue, authority or even defined semantics didn't
> stop them from defining their ACL model.  User extended attributes are
> harmless compared to that, even if everyone has subtly different
> flavours.
> 

The user xattrs are in principle harmless. What about "trusted.*"?

BTW: caching any xattrs is a problem. There is no close-to-open model that you can use and neither is there locking. This is already causing a problem for labeled nfs...

Trond

Re: XATTRs in NFS?

[Simo Sorce]

On Sun, 2013-10-27 at 18:07 +0000, Myklebust, Trond wrote:
> On Oct 27, 2013, at 12:56 PM, Christoph Hellwig <hch@infradead.org> wrote:
> 
> > On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
> >> The NFSv4 working group has no authority to do so. POSIX would be the right address.
> > 
> > The complete lack of clue, authority or even defined semantics didn't
> > stop them from defining their ACL model.  User extended attributes are
> > harmless compared to that, even if everyone has subtly different
> > flavours.
> > 
> 
> The user xattrs are in principle harmless. What about "trusted.*"?
> 
> BTW: caching any xattrs is a problem. There is no close-to-open model that you can use and neither is there locking. This is already causing a problem for labeled nfs...

You could start without caching them.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 27, 2013, at 2:30 PM, Simo Sorce <simo@redhat.com> wrote:

> On Sun, 2013-10-27 at 18:07 +0000, Myklebust, Trond wrote:
>> On Oct 27, 2013, at 12:56 PM, Christoph Hellwig <hch@infradead.org> wrote:
>> 
>>> On Sun, Oct 27, 2013 at 12:31:46PM +0000, Myklebust, Trond wrote:
>>>> The NFSv4 working group has no authority to do so. POSIX would be the right address.
>>> 
>>> The complete lack of clue, authority or even defined semantics didn't
>>> stop them from defining their ACL model.  User extended attributes are
>>> harmless compared to that, even if everyone has subtly different
>>> flavours.
>>> 
>> 
>> The user xattrs are in principle harmless. What about "trusted.*"?
>> 
>> BTW: caching any xattrs is a problem. There is no close-to-open model that you can use and neither is there locking. This is already causing a problem for labeled nfs...
> 
> You could start without caching them.
> 

I'm not going to start anything...

Trond

Re: XATTRs in NFS?

[J. Bruce Fields]

Please don't trim the cc list.

On Sat, Oct 26, 2013 at 07:12:27PM +0200, Christoph Anton Mitterer wrote:
> On Fri, 2013-10-25 at 10:08 -0400, J. Bruce Fields wrote:
> > Can you give any more details about your use case?  (E.g. is there an
> > article describing this system somewhere?)
> Okay let me try to explain the motivation more elaborately.
> 
> The general idea is getting data integrity, i.e. being able to see
> whether some files are in a valid state.
> 
> This is similar to what e.g. btrfs/zfs provide at (IIRC block/extent
> level) with checksuming.
> But a) btrfs is not yet fully production ready (IMHO) and zfs in Linux
> has it's (license) issues and more important b) the checksuming there
> happens always and automatically as soon as some valid filesystem
> operations occur on the files.
> 
> So what you basically notice are errors on the physical medium (or at
> least on block layers below the filesystem).
> 
> 
> You won't notice many other cases of file "corruptions":
> 
> - when you have programs which do subtly manipulate the files and you
> don't notice,...e.g. some image organisation programs store their
> meta-data crap in the Exif/XMP fields, even when you don't actively tell
> them to do so (and sometimes even when the files are read-only)
> 
> - when there are e.g. kernel bugs (in some other places of the kernel)
> and you copy the files around. Some years ago I found a bug (not the
> solution) where single bit errors happened more or less randomly every
> 40-100 GB of writing data. In the end it was found to be a IOMMU related
> bug and a single one liner of flushing some buffers cleared the problem.
> Since such errors would happen already at earlier stages, when writing
> such files btrfs/zfs would simply write the checksums of the corrupted
> data.

Was this problem actually caught using checksums stored in xattrs, or
did the problem predate your use of xattrs?

> So the idea of my integrity data is, that I really manually say "now the
> data is in the state where I consider it to be consistent and I want to
> have checksums stored and attached to the files, for exactly that
> state", e.g. after I have read out some images from the SD card (perhaps
> even twice with the cache cleared and the results diffed) and placed in
> my archive.
> Afterwards I can regularly verify the whole archive and if at some stage
> corruptions as the above would have happened, I can simply take the
> respective files from backups.

How long have you been using this for?  How many problems has it caught?
How often do you checksum or verify files, and how expensive is that?

--b.

> 
> 
> Obviously that cannot be stored *in* the files,... and a database
> solution fails since, it wouldn't track the changes when I move files
> around within the archive.
> 
> 
> So IMHO the best solution were XATTRs, which do work fine for that
> purpose... just the problem that I can't have it via NFS ;)
> 
> 
> Cheers,
> Chris.

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi,

I was going for the point you make later in this thread:  "user xattrs are in
principle harmless."

The exposure policy for non-harmless xattrs issue is real enough, but my point was,
it seems to be a Linux policy issue, at the NFSv4 level just as it was at the syscall level.

The caching and atomicity issues seem like they actually are issues for NFSv4 discussion.
If the named attribute interface actually fails to cover xattrs, that's a big gaffe.

Matt


----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:

> On Oct 26, 2013, at 10:01 AM, Matt W. Benjamin <matt@linuxbox.com>
> wrote:
> 
> > Hi,
> > 
> > Surely you don't need NFSv4 to standardize the the (empty?) set of
> system or magic attributes Linux
> > should export?
> 
> The NFSv4 working group has no authority to do so. POSIX would be the
> right address.
> 
> > Besides, as you're well aware, most people who ask for xattrs are
> looking for an ability to associate
> > arbitrary specific data, not a back door ioctl interface.  That's
> clearly what the NFSv4 named attributes as standardized were intended
> for.
> 
> No. I'm not aware of that.
> 
> > I'm well aware of other uses and plans that someone would want to
> standardize, but it seems
> > irrelevant to the discussion.
> 
> It's very relevant to the discussion as it defines what namespace
> applications can expect to work.
> 
> Trond

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1551 bytes --]

On Sun, 2013-10-27 at 15:15 -0400, J. Bruce Fields wrote:
> Was this problem actually caught using checksums stored in xattrs, or
> did the problem predate your use of xattrs?
Phew... don't remember actually... but I think I haven't used that
already back then and noticed it by chance when I did some diffs.


> > So the idea of my integrity data is, that I really manually say "now the
> > data is in the state where I consider it to be consistent and I want to
> > have checksums stored and attached to the files, for exactly that
> > state", e.g. after I have read out some images from the SD card (perhaps
> > even twice with the cache cleared and the results diffed) and placed in
> > my archive.
> > Afterwards I can regularly verify the whole archive and if at some stage
> > corruptions as the above would have happened, I can simply take the
> > respective files from backups.
> How long have you been using this for?
Uhm... about 3-4 years now.

> How many problems has it caught?
I do not keep exact statistics... but I remember a few cases where I
found damaged backups (optimal media) which I replaced as a consequence.

> How often do you checksum or verify files, and how expensive is that?
Not that often,... on my actual data disks,... about every 4 months...
on my backup media (I use to keep older generations of backups as well)
about once a year.

I've never really looked at how expensive it is,... all you need to do
is simply reading all data + have their hashes calculated.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 170 bytes --]

On Sun, 2013-10-27 at 18:41 +0000, Myklebust, Trond wrote:
> I'm not going to start anything...
So that's basically a: "XATTRs in NFS - never"?! :(

Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

On Sun, 2013-10-27 at 12:48 +0000, Myklebust, Trond wrote:
> Let's get the basic discussion of motivation right first.
Actually I might know another use case.

dCache[0] which implements a (p)NFS 4.1 server and is heavily used
within the high energy physics community, could benefit from XATTRs as
well.

Ever since, dCache has had a concept called PNFS tags (where the PNFS
had nothing to do with parallel NFS, but is called so for other
historical reasons), which are basically key/value pairs belonging to
some file (well at least directory, I don't remember where PNFS tags
exist for regular files).

These tags are used to control several things, indirectly, e.g. to which
data pool node a file may go, and via that whether it may be moved to
tape, whether it is duplicated, etc. pp.

Since there were no XATTRs (at least I guess that's the historical
reason), they used special files for this having the name:
".(tag)(key)" and the content of the file is the value.

So for a path,
/data/foo/bar/
one can set the key "sGroup" (aka storage group) to a value say
"ATLAS" (which (very simplified) will tell dCache in the end that this
is storage belonging to the ATLAS experiment) by creating a file
echo "ATLAS" > '/data/foo/bar/.(tag)(sGroup)'


Obviously having such special files isn't that nice, and having real
XATTRs would be the better solution.



Cheers,
Chris.

btw: Note that while I'm good friends with the dCache developers, I
don't speak for them, nor do I know whether they'd really like this or
not.

[0] http://www.dcache.org/

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 27, 2013, at 5:57 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> On Sun, 2013-10-27 at 15:15 -0400, J. Bruce Fields wrote:
>> Was this problem actually caught using checksums stored in xattrs, or
>> did the problem predate your use of xattrs?
> Phew... don't remember actually... but I think I haven't used that
> already back then and noticed it by chance when I did some diffs.
> 
> 
>>> So the idea of my integrity data is, that I really manually say "now the
>>> data is in the state where I consider it to be consistent and I want to
>>> have checksums stored and attached to the files, for exactly that
>>> state", e.g. after I have read out some images from the SD card (perhaps
>>> even twice with the cache cleared and the results diffed) and placed in
>>> my archive.
>>> Afterwards I can regularly verify the whole archive and if at some stage
>>> corruptions as the above would have happened, I can simply take the
>>> respective files from backups.
>> How long have you been using this for?
> Uhm... about 3-4 years now.
> 
>> How many problems has it caught?
> I do not keep exact statistics... but I remember a few cases where I
> found damaged backups (optimal media) which I replaced as a consequence.
> 
>> How often do you checksum or verify files, and how expensive is that?
> Not that often,... on my actual data disks,... about every 4 months...
> on my backup media (I use to keep older generations of backups as well)
> about once a year.
> 
> I've never really looked at how expensive it is,... all you need to do
> is simply reading all data + have their hashes calculated.

...and if the checksums are any good, then all you need to do to substitute a database is to realise that a good data checksum is invariant under renames.

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 27, 2013, at 8:14 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> On Sun, 2013-10-27 at 12:48 +0000, Myklebust, Trond wrote:
>> Let's get the basic discussion of motivation right first.
> Actually I might know another use case.
> 
> dCache[0] which implements a (p)NFS 4.1 server and is heavily used
> within the high energy physics community, could benefit from XATTRs as
> well.
> 
> Ever since, dCache has had a concept called PNFS tags (where the PNFS
> had nothing to do with parallel NFS, but is called so for other
> historical reasons), which are basically key/value pairs belonging to
> some file (well at least directory, I don't remember where PNFS tags
> exist for regular files).
> 
> These tags are used to control several things, indirectly, e.g. to which
> data pool node a file may go, and via that whether it may be moved to
> tape, whether it is duplicated, etc. pp.
> 
> Since there were no XATTRs (at least I guess that's the historical
> reason), they used special files for this having the name:
> ".(tag)(key)" and the content of the file is the value.
> 
> So for a path,
> /data/foo/bar/
> one can set the key "sGroup" (aka storage group) to a value say
> "ATLAS" (which (very simplified) will tell dCache in the end that this
> is storage belonging to the ATLAS experiment) by creating a file
> echo "ATLAS" > '/data/foo/bar/.(tag)(sGroup)'
> 
> 
> Obviously having such special files isn't that nice, and having real
> XATTRs would be the better solution.

Ordinary groups can do the same.

Trond

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

On Mon, 2013-10-28 at 00:19 +0000, Myklebust, Trond wrote:
> Ordinary groups can do the same.
For sure not,.. a) groups are already used for something else (namely
the normal semantics of groups, and the sGroup does not necessarily need
to correspond to the UNIX owning group (actually it usually does not).

b) sGroup is only one of several tags,... there are tags which control
space tokens, there are tags which contain dCache's own file checksums,
etc. pp.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

On Mon, 2013-10-28 at 00:17 +0000, Myklebust, Trond wrote:
> ...and if the checksums are any good, then all you need to do to
> substitute a database is to realise that a good data checksum is
> invariant under renames.

Don't quite see what you mean...

Sure the checksums stay the same, but consider you have many millions of
files, and you moved them around and thus the paths in the DB are
incorrect... verifying the files will become very much a pain in the
a**, especially when multiple files don't verify anymore.


Or what if you have many small similar files, where errors could lead to
a checksum that was a correct one for another file,... when the paths
are no longer valid you cannot know if this was a correct file or not.


Cheers,
Chris.


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 27, 2013, at 6:20 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> On Sun, 2013-10-27 at 18:41 +0000, Myklebust, Trond wrote:
>> I'm not going to start anything...
> So that's basically a: "XATTRs in NFS - never"?! :(

Your case for adding the feature seems to be "I have a couple of programs that use xattrs, so NFS should provide it.".

Well, that means at a minimum going to the IETF (possibly also POSIX) to convince them that the feature is worth adding, helping design the protocol to do so, and achieving consensus in a NFSv4.x spec, then doing the work designing and coding up a Linux implementation. Who is volunteering to do all that?

My statement above was that I'm not volunteering to do all this. You haven't convinced me that your programs cannot be modified to use other more portable solutions. You're not even generating the checksums when creating the data, so there is already an existing window during which corruption can occur.

Trond

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 27, 2013, at 8:27 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> On Mon, 2013-10-28 at 00:17 +0000, Myklebust, Trond wrote:
>> ...and if the checksums are any good, then all you need to do to
>> substitute a database is to realise that a good data checksum is
>> invariant under renames.
> 
> Don't quite see what you mean...
> 
> Sure the checksums stay the same, but consider you have many millions of
> files, and you moved them around and thus the paths in the DB are
> incorrect... verifying the files will become very much a pain in the
> a**, especially when multiple files don't verify anymore.
> 
> Or what if you have many small similar files, where errors could lead to
> a checksum that was a correct one for another file,... when the paths
> are no longer valid you cannot know if this was a correct file or not.

If you have lots of small files, and you really do need to associate them uniquely with the checksum, then try something like:

	ln <filename> /path/to/database/<SHA512 identifier>.<inode number>

Hard links and inode numbers are also generally invariant under 'mv'.

Trond

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On Mon, 2013-10-28 at 00:44 +0000, Myklebust, Trond wrote:
> If you have lots of small files, and you really do need to associate them uniquely with the checksum, then try something like:
> 
> 	ln <filename> /path/to/database/<SHA512 identifier>.<inode number>
> 
> Hard links and inode numbers are also generally invariant under 'mv'.

Well that seems like a workaround, IMHO not like a real solution,...
since when you remove <filename> the file will stay due
to /path/to/database/<SHA512 identifier>.<inode number> .

Of course you can run regular scrubbing jobs, which remove
and /path/to/database/<SHA512 identifier>.<inode number> for which the
link count is one.


Anyway as said, IMHO it's a (nice) workaround which I haven't thought
of, admittedly, but not a clean solution... and AFAICS it fixes only my
personal use case.

Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Hellwig Christoph]

On Sun, Oct 27, 2013 at 06:07:46PM +0000, Myklebust, Trond wrote:
> The user xattrs are in principle harmless. What about "trusted.*"?

trusted ones are still fairly harmless and exist in most implementations
under a slightly different name.  They aren't quite as essential,
though.

The important point is to not carry over the system and security namespacs.
Those are a bad and Linux specific mistake to abuse the xattr interface for
all kinds of other interfaces.  This isn't something that should be carried
over to other standards.

Re: XATTRs in NFS?

[James Morris]

On Sun, 27 Oct 2013, Myklebust, Trond wrote:

> > This sounds more like an NFS server problem than a client side NFS. Can we 
> > add support to the client side NFS and let server that think that they can 
> > handle XATTR implement it?
> 
> No. It's a real problem for clients too if the NFS server starts 
> exporting ACLs and security labels via this interface. Everything from 
> integer endianness problems when getfacl attempts to read raw binary 
> acls from the server, to breakage of caching models when we allow 
> setfacl and don't invalidate our file access caches...

Right, so any general xattrs for nfs would need to be specified as having 
no system effects, essentially exporting the Linux 'user' xattr namespace 
only.


> 
> > We should probably start a discussion in the IETF to resolve the issues 
> > you see with XATTR at least for NFSv4.3. 
> 
> Let's get the basic discussion of motivation right first.

It should be for user-managed metadata only.



-- 
James Morris
<jmorris@namei.org>

Re: XATTRs in NFS?

[Ric Wheeler]

On 10/27/2013 08:44 PM, Myklebust, Trond wrote:
> On Oct 27, 2013, at 8:27 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:
>
>> On Mon, 2013-10-28 at 00:17 +0000, Myklebust, Trond wrote:
>>> ...and if the checksums are any good, then all you need to do to
>>> substitute a database is to realise that a good data checksum is
>>> invariant under renames.
>> Don't quite see what you mean...
>>
>> Sure the checksums stay the same, but consider you have many millions of
>> files, and you moved them around and thus the paths in the DB are
>> incorrect... verifying the files will become very much a pain in the
>> a**, especially when multiple files don't verify anymore.
>>
>> Or what if you have many small similar files, where errors could lead to
>> a checksum that was a correct one for another file,... when the paths
>> are no longer valid you cannot know if this was a correct file or not.
> If you have lots of small files, and you really do need to associate them uniquely with the checksum, then try something like:
>
> 	ln <filename> /path/to/database/<SHA512 identifier>.<inode number>
>
> Hard links and inode numbers are also generally invariant under 'mv'.
>
> Trond--
>

Then you end up with large directories and an extra name per inode that needs to 
be stored and extra lookups for each file when you do a whole file system crawl.

Certainly not as easy as adding and xattrs with that information :)

Ric

Re: XATTRs in NFS?

[Ric Wheeler]

On 10/28/2013 09:25 AM, James Morris wrote:
> On Sun, 27 Oct 2013, Myklebust, Trond wrote:
>
>>> This sounds more like an NFS server problem than a client side NFS. Can we
>>> add support to the client side NFS and let server that think that they can
>>> handle XATTR implement it?
>> No. It's a real problem for clients too if the NFS server starts
>> exporting ACLs and security labels via this interface. Everything from
>> integer endianness problems when getfacl attempts to read raw binary
>> acls from the server, to breakage of caching models when we allow
>> setfacl and don't invalidate our file access caches...
> Right, so any general xattrs for nfs would need to be specified as having
> no system effects, essentially exporting the Linux 'user' xattr namespace
> only.
>
>
>>> We should probably start a discussion in the IETF to resolve the issues
>>> you see with XATTR at least for NFSv4.3.
>> Let's get the basic discussion of motivation right first.
> It should be for user-managed metadata only.
>

That would be what I would like to see over NFS...

ric

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
> Then you end up with large directories and an extra name per inode that needs to 
> be stored and extra lookups for each file when you do a whole file system crawl.
> 
> Certainly not as easy as adding and xattrs with that information :)

And I think there's another reason why it wouldn't work...

Imagine I change my system to encode what should be XATTRs in hardlink
pseudo files...

If I have such pair locally e.g. on my ext4:
/foo/bar/actual/file
/meta/<SHA512 identifier>.2342348324

And now move/copy the file via the network to the archive, I'd have to
copy both files (which is really annoying), and I'd guess the inode
coupling would get los (and at least the name wouldn't fit anymore).

So the whole thing is IMHO not even a workaround.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>> Then you end up with large directories and an extra name per inode that needs to 
>> be stored and extra lookups for each file when you do a whole file system crawl.
>> 
>> Certainly not as easy as adding and xattrs with that information :)
> 
> And I think there's another reason why it wouldn't work...
> 
> Imagine I change my system to encode what should be XATTRs in hardlink
> pseudo files...
> 
> If I have such pair locally e.g. on my ext4:
> /foo/bar/actual/file
> /meta/<SHA512 identifier>.2342348324
> 
> And now move/copy the file via the network to the archive, I'd have to
> copy both files (which is really annoying), and I'd guess the inode
> coupling would get los (and at least the name wouldn't fit anymore).
> 
> So the whole thing is IMHO not even a workaround.

OK. So you're going to do XATTRs for us?

Trond

Re: XATTRs in NFS?

[Ric Wheeler]

On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:
>
>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>> Then you end up with large directories and an extra name per inode that needs to
>>> be stored and extra lookups for each file when you do a whole file system crawl.
>>>
>>> Certainly not as easy as adding and xattrs with that information :)
>> And I think there's another reason why it wouldn't work...
>>
>> Imagine I change my system to encode what should be XATTRs in hardlink
>> pseudo files...
>>
>> If I have such pair locally e.g. on my ext4:
>> /foo/bar/actual/file
>> /meta/<SHA512 identifier>.2342348324
>>
>> And now move/copy the file via the network to the archive, I'd have to
>> copy both files (which is really annoying), and I'd guess the inode
>> coupling would get los (and at least the name wouldn't fit anymore).
>>
>> So the whole thing is IMHO not even a workaround.
> OK. So you're going to do XATTRs for us?
>
> Trond

Now that pNFS is perfect and labeled NFS has made it upstream, I think that 
Steve D must be looking for something to keep him busy :)

ric

Re: XATTRs in NFS?

[Dr Fields James Bruce]

On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
> >On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:
> >
> >>On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
> >>>Then you end up with large directories and an extra name per inode that needs to
> >>>be stored and extra lookups for each file when you do a whole file system crawl.
> >>>
> >>>Certainly not as easy as adding and xattrs with that information :)
> >>And I think there's another reason why it wouldn't work...
> >>
> >>Imagine I change my system to encode what should be XATTRs in hardlink
> >>pseudo files...
> >>
> >>If I have such pair locally e.g. on my ext4:
> >>/foo/bar/actual/file
> >>/meta/<SHA512 identifier>.2342348324
> >>
> >>And now move/copy the file via the network to the archive, I'd have to
> >>copy both files (which is really annoying), and I'd guess the inode
> >>coupling would get los (and at least the name wouldn't fit anymore).
> >>
> >>So the whole thing is IMHO not even a workaround.
> >OK. So you're going to do XATTRs for us?
> >
> >Trond
> 
> Now that pNFS is perfect and labeled NFS has made it upstream, I
> think that Steve D must be looking for something to keep him busy :)

I agree with Trond that we first really need good evidence about exactly
who wants this and why.

--b.

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

On Mon, 2013-10-28 at 17:49 +0000, Myklebust, Trond wrote:
> OK. So you're going to do XATTRs for us?
That's not what I've said ;-) ... neither have I said that you must or
should do it.

I asked for the status and mentioned the need of it. You asked me why -
so I tried to explain.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Ric Wheeler]

On 10/28/2013 02:08 PM, Dr Fields James Bruce wrote:
> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:
>>>
>>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>>>> Then you end up with large directories and an extra name per inode that needs to
>>>>> be stored and extra lookups for each file when you do a whole file system crawl.
>>>>>
>>>>> Certainly not as easy as adding and xattrs with that information :)
>>>> And I think there's another reason why it wouldn't work...
>>>>
>>>> Imagine I change my system to encode what should be XATTRs in hardlink
>>>> pseudo files...
>>>>
>>>> If I have such pair locally e.g. on my ext4:
>>>> /foo/bar/actual/file
>>>> /meta/<SHA512 identifier>.2342348324
>>>>
>>>> And now move/copy the file via the network to the archive, I'd have to
>>>> copy both files (which is really annoying), and I'd guess the inode
>>>> coupling would get los (and at least the name wouldn't fit anymore).
>>>>
>>>> So the whole thing is IMHO not even a workaround.
>>> OK. So you're going to do XATTRs for us?
>>>
>>> Trond
>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>> think that Steve D must be looking for something to keep him busy :)
> I agree with Trond that we first really need good evidence about exactly
> who wants this and why.
>
> --b.

For the user space xattrs, many applications store various types of metadata.  
Gluster for example heavily uses xattrs, other programs do things like data 
scrubbing (look for a long unchanged file, compute a has and store it as an 
xattr) or simply use it to annotate the file with the name of the program that 
created it. Think of it as file decorations or annotations.

Today, if we store files in NFS that have xattrs, we do in fact cause data loss.

I can understand an answer of "this would be hard to do for NFS and need to go 
through IETF" but think that xattrs are well enough established in Linux and 
supported in the tool chain that it is way too late to question whether or not 
supporting them is a worth our time.

Ric

Re: XATTRs in NFS?

[Marc Eshel]

Hi Spencer, 
Is it still possible to get on the agenda fir IETF 88, I just got approval 
to travel. We can use about 15 minutes, or what ever is available to 
discussion the future of named attributes in NFS. The two main questions 
that we need answer are: 
1. Do we need them? what applications use them and how. 
2. Can we have a more simple model that handles just user attributes.

Input is welcome to help me make the case at the meeting.

Thanks, Marc.




From:   Ric Wheeler <rwheeler@redhat.com>
To:     Dr Fields James Bruce <bfields@fieldses.org>, 
Cc:     "Myklebust, Trond" <Trond.Myklebust@netapp.com>, Christoph Anton 
Mitterer <calestyo@scientia.net>, Mailing List Linux NFS 
<linux-nfs@vger.kernel.org>, Steve Dickson <SteveD@redhat.com>
Date:   10/28/2013 11:32 AM
Subject:        Re: XATTRs in NFS?
Sent by:        linux-nfs-owner@vger.kernel.org



On 10/28/2013 02:08 PM, Dr Fields James Bruce wrote:
> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer 
<calestyo@scientia.net> wrote:
>>>
>>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>>>> Then you end up with large directories and an extra name per inode 
that needs to
>>>>> be stored and extra lookups for each file when you do a whole file 
system crawl.
>>>>>
>>>>> Certainly not as easy as adding and xattrs with that information :)
>>>> And I think there's another reason why it wouldn't work...
>>>>
>>>> Imagine I change my system to encode what should be XATTRs in 
hardlink
>>>> pseudo files...
>>>>
>>>> If I have such pair locally e.g. on my ext4:
>>>> /foo/bar/actual/file
>>>> /meta/<SHA512 identifier>.2342348324
>>>>
>>>> And now move/copy the file via the network to the archive, I'd have 
to
>>>> copy both files (which is really annoying), and I'd guess the inode
>>>> coupling would get los (and at least the name wouldn't fit anymore).
>>>>
>>>> So the whole thing is IMHO not even a workaround.
>>> OK. So you're going to do XATTRs for us?
>>>
>>> Trond
>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>> think that Steve D must be looking for something to keep him busy :)
> I agree with Trond that we first really need good evidence about exactly
> who wants this and why.
>
> --b.

For the user space xattrs, many applications store various types of 
metadata. 
Gluster for example heavily uses xattrs, other programs do things like 
data 
scrubbing (look for a long unchanged file, compute a has and store it as 
an 
xattr) or simply use it to annotate the file with the name of the program 
that 
created it. Think of it as file decorations or annotations.

Today, if we store files in NFS that have xattrs, we do in fact cause data 
loss.

I can understand an answer of "this would be hard to do for NFS and need 
to go 
through IETF" but think that xattrs are well enough established in Linux 
and 
supported in the tool chain that it is way too late to question whether or 
not 
supporting them is a worth our time.

Ric

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: XATTRs in NFS?

[Marc Eshel]

[-- Attachment #1.1: Type: text/plain, Size: 2733 bytes --]

linux-nfs-owner@vger.kernel.org wrote on 10/28/2013 02:04:57 PM:

> From: Chuck Lever <chuck.lever@oracle.com>
> To: "J. Bruce Fields" <bfields@fieldses.org>, 
> Cc: "Haynes, Tom" <Tom.Haynes@netapp.com>, Marc Eshel/Almaden/
> IBM@IBMUS, Spencer Shepler <spencer.shepler@gmail.com>, Mailing List
> Linux NFS <linux-nfs@vger.kernel.org>, Christoph Anton Mitterer 
> <calestyo@scientia.net>, "Myklebust, Trond" 
> <Trond.Myklebust@netapp.com>, Steve Dickson <SteveD@redhat.com>, 
> "nfsv4@ietf.org nfsv4@ietf.org" <nfsv4@ietf.org>, Ric Wheeler 
> <ric.wheeler@usenix.org>, "linux-nfs-owner@vger.kernel.org" <linux-
> nfs-owner@vger.kernel.org>
> Date: 10/28/2013 02:07 PM
> Subject: Re: [nfsv4] XATTRs in NFS?
> Sent by: linux-nfs-owner@vger.kernel.org
> 
> 
> On Oct 28, 2013, at 5:02 PM, "J. Bruce Fields" <bfields@fieldses.org> 
wrote:
> 
> > On Mon, Oct 28, 2013 at 08:55:06PM +0000, Haynes, Tom wrote:
> >> 
> >> On Oct 28, 2013, at 1:44 PM, Marc Eshel <eshel@us.ibm.com> wrote:
> >> 
> >>> Hi Spencer, 
> >>> Is it still possible to get on the agenda fir IETF 88, I just 
> got approval 
> >>> to travel. We can use about 15 minutes, or what ever is available to 

> >>> discussion the future of named attributes in NFS. The two main 
questions 
> >>> that we need answer are: 
> >>> 1. Do we need them? what applications use them and how. 
> >>> 2. Can we have a more simple model that handles just user 
attributes.
> >>> 
> >>> Input is welcome to help me make the case at the meeting.
> >>> 
> >>> Thanks, Marc.
> >>> 
> >> 
> >> 
> >> Be sure to call out why they currently do or do not work.
> > 
> > And we also need to keep clear the distinctions between:
> > 
> >    - NFSv4 named attributes/streams/forks and "extended attributes"
> >      as implemented in Linux and elsewhere.
> >    - user.* xattrs and "back-door ioctls".
> > 
> > (Your description above seems to confuse named attributes and Linux
> > extended attributes.)
> 
> I wonder if we will have a balanced discussion next week.  The 
> people who seem to know the most about these issues on Linux will 
> not be attending.  Perhaps it would be better to start this 
> discussion on the mailing list instead.

Like I said input is welcome to the mailing list and I can try to collect 
all the main points into a presentation at the IETF. The main objective 
for now is to see if this is an item for the working group to start 
dealing with for future NFS protocol enhancements.
Marc.

> 
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

[-- Attachment #1.2: Type: text/html, Size: 3993 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4

Re: XATTRs in NFS?

[Matt W. Benjamin]

Hi,

We are prepared to contribute a draft implementation of -something-, and have planned
to do so.

We're currently working on Ganesha support.  I have assumed that what we
would implement is mapping of from user xattr to current nfsv4 named attributes.

Matt

----- "Ric Wheeler" <rwheeler@redhat.com> wrote:

> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
> > On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer
> <calestyo@scientia.net> wrote:
> >
> >> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
> >>> Then you end up with large directories and an extra name per inode
> that needs to
> >>> be stored and extra lookups for each file when you do a whole file
> system crawl.
> >>>
> >>> Certainly not as easy as adding and xattrs with that information
> :)
> >> And I think there's another reason why it wouldn't work...
> >>
> >> Imagine I change my system to encode what should be XATTRs in
> hardlink
> >> pseudo files...
> >>
> >> If I have such pair locally e.g. on my ext4:
> >> /foo/bar/actual/file
> >> /meta/<SHA512 identifier>.2342348324
> >>
> >> And now move/copy the file via the network to the archive, I'd have
> to
> >> copy both files (which is really annoying), and I'd guess the
> inode
> >> coupling would get los (and at least the name wouldn't fit
> anymore).
> >>
> >> So the whole thing is IMHO not even a workaround.
> > OK. So you're going to do XATTRs for us?
> >
> > Trond
> 
> Now that pNFS is perfect and labeled NFS has made it upstream, I think
> that 
> Steve D must be looking for something to keep him busy :)
> 
> ric
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 28, 2013, at 8:22 PM, Anand Avati <aavati@redhat.com> wrote:

> On 10/28/2013 01:07 PM, Ric Wheeler wrote:
>> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>> >On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer
>>> <calestyo@scientia.net> wrote:
>>> >
>>> >>On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>> >>>Then you end up with large directories and an extra name per inode
>>> that needs to
>>> >>>be stored and extra lookups for each file when you do a whole file
>>> system crawl.
>>> >>>
>>> >>>Certainly not as easy as adding and xattrs with that information :)
>>> >>And I think there's another reason why it wouldn't work...
>>> >>
>>> >>Imagine I change my system to encode what should be XATTRs in hardlink
>>> >>pseudo files...
>>> >>
>>> >>If I have such pair locally e.g. on my ext4:
>>> >>/foo/bar/actual/file
>>> >>/meta/<SHA512 identifier>.2342348324
>>> >>
>>> >>And now move/copy the file via the network to the archive, I'd have to
>>> >>copy both files (which is really annoying), and I'd guess the inode
>>> >>coupling would get los (and at least the name wouldn't fit anymore).
>>> >>
>>> >>So the whole thing is IMHO not even a workaround.
>>> >OK. So you're going to do XATTRs for us?
>>> >
>>> >Trond
>>> 
>>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>>> think that Steve D must be looking for something to keep him busy :)
>> 
>> I agree with Trond that we first really need good evidence about exactly
>> who wants this and why.
>> 
> 
> Some reasons why XATTRs in NFS could be useful w/ glusterfs:
> 
> - glusterfs exposes data locality through virtual extended attributes. One could do a getxattr("filename", "glusterfs.pathinfo") and get a parsable response about which servers store what parts and copies of the file. Such a mechanism is already used to implement Hadoop plugins for example (Hadoop plugin internally mounts gluster through FUSE where xattrs work). In some use-cases we really want to use NFS and still retain the ability to expose data locality through virtual xattrs, but lack of xattr support limits that possibility.
> 
> - gluster implements a "merkel tree" like inode attribute called "xtime" which is the recursive max mtime of all files/dirs in a subtree, maintained in real-time on all dirs. This is an extremely handy and powerful feature for implementing backups. This xtime is both stored as an xattr and exposed as an xattr. Users who chose to mount gluster through NFS protocol are giving up access this feature which is available only through xattrs.
> 
> - A very similar recursive function also provided by gluster is real-time size of dir subtrees, also exposed as extended attributes. For e.g a user instead of doing "du -hs /mnt/gluster/some/subdir" can instead do "getfattr -n glusterfs.quota.size /mnt/gluster/some/dir" and get instantaneous results. Again such a feature is not available for users mounting through NFS because of the lack of generic xattrs.
> 
> - A lot of our users have asked many times for the ability to use existing NFS servers as "gluster bricks" - because they have paid a ton of money and/or have a lot of data in there and do not want to "move it out". A major roadblocker for such a use case is the lack of xattr support. Gluster stores a lot of metadata in xattrs and therefore avoids having a "metadata server" (for e.g it stores details about which of the copies of a file/dir is fresh and stale in xattrs of that inode, it stores "hash ranges" of directories as xattrs on the directory inode, etc.) If only NFS mounts supported storing of these xattrs, we could support pre-existing NFS volumes as gluster bricks.
> 
> These are just some reasons on how implementing xattrs in NFS can be useful to one project.
> 
> It would be interesting to see how the server can control the caching behavior of such xattrs. For ex some of the (virtual) xattrs are better not cached by the client ever.
> 
> Avati

..and here is a perfect example of exactly what is wrong with xattrs. You're describing a private syscall interface, not a data storage format.

Trond

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 28, 2013, at 8:39 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> I might add a similar use case which we have at the faculty respectively
> the local supercomputing centre:
> 
> What we have there is big cluster filesystems (used to be Lustre but
> nowadays I think they use GPFS).
> There are custom (i.e. local) applications where XATTRs are used to
> attach some metadata to files, which works fine in these cluster
> filesystems since they have native support.
> 
> Now what they sometimes do (AFAIU) is, export the cluster fs via NFS, to
> nodes which don't have support for the cluster fs itself (sometimes the
> OS is simply too old, but one common use case is also, that they simply
> don't allow direct mounts outside of e.g. the super computer).
> 
> At that point, NFS looses the XATTRs.

Why do these nodes need access to the xattrs? What applications are they running that need them? Why can't those applications run on the native cluster instead?

Trond

Re: XATTRs in NFS?

[Ric Wheeler]

On 10/28/2013 08:49 PM, Myklebust, Trond wrote:
> On Oct 28, 2013, at 8:22 PM, Anand Avati <aavati@redhat.com> wrote:
>
>> On 10/28/2013 01:07 PM, Ric Wheeler wrote:
>>> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>>>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>>>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer
>>>> <calestyo@scientia.net> wrote:
>>>>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>>>>>> Then you end up with large directories and an extra name per inode
>>>> that needs to
>>>>>>> be stored and extra lookups for each file when you do a whole file
>>>> system crawl.
>>>>>>> Certainly not as easy as adding and xattrs with that information :)
>>>>>> And I think there's another reason why it wouldn't work...
>>>>>>
>>>>>> Imagine I change my system to encode what should be XATTRs in hardlink
>>>>>> pseudo files...
>>>>>>
>>>>>> If I have such pair locally e.g. on my ext4:
>>>>>> /foo/bar/actual/file
>>>>>> /meta/<SHA512 identifier>.2342348324
>>>>>>
>>>>>> And now move/copy the file via the network to the archive, I'd have to
>>>>>> copy both files (which is really annoying), and I'd guess the inode
>>>>>> coupling would get los (and at least the name wouldn't fit anymore).
>>>>>>
>>>>>> So the whole thing is IMHO not even a workaround.
>>>>> OK. So you're going to do XATTRs for us?
>>>>>
>>>>> Trond
>>>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>>>> think that Steve D must be looking for something to keep him busy :)
>>> I agree with Trond that we first really need good evidence about exactly
>>> who wants this and why.
>>>
>> Some reasons why XATTRs in NFS could be useful w/ glusterfs:
>>
>> - glusterfs exposes data locality through virtual extended attributes. One could do a getxattr("filename", "glusterfs.pathinfo") and get a parsable response about which servers store what parts and copies of the file. Such a mechanism is already used to implement Hadoop plugins for example (Hadoop plugin internally mounts gluster through FUSE where xattrs work). In some use-cases we really want to use NFS and still retain the ability to expose data locality through virtual xattrs, but lack of xattr support limits that possibility.
>>
>> - gluster implements a "merkel tree" like inode attribute called "xtime" which is the recursive max mtime of all files/dirs in a subtree, maintained in real-time on all dirs. This is an extremely handy and powerful feature for implementing backups. This xtime is both stored as an xattr and exposed as an xattr. Users who chose to mount gluster through NFS protocol are giving up access this feature which is available only through xattrs.
>>
>> - A very similar recursive function also provided by gluster is real-time size of dir subtrees, also exposed as extended attributes. For e.g a user instead of doing "du -hs /mnt/gluster/some/subdir" can instead do "getfattr -n glusterfs.quota.size /mnt/gluster/some/dir" and get instantaneous results. Again such a feature is not available for users mounting through NFS because of the lack of generic xattrs.
>>
>> - A lot of our users have asked many times for the ability to use existing NFS servers as "gluster bricks" - because they have paid a ton of money and/or have a lot of data in there and do not want to "move it out". A major roadblocker for such a use case is the lack of xattr support. Gluster stores a lot of metadata in xattrs and therefore avoids having a "metadata server" (for e.g it stores details about which of the copies of a file/dir is fresh and stale in xattrs of that inode, it stores "hash ranges" of directories as xattrs on the directory inode, etc.) If only NFS mounts supported storing of these xattrs, we could support pre-existing NFS volumes as gluster bricks.
>>
>> These are just some reasons on how implementing xattrs in NFS can be useful to one project.
>>
>> It would be interesting to see how the server can control the caching behavior of such xattrs. For ex some of the (virtual) xattrs are better not cached by the client ever.
>>
>> Avati
> ..and here is a perfect example of exactly what is wrong with xattrs. You're describing a private syscall interface, not a data storage format.
>
> Trond

What Avati described is having an application store user defined attributes in a 
file in a standard way - pretty much every local file system does this.  I don't 
get the private syscall interface comment or the need to re-argue a battle that 
was waged and lost effectively *years* ago :)

Ric

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

On Tue, 2013-10-29 at 00:53 +0000, Myklebust, Trond wrote:
> Why do these nodes need access to the xattrs?
I have no concrete idea... I just know what these guys are doing not
why... AFAIK the files resembles event collections from scientific
measurements,... and what they store is how these events have been
processed.
I do not even claim that this couldn't be done otherwise... it's just
how things are.

> What applications are they running that need them?
Home brew event processing software... 

>  Why can't those applications run on the native cluster instead?
That's largely politics and money...
The super computer is highly expensive and you only get time there after
you've made a official proposal and a commission has granted you're
request.
So while some of the data lies there in the shared fs (the GPFS),...
people also want to process the data from the normal Linux cluster
(where getting computing time is far more easy)... so they somehow need
to access the data which is done via the NFS exports.


Don't get me wrong, Trond, I'm not saying that this is the only or best
solution... I say this is how things are done in reality.
Obviously they must have found some workaround now... but I just wanted
to point out that there *are* some use cases, whether they're perfect or
not.


Cheers,
Chris.

Re: XATTRs in NFS?

[Anand Avati]

On 10/28/2013 06:26 PM, Myklebust, Trond wrote:

>
> That battle may have been fought and won within the glusterfs community, but why should we wave the white flag without a discussion? I don't see how what he described above has anything to do with user defined attributes. He's describing how he wants to export quota information and xtime through a private xattr interface that is currently unique to glusterfs. How is that not a private syscall interface?
>

Exposing quota informtion is use "from the top". Note the other point I 
mention about using NFS volumes as "gluster bricks" where we store 
xattrs as dumb and persistent key/values associated with file/dir inodes 
(fresh/stale info for replication, hash ranges for dirs, quota acconting 
info per-dir, xtime per dir).

> Which of the mainstream filesystems have their own private xattr namespaces like the above?
>

Why should NFS need to worry? As long as it acts like a pass-through 
(like every other call it supports).

Avati

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 28, 2013, at 9:00 PM, Ric Wheeler <rwheeler@redhat.com> wrote:

> On 10/28/2013 08:49 PM, Myklebust, Trond wrote:
>> On Oct 28, 2013, at 8:22 PM, Anand Avati <aavati@redhat.com> wrote:
>> 
>>> On 10/28/2013 01:07 PM, Ric Wheeler wrote:
>>>> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>>>>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>>>>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer
>>>>> <calestyo@scientia.net> wrote:
>>>>>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>>>>>>> Then you end up with large directories and an extra name per inode
>>>>> that needs to
>>>>>>>> be stored and extra lookups for each file when you do a whole file
>>>>> system crawl.
>>>>>>>> Certainly not as easy as adding and xattrs with that information :)
>>>>>>> And I think there's another reason why it wouldn't work...
>>>>>>> 
>>>>>>> Imagine I change my system to encode what should be XATTRs in hardlink
>>>>>>> pseudo files...
>>>>>>> 
>>>>>>> If I have such pair locally e.g. on my ext4:
>>>>>>> /foo/bar/actual/file
>>>>>>> /meta/<SHA512 identifier>.2342348324
>>>>>>> 
>>>>>>> And now move/copy the file via the network to the archive, I'd have to
>>>>>>> copy both files (which is really annoying), and I'd guess the inode
>>>>>>> coupling would get los (and at least the name wouldn't fit anymore).
>>>>>>> 
>>>>>>> So the whole thing is IMHO not even a workaround.
>>>>>> OK. So you're going to do XATTRs for us?
>>>>>> 
>>>>>> Trond
>>>>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>>>>> think that Steve D must be looking for something to keep him busy :)
>>>> I agree with Trond that we first really need good evidence about exactly
>>>> who wants this and why.
>>>> 
>>> Some reasons why XATTRs in NFS could be useful w/ glusterfs:
>>> 
>>> - glusterfs exposes data locality through virtual extended attributes. One could do a getxattr("filename", "glusterfs.pathinfo") and get a parsable response about which servers store what parts and copies of the file. Such a mechanism is already used to implement Hadoop plugins for example (Hadoop plugin internally mounts gluster through FUSE where xattrs work). In some use-cases we really want to use NFS and still retain the ability to expose data locality through virtual xattrs, but lack of xattr support limits that possibility.
>>> 
>>> - gluster implements a "merkel tree" like inode attribute called "xtime" which is the recursive max mtime of all files/dirs in a subtree, maintained in real-time on all dirs. This is an extremely handy and powerful feature for implementing backups. This xtime is both stored as an xattr and exposed as an xattr. Users who chose to mount gluster through NFS protocol are giving up access this feature which is available only through xattrs.
>>> 
>>> - A very similar recursive function also provided by gluster is real-time size of dir subtrees, also exposed as extended attributes. For e.g a user instead of doing "du -hs /mnt/gluster/some/subdir" can instead do "getfattr -n glusterfs.quota.size /mnt/gluster/some/dir" and get instantaneous results. Again such a feature is not available for users mounting through NFS because of the lack of generic xattrs.
>>> 
>>> - A lot of our users have asked many times for the ability to use existing NFS servers as "gluster bricks" - because they have paid a ton of money and/or have a lot of data in there and do not want to "move it out". A major roadblocker for such a use case is the lack of xattr support. Gluster stores a lot of metadata in xattrs and therefore avoids having a "metadata server" (for e.g it stores details about which of the copies of a file/dir is fresh and stale in xattrs of that inode, it stores "hash ranges" of directories as xattrs on the directory inode, etc.) If only NFS mounts supported storing of these xattrs, we could support pre-existing NFS volumes as gluster bricks.
>>> 
>>> These are just some reasons on how implementing xattrs in NFS can be useful to one project.
>>> 
>>> It would be interesting to see how the server can control the caching behavior of such xattrs. For ex some of the (virtual) xattrs are better not cached by the client ever.
>>> 
>>> Avati
>> ..and here is a perfect example of exactly what is wrong with xattrs. You're describing a private syscall interface, not a data storage format.
>> 
>> Trond
> 
> What Avati described is having an application store user defined attributes in a file in a standard way - pretty much every local file system does this.  I don't get the private syscall interface comment or the need to re-argue a battle that was waged and lost effectively *years* ago :)
> 

That battle may have been fought and won within the glusterfs community, but why should we wave the white flag without a discussion? I don't see how what he described above has anything to do with user defined attributes. He's describing how he wants to export quota information and xtime through a private xattr interface that is currently unique to glusterfs. How is that not a private syscall interface?

Which of the mainstream filesystems have their own private xattr namespaces like the above?

Trond

Re: XATTRs in NFS?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

Trond,... since you ask for motivations pro NFS-XATTRs, I wondered what
are the strong reasons against?


Of course someone has to do the work,... but that's not really and
argument pro or con NFS-XATTRs, is it?

The security problems with namespaces like trusted. or so can probably
be solved quite easy, e.g. simply by not supporting or
ignoring/rejecting them.

You've mentioned caching issues,... could you elaborate a bit on that?
How is that much different from caching any other file metadata NFS
transfers?

Are there other (technical) reasons?


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 28, 2013, at 9:24 PM, Anand Avati <aavati@redhat.com> wrote:

> On 10/28/2013 06:26 PM, Myklebust, Trond wrote:
> 
>> 
>> That battle may have been fought and won within the glusterfs community, but why should we wave the white flag without a discussion? I don't see how what he described above has anything to do with user defined attributes. He's describing how he wants to export quota information and xtime through a private xattr interface that is currently unique to glusterfs. How is that not a private syscall interface?
>> 
> 
> Exposing quota informtion is use "from the top". Note the other point I mention about using NFS volumes as "gluster bricks" where we store xattrs as dumb and persistent key/values associated with file/dir inodes (fresh/stale info for replication, hash ranges for dirs, quota acconting info per-dir, xtime per dir).

>> Which of the mainstream filesystems have their own private xattr namespaces like the above?
>> 
> 
> Why should NFS need to worry? As long as it acts like a pass-through (like every other call it supports).

We need to worry because we don't know what side-effects your private interface will have. How does it affect our caching model? How do we debug any problems that arise? Are there any security implications that we need to know about?

Trond

RE: XATTRs in NFS?

[Anand Avati]

[-- Attachment #1: Type: text/plain, Size: 1986 bytes --]

(Sorry for top post)

Ah, note that by "glusterfs.<something>" i was meaning "<ns>.glusterfs.<something>" where <ns> is like "user". It just needs to be a mechanism to store key values on the inode.

Avati

Sent from my Android phone using TouchDown (www.nitrodesk.com)


-----Original Message-----
From: Myklebust, Trond [Trond.Myklebust@netapp.com]
Received: Monday, 28 Oct 2013, 18:52
To: Anand Avati [aavati@redhat.com]
CC: Wheeler Ric [rwheeler@redhat.com]; Dr Fields James Bruce [bfields@redhat.com]; Christoph Anton Mitterer [calestyo@scientia.net]; Mailing List Linux NFS [linux-nfs@vger.kernel.org]; Dickson Steve [steved@redhat.com]
Subject: Re: XATTRs in NFS?



On Oct 28, 2013, at 9:24 PM, Anand Avati <aavati@redhat.com> wrote:

> On 10/28/2013 06:26 PM, Myklebust, Trond wrote:
> 
>> 
>> That battle may have been fought and won within the glusterfs community, but why should we wave the white flag without a discussion? I don't see how what he described above has anything to do with user defined attributes. He's describing how he wants to export quota information and xtime through a private xattr interface that is currently unique to glusterfs. How is that not a private syscall interface?
>> 
> 
> Exposing quota informtion is use "from the top". Note the other point I mention about using NFS volumes as "gluster bricks" where we store xattrs as dumb and persistent key/values associated with file/dir inodes (fresh/stale info for replication, hash ranges for dirs, quota acconting info per-dir, xtime per dir).

>> Which of the mainstream filesystems have their own private xattr namespaces like the above?
>> 
> 
> Why should NFS need to worry? As long as it acts like a pass-through (like every other call it supports).

We need to worry because we don't know what side-effects your private interface will have. How does it affect our caching model? How do we debug any problems that arise? Are there any security implications that we need to know about?

Trond

Re: XATTRs in NFS?

[Myklebust, Trond]

On Oct 28, 2013, at 9:39 PM, Christoph Anton Mitterer <calestyo@scientia.net> wrote:

> Trond,... since you ask for motivations pro NFS-XATTRs, I wondered what
> are the strong reasons against?

As I've already told you: you're asking for the addition of a feature which is inadequately defined, and is not documented in any standard.

> Of course someone has to do the work,... but that's not really and
> argument pro or con NFS-XATTRs, is it?
> 
> The security problems with namespaces like trusted. or so can probably
> be solved quite easy, e.g. simply by not supporting or
> ignoring/rejecting them.
> 
> You've mentioned caching issues,... could you elaborate a bit on that?
> How is that much different from caching any other file metadata NFS
> transfers?

The standard metadata such as ctime, mtime, size, .... are all part of an existing NFS caching model called close-to-open. We know how they change when the filesystem data changes.

How do I know when it is safe to cache your checksum xattr? I don't even know what it is, let alone what its relation is to the other filesystem objects.
Let's say client B modifies the file and updates the checksum. When client A opens the file, it will see that the data has changed, but how does it know that it also needs to update the xattr information?
Alternatively, let's say that client A reads the file and checksum data after client B has updated the file, but before it updates the checksum. What will cause client A to stop caching the stale checksum once client B has updated it?

Trond

Re: XATTRs in NFS?

[Marc Eshel]

Hi Trond,
I am just trying to see if there any path here to get xattr over NFS.

Assuming that there is agreement that xattr are used and needed, and we 
need to build a better case here but just give alternative solutions 
doesn't mean that is not needed, almost every thing can be done with an 
additional DB that associates attributes with files but it is not the best 
or right way.

So if we had an NFS protocol extension (might not be named attributes) 
that mapped  to and was semantically compatibly to Linux xattr, as Nico 
wrote, would that be enough to consider adding it to the Linux NFS?

I see the need to show requirement and better interface over the NFS 
protocol (with no security holes)  but not sure if we have another problem 
with standardizing xattr in Linux?

Thanks, Marc. 



From:   "Myklebust, Trond" <Trond.Myklebust@netapp.com>
To:     Christoph Anton Mitterer <calestyo@scientia.net>, 
Cc:     Wheeler Ric <rwheeler@redhat.com>, Anand Avati 
<aavati@redhat.com>, "Dr Fields James Bruce" <bfields@redhat.com>, Mailing 
List Linux NFS <linux-nfs@vger.kernel.org>, Dickson Steve 
<steved@redhat.com>
Date:   10/28/2013 07:28 PM
Subject:        Re: XATTRs in NFS?
Sent by:        linux-nfs-owner@vger.kernel.org




On Oct 28, 2013, at 9:39 PM, Christoph Anton Mitterer 
<calestyo@scientia.net> wrote:

> Trond,... since you ask for motivations pro NFS-XATTRs, I wondered what
> are the strong reasons against?

As I've already told you: you're asking for the addition of a feature 
which is inadequately defined, and is not documented in any standard.

> Of course someone has to do the work,... but that's not really and
> argument pro or con NFS-XATTRs, is it?
> 
> The security problems with namespaces like trusted. or so can probably
> be solved quite easy, e.g. simply by not supporting or
> ignoring/rejecting them.
> 
> You've mentioned caching issues,... could you elaborate a bit on that?
> How is that much different from caching any other file metadata NFS
> transfers?

The standard metadata such as ctime, mtime, size, .... are all part of an 
existing NFS caching model called close-to-open. We know how they change 
when the filesystem data changes.

How do I know when it is safe to cache your checksum xattr? I don't even 
know what it is, let alone what its relation is to the other filesystem 
objects.
Let's say client B modifies the file and updates the checksum. When client 
A opens the file, it will see that the data has changed, but how does it 
know that it also needs to update the xattr information?
Alternatively, let's say that client A reads the file and checksum data 
after client B has updated the file, but before it updates the checksum. 
What will cause client A to stop caching the stale checksum once client B 
has updated it?

Trond--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html