What's a realistic size for xattr?

What's a realistic size for xattr?

[Charles Manning]

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

>From my (limited) knowledge of xattr it seems that in theory you could store a 
multi-Mbyte database in xattr, but in practice a smaller size is far more 
reasonable. Clearly storing/managing a small blob is going to be a lot 
simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

Thanx.

Charles

Re: What's a realistic size for xattr?

[Nathan Scott]

On Thu, 2008-03-13 at 11:06 +1300, Charles Manning wrote:
> I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.
> 
> From my (limited) knowledge of xattr it seems that in theory you could store a 
> multi-Mbyte database in xattr, but in practice a smaller size is far more 
> reasonable. Clearly storing/managing a small blob is going to be a lot 
> simpler.
> 
> What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...
> 

XATTR_SIZE_MAX.

-- 
Nathan

Re: What's a realistic size for xattr?

[Dave Quigley]

On Thu, 2008-03-13 at 11:06 +1300, Charles Manning wrote:
> I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.
> 
> >From my (limited) knowledge of xattr it seems that in theory you could store a 
> multi-Mbyte database in xattr, but in practice a smaller size is far more 
> reasonable. Clearly storing/managing a small blob is going to be a lot 
> simpler.
> 
> What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...
> 
> Thanx.
> 
> Charles
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

I don't know about specific file system limits on xattrs but from a
user-space perspective the sys_setxattr->setxattr call chain enforces a
limit of no more than 64k on an xattr value. In addition the list of
xattr names is also limited to 64k with a name being no longer than 255
chars. From your wording above it seems that you want to store the
xattrs as their own records. One approach (which is different from what
I think you want) is to store the xattrs in the inode upto a point and
then if necessary allocate space from them elsewhere. I'm not familiar
with what you are trying to implement but I believe this is the way XFS
handles xattrs.

#define XATTR_NAME_MAX   255    /* # chars in an extended attribute name */
#define XATTR_SIZE_MAX 65536    /* size of an extended attribute value (64k) */
#define XATTR_LIST_MAX 65536    /* size of extended attribute namelist (64k) */

Dave

Re: What's a realistic size for xattr?

[Dave Quigley]

On Thu, 2008-03-13 at 11:06 +1300, Charles Manning wrote:
> I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.
> 
> >From my (limited) knowledge of xattr it seems that in theory you could store a 
> multi-Mbyte database in xattr, but in practice a smaller size is far more 
> reasonable. Clearly storing/managing a small blob is going to be a lot 
> simpler.
> 
> What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...
> 
> Thanx.
> 
> Charles
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

I just realized that I never really answered your question. I've yet to
see a reasonable SELinux context over 128 characters and Smack labels
should be shorter than that. I don't know what beagle places in xattrs
but I think that would be a good place to take a look for more practical
xattr sizes. 

Dave

Re: What's a realistic size for xattr?

[Charles Manning]

On Thursday 13 March 2008 12:32:33 Dave Quigley wrote:
> On Thu, 2008-03-13 at 11:06 +1300, Charles Manning wrote:
> > I'm trying to figure out a reasonable approach to implementing xattr in
> > YAFFS.
> >
> > >From my (limited) knowledge of xattr it seems that in theory you could
> > > store a
> >
> > multi-Mbyte database in xattr, but in practice a smaller size is far more
> > reasonable. Clearly storing/managing a small blob is going to be a lot
> > simpler.
> >
> > What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...
> >
> > Thanx.
> >
> > Charles
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel"
> > in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> I just realized that I never really answered your question. I've yet to
> see a reasonable SELinux context over 128 characters and Smack labels
> should be shorter than that. I don't know what beagle places in xattrs
> but I think that would be a good place to take a look for more practical
> xattr sizes.
>
> Dave


Is that 128 characters per xattr value or for the whole set of xattribs 
attached to a file?

What I'm trying to figure out is a reasonable size for all the xattribs 
together. ie. If I store xattribs as a single blob containing all the 
name:value pairs, how big will that blob need to be?

I should have perhaps given some extra context here.

I'm trying to find the bang-for-bucks trade-off point for a simple 
implementation.

YAFFS is a flash file system and is typically used in embedded systems 
(phones, routers, printers, point-of-sale,... etc). I guess limited use of 
SELinux is a reasonable benchmark for what is used here.

Huge fs for corporate servers don't really fit in this picture so I guess 
vastly complex ACLs , while theoretically possible, don't really fall into 
the yaffs space.

Re: What's a realistic size for xattr?

[Dave Quigley]

On Thu, 2008-03-13 at 13:03 +1300, Charles Manning wrote:
> On Thursday 13 March 2008 12:32:33 Dave Quigley wrote:
> > On Thu, 2008-03-13 at 11:06 +1300, Charles Manning wrote:
> > > I'm trying to figure out a reasonable approach to implementing xattr in
> > > YAFFS.
> > >
> > > >From my (limited) knowledge of xattr it seems that in theory you could
> > > > store a
> > >
> > > multi-Mbyte database in xattr, but in practice a smaller size is far more
> > > reasonable. Clearly storing/managing a small blob is going to be a lot
> > > simpler.
> > >
> > > What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...
> > >
> > > Thanx.
> > >
> > > Charles
> > >
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel"
> > > in the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > I just realized that I never really answered your question. I've yet to
> > see a reasonable SELinux context over 128 characters and Smack labels
> > should be shorter than that. I don't know what beagle places in xattrs
> > but I think that would be a good place to take a look for more practical
> > xattr sizes.
> >
> > Dave
> 
> 
> Is that 128 characters per xattr value or for the whole set of xattribs 
> attached to a file?

I'm talking about the value part so your name will be security.selinux
and a value of usually no more than 128 characters. However with
multiple LSMs its possible that you may have security.selinux
security.smack security.newlsmnotcreatedyet.
> 
> What I'm trying to figure out is a reasonable size for all the xattribs 
> together. ie. If I store xattribs as a single blob containing all the 
> name:value pairs, how big will that blob need to be?
> 
> I should have perhaps given some extra context here.
> 
> I'm trying to find the bang-for-bucks trade-off point for a simple 
> implementation.
> 
> YAFFS is a flash file system and is typically used in embedded systems 
> (phones, routers, printers, point-of-sale,... etc). I guess limited use of 
> SELinux is a reasonable benchmark for what is used here.
> 
> Huge fs for corporate servers don't really fit in this picture so I guess 
> vastly complex ACLs , while theoretically possible, don't really fall into 
> the yaffs space.
> 
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What's a realistic size for xattr?

[Jamie Lokier]

Dave Quigley wrote:
> > >From my (limited) knowledge of xattr it seems that in theory you could store a 
> > multi-Mbyte database in xattr, but in practice a smaller size is far more 
> > reasonable. Clearly storing/managing a small blob is going to be a lot 
> > simpler.
> > 
> > What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...
> 
> I just realized that I never really answered your question. I've yet to
> see a reasonable SELinux context over 128 characters and Smack labels
> should be shorter than that. I don't know what beagle places in xattrs
> but I think that would be a good place to take a look for more practical
> xattr sizes. 

People copy files from other systems using rsync, so it's not just
Linux uses of xattrs.  Also, some programs can store their own data,
such as search metadata, desktop metadata, checksums, descriptions
(similar to MP3 id tags, but in xattrs) and all sorts of things.

On Macs, xattrs are used to store the "resource fork", which contains
icons etc. so can be quite large.  Of course that's not done on Linux.
People do sometimes try to copy those files to Linux and back, and it
generally fails because Linux refuses the large xattrs.

-- Jamie

Re: What's a realistic size for xattr?

[Brad Boyer]

On Thu, Mar 13, 2008 at 12:07:37AM +0000, Jamie Lokier wrote:
> On Macs, xattrs are used to store the "resource fork", which contains
> icons etc. so can be quite large.  Of course that's not done on Linux.
> People do sometimes try to copy those files to Linux and back, and it
> generally fails because Linux refuses the large xattrs.

The resource fork in HFS/HFS+ is not considered an xattr at all. In fact,
it's effectively another file that just happens to be attached to the
same inode-equivalent. There is a data fork (seen as a normal file
for unix) and the resource fork, and it's just the posix style API that
treats one as more important than the other. The two are considered
peers in the on-disk format and in the classic mac API.

The only way I've seen the resource fork stored on a non-Apple fs is
as a named stream style format or some kind of encoded single file
format like AppleSingle/AppleDouble or BinHex.

If you really want to talk about size, there is a lot more than icons
and metadata stored in the resource fork. In the m68k era, the compiled
code was also stored in the resource fork with each segment showing up
as a numbered resource of type 'CODE' that could be dynamically loaded
and released. If you look at an application from those days, it most
likely has a 0-length data fork.

	Brad Boyer
	flar@allandria.com

Re: What's a realistic size for xattr?

[Nicholas Miell]

On Thu, 2008-03-13 at 00:07 +0000, Jamie Lokier wrote:
> Dave Quigley wrote:
> > > >From my (limited) knowledge of xattr it seems that in theory you could store a 
> > > multi-Mbyte database in xattr, but in practice a smaller size is far more 
> > > reasonable. Clearly storing/managing a small blob is going to be a lot 
> > > simpler.
> > > 
> > > What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...
> > 
> > I just realized that I never really answered your question. I've yet to
> > see a reasonable SELinux context over 128 characters and Smack labels
> > should be shorter than that. I don't know what beagle places in xattrs
> > but I think that would be a good place to take a look for more practical
> > xattr sizes. 
> 
> People copy files from other systems using rsync, so it's not just
> Linux uses of xattrs.  Also, some programs can store their own data,
> such as search metadata, desktop metadata, checksums, descriptions
> (similar to MP3 id tags, but in xattrs) and all sorts of things.
> 
> On Macs, xattrs are used to store the "resource fork", which contains
> icons etc. so can be quite large.  Of course that's not done on Linux.
> People do sometimes try to copy those files to Linux and back, and it
> generally fails because Linux refuses the large xattrs.
> 
> -- Jamie

Linux xattrs are nothing more than named extensions to struct stat. It's
unfortunate that Solaris reused the xattr name for a completely
different concept that already had existing names (named streams AKA
forks), but they aren't equivalent and are not intended to be
equivalent.

Pretending that xattrs are named streams is just a path to madness and
suffering, considering the Linux API is based around atomic attribute
querying and modification instead of the standard Unix I/O primitives
and the Powers That Be are vehemently against the implementation of
named streams in the Linux VFS.

-- 
Nicholas Miell <nmiell@comcast.net>

Re: What's a realistic size for xattr?

[Andreas Dilger]

On Mar 13, 2008  11:06 +1300, Charles Manning wrote:
> I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.
> 
> From my (limited) knowledge of xattr it seems that in theory you could store a
> multi-Mbyte database in xattr, but in practice a smaller size is far more 
> reasonable. Clearly storing/managing a small blob is going to be a lot 
> simpler.
> 
> What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

As a point of reference, ext2/3/4 limit the total xattr size to 1 data block
(4kB commonly), and I believe XFS has a total limit of 64kB.  ZFS and NTFS
have resource forks, which can store arbitrary data, but they have a different
API.  On NTFS the word is that resource forks are mostly used by root kits.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: What's a realistic size for xattr?

[Florian Weimer]

* Andreas Dilger:

> On NTFS the word is that resource forks are mostly used by root kits.

They are also used to mark programs you've downloaded from the Internet,
so that a warning can be displayed when you open them.