xattr performance

xattr performance

[Christian Kujau]

Hi,

a while ago I was setting & reading extended attributes to ~25000 files 
in a directory structure on an XFS filesystem. The files were usually a 
few MB in size, but some where up to 2GB in size.

Anyway, I *felt* that setting or reading these xattrs was going very
slowly. While the storage may be not the fastest, stat()'ing these
files was fine, but getfattr/setfattr took very long.

I got curious and while it turned out that the slowness was related to the 
wrapper script I used to read/set those values, I created a little test 
suite to 1) create a few thousand files and 2) do xattr operations on 
them and see if xattr performance was filesystem specific:

   http://nerdbynature.de/bits/xattr/

Not very sophisticated, true. But it was interesting to see that 
ext3/ext4/xfs behaved kinda well for all these tests; btrfs/jfs/reiserfs
sometimes took way longer than the others.

Christian.
-- 
BOFH excuse #43:

boss forgot system password

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xattr performance

[Ben Myers]

Hey Christian,

On Fri, May 17, 2013 at 05:02:21AM -0700, Christian Kujau wrote:
> a while ago I was setting & reading extended attributes to ~25000 files 
> in a directory structure on an XFS filesystem. The files were usually a 
> few MB in size, but some where up to 2GB in size.
> 
> Anyway, I *felt* that setting or reading these xattrs was going very
> slowly. While the storage may be not the fastest, stat()'ing these
> files was fine, but getfattr/setfattr took very long.
> 
> I got curious and while it turned out that the slowness was related to the 
> wrapper script I used to read/set those values, I created a little test 
> suite to 1) create a few thousand files and 2) do xattr operations on 
> them and see if xattr performance was filesystem specific:
> 
>    http://nerdbynature.de/bits/xattr/
> 
> Not very sophisticated, true. But it was interesting to see that 
> ext3/ext4/xfs behaved kinda well for all these tests; btrfs/jfs/reiserfs
> sometimes took way longer than the others.

Very interesting results!  One wrinkle that you might want to consider with XFS
is the overall size of the attributes versus the size of the inode.  You can
choose inode sizes between 256 bytes and 2k in powers of two, and we always
allocate them in chunks of 64.  The 'literal' area is the space after the inode
core and before the next one... it's best described here:
http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/On-disk_Inode.html

The short version:

inode core (96 bytes) + literal area == inode size

The data and attribute forks share the literal area.  If the attributes get too
big to fit inside the literal area with the data fork they will go out of line
and be stored elsewhere in the filesystem.  The performance characteristics of
inline vs out-of-line attributes are significantly different.  That might be
what you experienced when you felt that setting/reading xattrs was taking a
long time. 

Anyway... If you're a heavy user of EAs you might benefit from using larger
inodes.  Just something to consider.  Cool tests!  ;)

Regards,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xattr performance

[Christian Kujau]

On Fri, 17 May 2013 at 10:44, Ben Myers wrote:
> The short version:
> inode core (96 bytes) + literal area == inode size

Ah, yes indeed. I had another issue earlier on the same filesystem and 
Christoph told me[0] that I may have run out of inode attributes. 

xfs_info reports isize=256 for this filesystem and now that I'm using 
xattr even more (I'm storing sha256 checksum plus the filename in each 
file's xattr) this looks like it might exceed the literal area after all. 

> Anyway... If you're a heavy user of EAs you might benefit from using larger
> inodes.  Just something to consider.  Cool tests!  ;)

Yeah, next time I'll have to take this into consideration, but the fs was 
created long ago and I didn't plan to use xattr. Now I do but mkfs is not 
an option right now.

Thanks for the insight,
Christian.

[0] http://oss.sgi.com/archives/xfs/2012-07/msg00116.html
-- 
BOFH excuse #247:

Due to Federal Budget problems we have been forced to cut back on the number of users able to access the system at one time. (namely none allowed....)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs