Deleting files with extended attributes is dead slow

Deleting files with extended attributes is dead slow

[Bernd Schubert]

Hi there,

with FhGFS we may use extended attributes to store file meta data and 
with ext3/ext4 that also works very well and the rate to create files 
and to write those EAs (create() + fsetxattr()  is about 2.5 to 3 times 
faster than with a create() + write(). Size of those EA data is about 
256 bytes depending on the number of storage stripes.
However, with XFS using extended attributes is *extremely* slow. Here 
are some numbers with ext4 and xfs using a patched [1] bonnie++

> schubert@fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestXFS/ -s0 -n 1:256:256:1 -r 0 -X
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version  1.96       ------Sequential Create------ --------Random Create--------
> fsdevel3            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>           1:256:256   330   6 +++++ +++    47   1   469   9 +++++ +++    31   1
> Latency               962ms      36us     140ms     866ms      28us     311ms
> 1.96,1.96,fsdevel3,1,1314378941,,,,,,,,,,,,,,1,256,256,,,330,6,+++++,+++,47,1,469,9,+++++,+++,31,1,,,,,,,962ms,36us,140ms,866ms,28us,311ms



> schubert@fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestEXT4/ -s0 -n 100:256:256:10 -r 0 -X
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version  1.96       ------Sequential Create------ --------Random Create--------
> fsdevel3            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>      100:256:256/10 21348  52 +++++ +++ 32878  61 25286  60 +++++ +++ 25873  53
> Latency               746us    1926us    2553us     653us     118us   32250us
> 1.96,1.96,fsdevel3,1,1314379136,,,,,,,,,,,,,,100,256,256,,10,21348,52,+++++,+++,32878,61,25286,60,+++++,+++,25873,53,,,,,,,746us,1926us,2553us,653us,118us,32250us


NOTE: For ext4 I had to increase the number of files by factor 100 to 
get any sane number (it would only print '+++++' otherwise). Running the 
benchmark with the same numbers on xfs deleted in so slow delete 
numbers, that it probably still would be busy to delete files by tomorrow.


The xfs file system was formated with these parameters:
> mkfs.xfs -f -i size=512 -i maxpct=90  -l lazy-count=1 -n size=64k /dev/mapper/vg0fsdev3-XFStest


ext4 was formated to have an inode size of 512B and to have a joural 
size of 400MB.

Both file systems are mounted with "nobarrier" (real FhGFS production 
systems usually have raid controllers with battery backups).


Any ideas why xfs is so slow with extended attributes? Without extended 
attributes, so skipping the "-X" option in our patched bonnie also 
results in slower Create numbers (about 4500 on xfs vs. 7500 on ext4) 
compared to ext4, but that is 'only' factor 1.67 and not 50 or more as 
with EAs.


Thanks,
Bernd


[1] Updated bonnie to support extended attributes:
https://bitbucket.org/aakef/bonnie/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Deleting files with extended attributes is dead slow

[Christoph Hellwig]

On Fri, Aug 12, 2011 at 08:19:53PM +0200, Bernd Schubert wrote:
> Hi there,
> 
> with FhGFS we may use extended attributes to store file meta data
> and with ext3/ext4 that also works very well and the rate to create
> files and to write those EAs (create() + fsetxattr()  is about 2.5
> to 3 times faster than with a create() + write(). Size of those EA
> data is about 256 bytes depending on the number of storage stripes.
> However, with XFS using extended attributes is *extremely* slow.
> Here are some numbers with ext4 and xfs using a patched [1] bonnie++

Currently XFS uses synchronous transactions when retiring inodes with
xattrs on them.  So far I've not actually found a good reason for that,
so I suspect we can remove it.  I'll do another audit of it, and will
send you a patch for it after doing basic QA.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Deleting files with extended attributes is dead slow

[Christoph Hellwig]

The patch below should help with the performance of deleting files
that have external attributes.  But I still wonder why you see
attributes outside the inode if you are using 512 byte inodes and around
256k of attrs.  Can you check tat you really have external blocks for
attributes using xfs_bmap -a on the files in your workload.  Can you
also check if this goes away using 1k or 2k inode sizes?


---
From: Christoph Hellwig <hch@lst.de>
Subject: xfs: avoid synchronous transactions when deleting attr blocks

Currently xfs_attr_inactive causes a synchronous transactions if we are
removing a file that has any extents allocated to the attribute fork.  The
code looks a like a relict from the days before the busy extent list, but
with the busy extent list we avoid reusing data and attr extents that have
been freed but not commited yet, so this code is just as superflous as
the synchronous transactions for data blocks.

Signed-off-by: Christoph Hellwig <hch@lst.de>

Index: xfs/fs/xfs/xfs_attr.c
===================================================================
--- xfs.orig/fs/xfs/xfs_attr.c	2011-08-15 14:31:15.485329718 -0700
+++ xfs/fs/xfs/xfs_attr.c	2011-08-15 14:31:32.025240113 -0700
@@ -823,18 +823,6 @@ xfs_attr_inactive(xfs_inode_t *dp)
 	if (error)
 		goto out;
 
-	/*
-	 * Signal synchronous inactive transactions unless this is a
-	 * synchronous mount filesystem in which case we know that we're here
-	 * because we've been called out of xfs_inactive which means that the
-	 * last reference is gone and the unlink transaction has already hit
-	 * the disk so async inactive transactions are safe.
-	 */
-	if (!(mp->m_flags & XFS_MOUNT_WSYNC)) {
-		if (dp->i_d.di_anextents > 0)
-			xfs_trans_set_sync(trans);
-	}
-
 	error = xfs_itruncate_extents(&trans, dp, XFS_ATTR_FORK, 0);
 	if (error)
 		goto out;
Index: xfs/fs/xfs/xfs_bmap.c
===================================================================
--- xfs.orig/fs/xfs/xfs_bmap.c	2011-08-15 14:33:59.801106208 -0700
+++ xfs/fs/xfs/xfs_bmap.c	2011-08-16 09:01:24.716497619 -0700
@@ -3782,19 +3782,11 @@ xfs_bmap_compute_maxlevels(
  * Routine to be called at transaction's end by xfs_bmapi, xfs_bunmapi
  * caller.  Frees all the extents that need freeing, which must be done
  * last due to locking considerations.  We never free any extents in
- * the first transaction.  This is to allow the caller to make the first
- * transaction a synchronous one so that the pointers to the data being
- * broken in this transaction will be permanent before the data is actually
- * freed.  This is necessary to prevent blocks from being reallocated
- * and written to before the free and reallocation are actually permanent.
- * We do not just make the first transaction synchronous here, because
- * there are more efficient ways to gain the same protection in some cases
- * (see the file truncation code).
+ * the first transaction.
  *
  * Return 1 if the given transaction was committed and a new one
  * started, and 0 otherwise in the committed parameter.
  */
-/*ARGSUSED*/
 int						/* error */
 xfs_bmap_finish(
 	xfs_trans_t		**tp,		/* transaction pointer addr */

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Deleting files with extended attributes is dead slow

[Christoph Hellwig]

On Wed, Aug 17, 2011 at 03:05:28PM +0200, Bernd Schubert wrote:
> >(squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
> >/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
> >        0: [0..7]: 92304..92311
> 
> (Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to
> tell me).

It means that you are having an extent spanning 8 blocks for xattr
storage, that map to physical blocks 92304 to 9231 in the filesystem.

It sounds to me like your workload has a lot more than 256 bytes of
xattrs, or the underlying code is doing something rather stupid.

> Looking at 'top' and 'iostat -x' outout, I noticed we are actually
> not limited by io to disk, but CPU bound. If you should be
> interested, I have attached 'perf record -g' and 'perf report -g'
> outout, of the bonnie file create (create + fsetfattr() ) phase.

It's mostly spending a lot of time on copying things into the CIL
buffers, which is expected and intentional as that allows for additional
parallelity.  I you'd switch the workload to multiple intances doing
the create in parallel you should be able to scale to better numbers.

> >     100:256:256/10 37026  91 +++++ +++ 43691  93 35960  92 +++++ +++ 40708  92
> >Latency              4328us     765us    2960us     527us     440us    1075us
> >1.96,1.96,fslab2,1,1313594619,,,,,,,,,,,,,,100,256,256,,10,37026,91,+++++,+++,43691,93,35960,92,+++++,+++,40708,92,,,,,,,4328us,765us,2960us,527us,440us,1075us
> 
> 
> xfs:
> mkfs.xfs -f -i size=512 -i maxpct=90  -l lazy-count=1 -n size=64k /dev/sdd

Do 64k dir blocks actually help you with the workload?  They also tend
to do a lot of useless memcpys in their current form, although these
didn't show up on your profile.  Did you try using a larger inode size
as suggested in my previous mail?

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Deleting files with extended attributes is dead slow

[Bernd Schubert]

On 08/17/2011 07:02 PM, Christoph Hellwig wrote:
> On Wed, Aug 17, 2011 at 03:05:28PM +0200, Bernd Schubert wrote:
>>> (squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
>>> /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
>>>         0: [0..7]: 92304..92311
>>
>> (Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to
>> tell me).
>
> It means that you are having an extent spanning 8 blocks for xattr
> storage, that map to physical blocks 92304 to 9231 in the filesystem.
>
> It sounds to me like your workload has a lot more than 256 bytes of
> xattrs, or the underlying code is doing something rather stupid.

Well, the workload I described here is a controlled bonnie test, so 
there cannot be more than 256 bytes (unless there is a bug in the code, 
will double check later on).

>
>> Looking at 'top' and 'iostat -x' outout, I noticed we are actually
>> not limited by io to disk, but CPU bound. If you should be
>> interested, I have attached 'perf record -g' and 'perf report -g'
>> outout, of the bonnie file create (create + fsetfattr() ) phase.
>
> It's mostly spending a lot of time on copying things into the CIL
> buffers, which is expected and intentional as that allows for additional
> parallelity.  I you'd switch the workload to multiple intances doing
> the create in parallel you should be able to scale to better numbers.

I just tried to bonnies in parallel and that didn't improve anything. 
FhGFS code has several threads anyway. But it would be good, if the 
underlying file system wouldn't take all the CPU time...

>
>>>      100:256:256/10 37026  91 +++++ +++ 43691  93 35960  92 +++++ +++ 40708  92
>>> Latency              4328us     765us    2960us     527us     440us    1075us
>>> 1.96,1.96,fslab2,1,1313594619,,,,,,,,,,,,,,100,256,256,,10,37026,91,+++++,+++,43691,93,35960,92,+++++,+++,40708,92,,,,,,,4328us,765us,2960us,527us,440us,1075us
>>
>>
>> xfs:
>> mkfs.xfs -f -i size=512 -i maxpct=90  -l lazy-count=1 -n size=64k /dev/sdd
>
> Do 64k dir blocks actually help you with the workload?  They also tend

Also just tested, with or without doesn't improve anything.

> to do a lot of useless memcpys in their current form, although these
> didn't show up on your profile.  Did you try using a larger inode size
> as suggested in my previous mail?

I just tried and now that I understand the xfs_bmap output, it is 
interesting to see, that an xattr size up to 128 byte does not need an 
extent + blocks, but 256 byte do have one extent and 8 blocks even with 
an inode size of 2K. xfs_info tells me that isize=2048 was accepted. I 
didn't test any sizes in between 128 and 256 byte yet.
Now while I can set the data/xattr size for the bonnie test to less than 
256 byte, that is not so easy with our real target FhGFS ;)


Btw, the create() + fsetfattr() reate with 128B xattr data is between 
13000 and 17000, so due to inlining much better than with 256 xattr.


Thanks,
Bernd

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Deleting files with extended attributes is dead slow

[Dave Chinner]

On Wed, Aug 17, 2011 at 07:39:01PM +0200, Bernd Schubert wrote:
> On 08/17/2011 07:02 PM, Christoph Hellwig wrote:
> >On Wed, Aug 17, 2011 at 03:05:28PM +0200, Bernd Schubert wrote:
> >>>(squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
> >>>/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
> >>>        0: [0..7]: 92304..92311
> >>
> >>(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to
> >>tell me).
> >
> >It means that you are having an extent spanning 8 blocks for xattr
> >storage, that map to physical blocks 92304 to 9231 in the filesystem.
> >
> >It sounds to me like your workload has a lot more than 256 bytes of
> >xattrs, or the underlying code is doing something rather stupid.
> 
> Well, the workload I described here is a controlled bonnie test, so
> there cannot be more than 256 bytes (unless there is a bug in the
> code, will double check later on).
> 
> >
> >>Looking at 'top' and 'iostat -x' outout, I noticed we are actually
> >>not limited by io to disk, but CPU bound. If you should be
> >>interested, I have attached 'perf record -g' and 'perf report -g'
> >>outout, of the bonnie file create (create + fsetfattr() ) phase.
> >
> >It's mostly spending a lot of time on copying things into the CIL
> >buffers, which is expected and intentional as that allows for additional
> >parallelity.  I you'd switch the workload to multiple intances doing
> >the create in parallel you should be able to scale to better numbers.
> 
> I just tried to bonnies in parallel and that didn't improve
> anything. FhGFS code has several threads anyway. But it would be
> good, if the underlying file system wouldn't take all the CPU
> time...

XFS directory algorithms are significantly more complex than ext4.
They trade off CPU usage for significantly better layout and
scalability at large sizes. i.e. CPU costs less than IO so we burn
more CPU to reduce IO. You don't see the benefits of that until
directories start to get large (e.g. > 100k entries) and you are
doing cold cache lookups.

> >>xfs:
> >>mkfs.xfs -f -i size=512 -i maxpct=90  -l lazy-count=1 -n size=64k /dev/sdd

What is the output of this command?

> >Do 64k dir blocks actually help you with the workload?  They also tend
> 
> Also just tested, with or without doesn't improve anything.

Right, 64k directory blocks make a difference on cold cache
traversals and lookups by flattening the btrees. They also make a
difference in create/unlink performance once you get over a few
million files in the one directory (once again due to reduced IO).

> >to do a lot of useless memcpys in their current form, although these
> >didn't show up on your profile.  Did you try using a larger inode size
> >as suggested in my previous mail?
> 
> I just tried and now that I understand the xfs_bmap output, it is
> interesting to see, that an xattr size up to 128 byte does not need
> an extent + blocks, but 256 byte do have one extent and 8 blocks
> even with an inode size of 2K. xfs_info tells me that isize=2048 was
> accepted. I didn't test any sizes in between 128 and 256 byte yet.
> Now while I can set the data/xattr size for the bonnie test to less
> than 256 byte, that is not so easy with our real target FhGFS ;)

That tells me your filesystem is either not using dynamic attribute
fork offsets or that code is broken. The output of the above mkfs
command will tell us what attribute fork behaviour is expected, so 
which of the two cases you are seeing.

Also, what kernel are you testing on?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Deleting files with extended attributes is dead slow

[Dave Chinner]

On Thu, Aug 18, 2011 at 12:08:48PM +1000, Dave Chinner wrote:
> On Wed, Aug 17, 2011 at 07:39:01PM +0200, Bernd Schubert wrote:
> > On 08/17/2011 07:02 PM, Christoph Hellwig wrote:
> > >On Wed, Aug 17, 2011 at 03:05:28PM +0200, Bernd Schubert wrote:
> > >>>(squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
> > >>>/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
> > >>>        0: [0..7]: 92304..92311
> > >>
> > >>(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to
> > >>tell me).
> > >
> > >It means that you are having an extent spanning 8 blocks for xattr
> > >storage, that map to physical blocks 92304 to 9231 in the filesystem.
> > >
> > >It sounds to me like your workload has a lot more than 256 bytes of
> > >xattrs, or the underlying code is doing something rather stupid.
> > 
> > Well, the workload I described here is a controlled bonnie test, so
> > there cannot be more than 256 bytes (unless there is a bug in the
> > code, will double check later on).
> > 
> > >
> > >>Looking at 'top' and 'iostat -x' outout, I noticed we are actually
> > >>not limited by io to disk, but CPU bound. If you should be
> > >>interested, I have attached 'perf record -g' and 'perf report -g'
> > >>outout, of the bonnie file create (create + fsetfattr() ) phase.
> > >
> > >It's mostly spending a lot of time on copying things into the CIL
> > >buffers, which is expected and intentional as that allows for additional
> > >parallelity.  I you'd switch the workload to multiple intances doing
> > >the create in parallel you should be able to scale to better numbers.
> > 
> > I just tried to bonnies in parallel and that didn't improve
> > anything. FhGFS code has several threads anyway. But it would be
> > good, if the underlying file system wouldn't take all the CPU
> > time...
> 
> XFS directory algorithms are significantly more complex than ext4.
> They trade off CPU usage for significantly better layout and
> scalability at large sizes. i.e. CPU costs less than IO so we burn
> more CPU to reduce IO. You don't see the benefits of that until
> directories start to get large (e.g. > 100k entries) and you are
> doing cold cache lookups.
> 
> > >>xfs:
> > >>mkfs.xfs -f -i size=512 -i maxpct=90  -l lazy-count=1 -n size=64k /dev/sdd
> 
> What is the output of this command?
> 
> > >Do 64k dir blocks actually help you with the workload?  They also tend
> > 
> > Also just tested, with or without doesn't improve anything.
> 
> Right, 64k directory blocks make a difference on cold cache
> traversals and lookups by flattening the btrees. They also make a
> difference in create/unlink performance once you get over a few
> million files in the one directory (once again due to reduced IO).
> 
> > >to do a lot of useless memcpys in their current form, although these
> > >didn't show up on your profile.  Did you try using a larger inode size
> > >as suggested in my previous mail?
> > 
> > I just tried and now that I understand the xfs_bmap output, it is
> > interesting to see, that an xattr size up to 128 byte does not need
> > an extent + blocks, but 256 byte do have one extent and 8 blocks
> > even with an inode size of 2K. xfs_info tells me that isize=2048 was
> > accepted. I didn't test any sizes in between 128 and 256 byte yet.
> > Now while I can set the data/xattr size for the bonnie test to less
> > than 256 byte, that is not so easy with our real target FhGFS ;)
> 
> That tells me your filesystem is either not using dynamic attribute
> fork offsets or that code is broken.

It is neither of these.

> The output of the above mkfs
> command will tell us what attribute fork behaviour is expected, so 
> which of the two cases you are seeing.
> 
> Also, what kernel are you testing on?

Ok, I've reproduced it on a 3.0-rc2 kernel with attr=2 and 2k
inodes. An attribute of 254 bytes stays in line:

....
u = (empty)
a.sfattr.hdr.totsize = 270
a.sfattr.hdr.count = 1
a.sfattr.list[0].namelen = 9
a.sfattr.list[0].valuelen = 254
a.sfattr.list[0].root = 0
a.sfattr.list[0].secure = 0
a.sfattr.list[0].name = "user.test"
a.sfattr.list[0].value = " ten chars ten chars ten chars ten chars
ten chars ten chars ten chars ten chars ten chars ten chars ten
chars ten chars ten chars ten chars ten chars ten chars ten chars
ten chars ten chars ten chars ten chars ten chars ten chars ten
chars ten chars ten"

but at 255 bytes:

u = (empty)
a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,40,1,0]

it goes out of line. There's something strange happening there....

/me looks

#define XFS_ATTR_SF_ENTSIZE_MAX                 /* max space for name&value */ \
        ((1 << (NBBY*(int)sizeof(__uint8_t))) - 1)

There's the issue - shortform attribute structures (i.e. inline
attributes) have only 8 bits for the value/name lengths, so the
attribute is going out of line at a value length of 255 bytes.

So, the reason you are seeing this is that in line attributes must
be 254 bytes or smaller to remain in line regardless of the inode
size. If FhGFS is using multiple attributes smaller than 254 bytes,
then they will all stay inline until all the inode attribute fork
space is used. However, the first one that goes over this limit will
push them all out of line.

Unfortunately, that limit is fixed into the on-disk format for
attributes, so is kind of hard to change. :/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Deleting files with extended attributes is dead slow

[Christoph Hellwig]

Btw, one interesting thing I noticed is that xfs_attr_leaf_create always
zeroes and logs whe whole attr block.  I'll need to audit this a bit
more, but from the highlevel POV I don't see why we can't simply just
logs the bits we actually added.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs