Re: Slow delete

Re: Slow delete

[Emmanuel Florac]

Le Mon, 5 Jul 2010 17:27:44 +0300
Andrei Deftu <andreideftu@gmail.com> écrivait:

> As many people have pointed out, xfs is very slow when deleting a
> large number of files. Does anyone has a technical answer to this
> problem? Also, I am interested to know if there is a way to
> temporarily speed up the delete process, regardless of all the other
> parameters.

Using the "nobarrier" mount option (when applicable/advisable)
considerably speeds up metadata operations.
Another possibility is to use a ramdisk (or /dev/null) as a log device
though it could be... dangerous :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Andrei Deftu]

On Mon, Jul 5, 2010 at 6:13 PM, Emmanuel Florac <eflorac@intellique.com> wrote:
> Le Mon, 5 Jul 2010 17:27:44 +0300
> Andrei Deftu <andreideftu@gmail.com> écrivait:
>
>> As many people have pointed out, xfs is very slow when deleting a
>> large number of files. Does anyone has a technical answer to this
>> problem? Also, I am interested to know if there is a way to
>> temporarily speed up the delete process, regardless of all the other
>> parameters.
>
> Using the "nobarrier" mount option (when applicable/advisable)
> considerably speeds up metadata operations.
> Another possibility is to use a ramdisk (or /dev/null) as a log device
> though it could be... dangerous :)

Thanks. Yes, I know about these tricks like 'nobarrier' option,
separate partition for metadata log, increased size for the log, more
log buffers or atime disabled. But I was interested more in what is
the real cause of this slowness.

Andrei

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Peter Grandi]

>>> [ ... ] xfs is very slow when deleting a large number of
>>> files. [ ... ]

Not quite -- it is nearly as fast as possible.

> Thanks. Yes, I know about these tricks like 'nobarrier'
> option, separate partition for metadata log, increased size
> for the log, more log buffers or atime disabled. But I was
> interested more in what is the real cause of this slowness.

Tricks like those indicate quite clearly the real cause.

To make it even more explicit, the real cause is that those who
complain tha XFS is slow at deleting lots of files have a storage
subsystem can only do a limited number of committed transactions
per second and XFS does not "cheat".

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Christoph Hellwig]

Using nobarrier really is a bad idea in general as it does not
actually provides any transactional guarantee.  But if you're the kind
of person that would use it anyway please upgrade to a 2.6.35-rc kernel
and use the "delaylog" mount option, which will uses a new logging
mechanism that will get you much better delete performance.  It's still
experimental, but it's defintively not any worse than using nobarrier.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Dave Chinner]

On Mon, Jul 05, 2010 at 05:27:44PM +0300, Andrei Deftu wrote:
> Hi,
> 
> As many people have pointed out, xfs is very slow when deleting a
> large number of files. Does anyone has a technical answer to this
> problem? Also, I am interested to know if there is a way to
> temporarily speed up the delete process, regardless of all the other
> parameters.

The technical reason is to do with the architecture of the
transaction and logging subsystem of XFS. The key concept: relogging
of dirty objects. The reason it affects unlinks: removing files
from the same directory modifies the same objects repeatedly (e.g.
the directory btree blocks).

For a detailed explanation of the concept and how delayed logging
takes advantage of it to speed up stuff like deleting a large number
of files, see:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs-delayed-logging-design.txt

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1055 bytes --]

On Dienstag, 6. Juli 2010 Dave Chinner wrote:
> For a detailed explanation of the concept and how delayed logging
> takes advantage of it to speed up stuff like deleting a large number
> of files, see:
> 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=bl
> ob;f=Documentation/filesystems/xfs-delayed-logging-design.txt
 
Great docu, although my head still exploded on reading this, and at some 
point I started to just read over it. Luckily, I'm no dev, otherwise I'd 
need to grow a second head for the extra brain space needed to 
understand all that stuff, and that would make my appearance even worse 
than it is already. Also, I'm not sure if the two-head solution isn't 
patented by Zaphod Beeblebrox already, at least in the U.S. region.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Alex Elder]

On Mon, 2010-07-12 at 14:17 +0200, Michael Monnerie wrote:
> On Dienstag, 6. Juli 2010 Dave Chinner wrote:
> > For a detailed explanation of the concept and how delayed logging
> > takes advantage of it to speed up stuff like deleting a large number
> > of files, see:
> > 
> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=bl
> > ob;f=Documentation/filesystems/xfs-delayed-logging-design.txt
>  
> Great docu, although my head still exploded on reading this, and at some 
> point I started to just read over it. Luckily, I'm no dev, otherwise I'd 
> need to grow a second head for the extra brain space needed to 
> understand all that stuff, and that would make my appearance even worse 
> than it is already. Also, I'm not sure if the two-head solution isn't 
> patented by Zaphod Beeblebrox already, at least in the U.S. region.

Assuming your single head is still in good enough shape
to understand this, here's a high-level (though imprecise)
explanation.

Like many operations in XFS, deleting an entry from a
directory requires the creation of entries in the XFS
log.  These entries record the net effect of the delete
operation, allowing the delete operation to be reconstructed
(more or less) in the event of a crash.

A transaction in the log records information about
critical data structures and how they will change as
the result of an operation.  Log entries get saved to
disk in such a way that following a crash, the file
system can be put into a consistent state quickly by
"replaying" entries recently recorded to the log.

If you perform a number of operations on the same object
(such as removing multiple files from the same directory),
XFS currently sends copies of some of the affected data
structures to the log repeatedly--once for each operation
in sequence.

So for example if you delete lots of files from the same
directory, there is a consequently large volume of log
activity that needs to be performed, often describing
updates to the same data structure.  Such a structure
will (currently) get written to disk many many times
as a result.

Dave recently added support for "delayed logging" to
XFS.  It's currently an experimental feature, enabled
via a mount option.  When delayed logging is active,
some log activity is held off, allowing numerous
updates to the same data structure to be aggregated.
A logged data structure can then be committed to disk
much less frequently, reducing the amount of data that
needs to be written to disk while preserving the ability
to recover the file system following a crash.

This optimization allows certain operations (like
mass deletions) to be completed much more efficiently,
and in the case of large directories the performance
improvement that results can be significant.

In any crash situation, there will be a certain amount
of file system activity underway at the time of the
crash that will be lost; logging file system operations
doesn't eliminate that possibility.  The down side of
delayed logging is this type of loss is likely to be
greater than it would be with the more conventional
XFS logging mechanism.  This is a tradeoff between
file system performance in normal operation and the
amount that can be recovered after a crash.

					-Alex


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1305 bytes --]

On Montag, 19. Juli 2010 Alex Elder wrote:
> Assuming your single head is still in good enough shape
> to understand this, here's a high-level (though imprecise)
> explanation.
 
Thank you Alex for the summary, but that was the part I understood. I 
didn't understand the in-depth explanation later, where XFS internals 
were described. But I guess it's not something I must understand. The 
short summary is: XFS will be faster, but really shouldn't crash when 
using delayed logging.

It will be interesting to see what happens on a busy system during a 
crash. Deleted files will appear again, some file changes won't be 
committed, and again the big problem: some config files might be gone as 
the rename/recreate transaction will not be done safe via fsync, and 
people will cry. The plus in performance might be worth it.

Can someone guess the exact problems that can happen with delayed 
transaction on a crash?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Stan Hoeppner]

Michael Monnerie put forth on 7/20/2010 6:04 AM:

> Can someone guess the exact problems that can happen with delayed 
> transaction on a crash?

The only real crash scenario difference, as I understand it, is that with
delayed logging enabled you'll potentially lose more data due to crash as more
is held memory resident with delayed logging enabled.  The characteristics of
the data loss are the same with and without this enabled, it's just
potentially more severe with delayed logging enabled.

If your systems are prone to power loss, hardware failures, or kernel panics,
I'd not enable delayed logging.  If your systems are rock solid, and you can
benefit from the minor decrease in file fragmentation and the increase in mass
file delete speed, then enable it.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Slow delete

[Dave Chinner]

On Tue, Jul 20, 2010 at 01:04:28PM +0200, Michael Monnerie wrote:
> Can someone guess the exact problems that can happen with delayed 
> transaction on a crash?

There should be no difference in behaviour, just that more
operations will appear not to have been executed after crash
recovery.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs