* Re: Slow delete [not found] <AANLkTinPmhJRD3CDdsHtkLFzYd2jF9ee7gPqgO6XBSfl@mail.gmail.com> @ 2010-07-05 15:13 ` Emmanuel Florac 2010-07-05 15:33 ` Andrei Deftu 2010-07-05 18:54 ` Christoph Hellwig 2010-07-05 23:34 ` Dave Chinner 1 sibling, 2 replies; 10+ messages in thread From: Emmanuel Florac @ 2010-07-05 15:13 UTC (permalink / raw) To: Andrei Deftu; +Cc: xfs Le Mon, 5 Jul 2010 17:27:44 +0300 Andrei Deftu <andreideftu@gmail.com> écrivait: > As many people have pointed out, xfs is very slow when deleting a > large number of files. Does anyone has a technical answer to this > problem? Also, I am interested to know if there is a way to > temporarily speed up the delete process, regardless of all the other > parameters. Using the "nobarrier" mount option (when applicable/advisable) considerably speeds up metadata operations. Another possibility is to use a ramdisk (or /dev/null) as a log device though it could be... dangerous :) -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-05 15:13 ` Slow delete Emmanuel Florac @ 2010-07-05 15:33 ` Andrei Deftu 2010-07-05 18:21 ` Peter Grandi 2010-07-05 18:54 ` Christoph Hellwig 1 sibling, 1 reply; 10+ messages in thread From: Andrei Deftu @ 2010-07-05 15:33 UTC (permalink / raw) To: Emmanuel Florac; +Cc: xfs On Mon, Jul 5, 2010 at 6:13 PM, Emmanuel Florac <eflorac@intellique.com> wrote: > Le Mon, 5 Jul 2010 17:27:44 +0300 > Andrei Deftu <andreideftu@gmail.com> écrivait: > >> As many people have pointed out, xfs is very slow when deleting a >> large number of files. Does anyone has a technical answer to this >> problem? Also, I am interested to know if there is a way to >> temporarily speed up the delete process, regardless of all the other >> parameters. > > Using the "nobarrier" mount option (when applicable/advisable) > considerably speeds up metadata operations. > Another possibility is to use a ramdisk (or /dev/null) as a log device > though it could be... dangerous :) Thanks. Yes, I know about these tricks like 'nobarrier' option, separate partition for metadata log, increased size for the log, more log buffers or atime disabled. But I was interested more in what is the real cause of this slowness. Andrei _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-05 15:33 ` Andrei Deftu @ 2010-07-05 18:21 ` Peter Grandi 0 siblings, 0 replies; 10+ messages in thread From: Peter Grandi @ 2010-07-05 18:21 UTC (permalink / raw) To: Andrei Deftu; +Cc: xfs >>> [ ... ] xfs is very slow when deleting a large number of >>> files. [ ... ] Not quite -- it is nearly as fast as possible. > Thanks. Yes, I know about these tricks like 'nobarrier' > option, separate partition for metadata log, increased size > for the log, more log buffers or atime disabled. But I was > interested more in what is the real cause of this slowness. Tricks like those indicate quite clearly the real cause. To make it even more explicit, the real cause is that those who complain tha XFS is slow at deleting lots of files have a storage subsystem can only do a limited number of committed transactions per second and XFS does not "cheat". _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-05 15:13 ` Slow delete Emmanuel Florac 2010-07-05 15:33 ` Andrei Deftu @ 2010-07-05 18:54 ` Christoph Hellwig 1 sibling, 0 replies; 10+ messages in thread From: Christoph Hellwig @ 2010-07-05 18:54 UTC (permalink / raw) To: Emmanuel Florac; +Cc: Andrei Deftu, xfs Using nobarrier really is a bad idea in general as it does not actually provides any transactional guarantee. But if you're the kind of person that would use it anyway please upgrade to a 2.6.35-rc kernel and use the "delaylog" mount option, which will uses a new logging mechanism that will get you much better delete performance. It's still experimental, but it's defintively not any worse than using nobarrier. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete [not found] <AANLkTinPmhJRD3CDdsHtkLFzYd2jF9ee7gPqgO6XBSfl@mail.gmail.com> 2010-07-05 15:13 ` Slow delete Emmanuel Florac @ 2010-07-05 23:34 ` Dave Chinner 2010-07-12 12:17 ` Michael Monnerie 1 sibling, 1 reply; 10+ messages in thread From: Dave Chinner @ 2010-07-05 23:34 UTC (permalink / raw) To: Andrei Deftu; +Cc: xfs On Mon, Jul 05, 2010 at 05:27:44PM +0300, Andrei Deftu wrote: > Hi, > > As many people have pointed out, xfs is very slow when deleting a > large number of files. Does anyone has a technical answer to this > problem? Also, I am interested to know if there is a way to > temporarily speed up the delete process, regardless of all the other > parameters. The technical reason is to do with the architecture of the transaction and logging subsystem of XFS. The key concept: relogging of dirty objects. The reason it affects unlinks: removing files from the same directory modifies the same objects repeatedly (e.g. the directory btree blocks). For a detailed explanation of the concept and how delayed logging takes advantage of it to speed up stuff like deleting a large number of files, see: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs-delayed-logging-design.txt Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-05 23:34 ` Dave Chinner @ 2010-07-12 12:17 ` Michael Monnerie 2010-07-19 18:54 ` Alex Elder 0 siblings, 1 reply; 10+ messages in thread From: Michael Monnerie @ 2010-07-12 12:17 UTC (permalink / raw) To: xfs; +Cc: lwn [-- Attachment #1.1: Type: Text/Plain, Size: 1055 bytes --] On Dienstag, 6. Juli 2010 Dave Chinner wrote: > For a detailed explanation of the concept and how delayed logging > takes advantage of it to speed up stuff like deleting a large number > of files, see: > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=bl > ob;f=Documentation/filesystems/xfs-delayed-logging-design.txt Great docu, although my head still exploded on reading this, and at some point I started to just read over it. Luckily, I'm no dev, otherwise I'd need to grow a second head for the extra brain space needed to understand all that stuff, and that would make my appearance even worse than it is already. Also, I'm not sure if the two-head solution isn't patented by Zaphod Beeblebrox already, at least in the U.S. region. -- mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/ [-- Attachment #1.2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 198 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-12 12:17 ` Michael Monnerie @ 2010-07-19 18:54 ` Alex Elder 2010-07-20 11:04 ` Michael Monnerie 0 siblings, 1 reply; 10+ messages in thread From: Alex Elder @ 2010-07-19 18:54 UTC (permalink / raw) To: Michael Monnerie; +Cc: XFS Mailing List On Mon, 2010-07-12 at 14:17 +0200, Michael Monnerie wrote: > On Dienstag, 6. Juli 2010 Dave Chinner wrote: > > For a detailed explanation of the concept and how delayed logging > > takes advantage of it to speed up stuff like deleting a large number > > of files, see: > > > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=bl > > ob;f=Documentation/filesystems/xfs-delayed-logging-design.txt > > Great docu, although my head still exploded on reading this, and at some > point I started to just read over it. Luckily, I'm no dev, otherwise I'd > need to grow a second head for the extra brain space needed to > understand all that stuff, and that would make my appearance even worse > than it is already. Also, I'm not sure if the two-head solution isn't > patented by Zaphod Beeblebrox already, at least in the U.S. region. Assuming your single head is still in good enough shape to understand this, here's a high-level (though imprecise) explanation. Like many operations in XFS, deleting an entry from a directory requires the creation of entries in the XFS log. These entries record the net effect of the delete operation, allowing the delete operation to be reconstructed (more or less) in the event of a crash. A transaction in the log records information about critical data structures and how they will change as the result of an operation. Log entries get saved to disk in such a way that following a crash, the file system can be put into a consistent state quickly by "replaying" entries recently recorded to the log. If you perform a number of operations on the same object (such as removing multiple files from the same directory), XFS currently sends copies of some of the affected data structures to the log repeatedly--once for each operation in sequence. So for example if you delete lots of files from the same directory, there is a consequently large volume of log activity that needs to be performed, often describing updates to the same data structure. Such a structure will (currently) get written to disk many many times as a result. Dave recently added support for "delayed logging" to XFS. It's currently an experimental feature, enabled via a mount option. When delayed logging is active, some log activity is held off, allowing numerous updates to the same data structure to be aggregated. A logged data structure can then be committed to disk much less frequently, reducing the amount of data that needs to be written to disk while preserving the ability to recover the file system following a crash. This optimization allows certain operations (like mass deletions) to be completed much more efficiently, and in the case of large directories the performance improvement that results can be significant. In any crash situation, there will be a certain amount of file system activity underway at the time of the crash that will be lost; logging file system operations doesn't eliminate that possibility. The down side of delayed logging is this type of loss is likely to be greater than it would be with the more conventional XFS logging mechanism. This is a tradeoff between file system performance in normal operation and the amount that can be recovered after a crash. -Alex _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-19 18:54 ` Alex Elder @ 2010-07-20 11:04 ` Michael Monnerie 2010-07-20 17:09 ` Stan Hoeppner 2010-07-20 23:18 ` Dave Chinner 0 siblings, 2 replies; 10+ messages in thread From: Michael Monnerie @ 2010-07-20 11:04 UTC (permalink / raw) To: xfs, aelder [-- Attachment #1.1: Type: Text/Plain, Size: 1305 bytes --] On Montag, 19. Juli 2010 Alex Elder wrote: > Assuming your single head is still in good enough shape > to understand this, here's a high-level (though imprecise) > explanation. Thank you Alex for the summary, but that was the part I understood. I didn't understand the in-depth explanation later, where XFS internals were described. But I guess it's not something I must understand. The short summary is: XFS will be faster, but really shouldn't crash when using delayed logging. It will be interesting to see what happens on a busy system during a crash. Deleted files will appear again, some file changes won't be committed, and again the big problem: some config files might be gone as the rename/recreate transaction will not be done safe via fsync, and people will cry. The plus in performance might be worth it. Can someone guess the exact problems that can happen with delayed transaction on a crash? -- mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Aktuelles Radiointerview! ****** http://www.it-podcast.at/aktuelle-sendung.html // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/ [-- Attachment #1.2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 198 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-20 11:04 ` Michael Monnerie @ 2010-07-20 17:09 ` Stan Hoeppner 2010-07-20 23:18 ` Dave Chinner 1 sibling, 0 replies; 10+ messages in thread From: Stan Hoeppner @ 2010-07-20 17:09 UTC (permalink / raw) To: xfs Michael Monnerie put forth on 7/20/2010 6:04 AM: > Can someone guess the exact problems that can happen with delayed > transaction on a crash? The only real crash scenario difference, as I understand it, is that with delayed logging enabled you'll potentially lose more data due to crash as more is held memory resident with delayed logging enabled. The characteristics of the data loss are the same with and without this enabled, it's just potentially more severe with delayed logging enabled. If your systems are prone to power loss, hardware failures, or kernel panics, I'd not enable delayed logging. If your systems are rock solid, and you can benefit from the minor decrease in file fragmentation and the increase in mass file delete speed, then enable it. -- Stan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Slow delete 2010-07-20 11:04 ` Michael Monnerie 2010-07-20 17:09 ` Stan Hoeppner @ 2010-07-20 23:18 ` Dave Chinner 1 sibling, 0 replies; 10+ messages in thread From: Dave Chinner @ 2010-07-20 23:18 UTC (permalink / raw) To: Michael Monnerie; +Cc: aelder, xfs On Tue, Jul 20, 2010 at 01:04:28PM +0200, Michael Monnerie wrote: > Can someone guess the exact problems that can happen with delayed > transaction on a crash? There should be no difference in behaviour, just that more operations will appear not to have been executed after crash recovery. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-07-20 23:15 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <AANLkTinPmhJRD3CDdsHtkLFzYd2jF9ee7gPqgO6XBSfl@mail.gmail.com>
2010-07-05 15:13 ` Slow delete Emmanuel Florac
2010-07-05 15:33 ` Andrei Deftu
2010-07-05 18:21 ` Peter Grandi
2010-07-05 18:54 ` Christoph Hellwig
2010-07-05 23:34 ` Dave Chinner
2010-07-12 12:17 ` Michael Monnerie
2010-07-19 18:54 ` Alex Elder
2010-07-20 11:04 ` Michael Monnerie
2010-07-20 17:09 ` Stan Hoeppner
2010-07-20 23:18 ` Dave Chinner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox