Re: ext3: massive latencies for some write operations

[Andreas Dilger <adilger@sun.com>]

On Jun 25, 2009  22:18 -0400, Nick Bowler wrote:
> I wasn't entirely sure which list to post this on, this one seemed to
> fit best.  Apologies if there was a better place.
> 
> Please CC any replies to me me as I am not subscribed to the list.

This is a relatively well-known problem with ext3.

> Recently, I've been experiencing enormous (more than a minute)
> latencies during some write operations to my /home filesystem.  During
> such a time, many programs tend to completely stop responding,
> although programs doing continuous reads (e.g. music or video players)
> appear to be unaffected.
> 
> The filesystem is a 3TB ext3 filesystem on a software raid-5 array (4x
> 1TB drives).  Dmesg is completely silent.
> 
> I'm on an amd64 running 2.6.30.  However, I had these issues on 2.6.26
> and 2.6.29 as well.  Latencytop output follows.
> 
>    LatencyTOP version 0.4       (C) 2008 Intel Corporation
> 
> Process amuled (19428)                     Total: 69182.1 msec
> start_this_handle journal_start ext3_journal_start68746.9 msec       99.4 %
> sync_page __lock_page find_lock_page filemap_fault 84.5 msec          0.3 %
> sync_page sync_page_killable __lock_page_killable  20.9 msec          0.0 %

You have some application that is doing a lot of "fsync" operations,
but because there is likely also a lot of other IO going on the
fsync operations are taking a long time to sync the journal.

If you use ext4 this problem should go away.  I would recommend the
FC11 2.6.29 kernel, since it has all of the latest ext4 fixes in it.

If you wanted to throw some hardware at the problem, adding an SSD device
for the journal should also solve the problem for ext3.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.