From: Dave Chinner <david@fromorbit.com>
To: Carlos Carvalho <carlos@fisica.ufpr.br>
Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 0/17] fs: Inode cache scalability
Date: Mon, 4 Oct 2010 18:22:13 +1100 [thread overview]
Message-ID: <20101004072213.GI4681@dastard> (raw)
In-Reply-To: <19623.48074.873182.970865@fisica.ufpr.br>
On Sat, Oct 02, 2010 at 08:10:02PM -0300, Carlos Carvalho wrote:
> We have serious problems with 34.6 in a machine with ~11TiB xfs, with
> a lot of simultaneous IO, particularly hundreds of rm and a sync
> afterwards. Maybe they're related to these issues.
>
> The machine is a file server (almost all via http/apache) and has
> several thousand connections all the time. It behaves quite well for
> at most 4 days; from then on kswapd's start appearing on the display
> of top consuming ever increasing percentages of cpu. This is no
> problem, the machine has 16 nearly idle cores. However, after about
> 5-7 days there's an abrupt transition: in about 30s the load goes to
> several thousand, apache shows up consuming all possible cpu and
> downloads nearly stop. I have to reboot the machine to get service
> back. It manages to unmount the filesystems and reboot properly.
>
> Stopping/restarting apache restores the situation but only for
> a short while; after about 2-3h the problem reappears. That's why I
> have to reboot.
>
> With 35.6 the behaviour seems to have changed: now often
> CONFIG_DETECT_HUNG_TASK produces this kind of call trace in the log:
>
> [<ffffffff81098578>] ? igrab+0x10/0x30
> [<ffffffff811160fe>] ? xfs_sync_inode_valid+0x4c/0x76
> [<ffffffff81116241>] ? xfs_sync_inode_data+0x1b/0xa8
> [<ffffffff811163e0>] ? xfs_inode_ag_walk+0x96/0xe4
> [<ffffffff811163dd>] ? xfs_inode_ag_walk+0x93/0xe4
> [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8
> [<ffffffff81116495>] ? xfs_inode_ag_iterator+0x67/0xc4
> [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8
> [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e
> [<ffffffff81116712>] ? xfs_sync_data+0x22/0x42
> [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e
> [<ffffffff8111678b>] ? xfs_quiesce_data+0x2b/0x94
> [<ffffffff81113f03>] ? xfs_fs_sync_fs+0x2d/0xd7
> [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e
> [<ffffffff810a48c4>] ? __sync_filesystem+0x62/0x7b
> [<ffffffff8108993e>] ? iterate_supers+0x60/0x9d
> [<ffffffff810a493a>] ? sys_sync+0x3f/0x53
> [<ffffffff81001dab>] ? system_call_fastpath+0x16/0x1b
>
> It doesn't seem to cause service disruption (at least the flux graphs
> don't show drops). I didn't see it happen while I was watching so it
> may be that service degrades for short intervals. Uptime with 35.6 is
> only 3d8h so it's still not sure that the breakdown of 34.6 is gone
> but kswapd's cpu usages are very small, less than with 34.6 for a
> similar uptime. There are only 2 filesystems, and the big one has 256
> AGs. They're not mounted with delaylog.
Apply this:
http://www.oss.sgi.com/archives/xfs/2010-10/msg00000.html
And in future, can you please report bugs in a new thread to the
appropriate lists (xfs@oss.sgi.com), not as a reply to a completely
unrelated development thread....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
prev parent reply other threads:[~2010-10-04 7:22 UTC|newest]
Thread overview: 120+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-09-29 12:18 [PATCH 0/17] fs: Inode cache scalability Dave Chinner
2010-09-29 12:18 ` [PATCH 01/17] kernel: add bl_list Dave Chinner
2010-09-30 4:52 ` Andrew Morton
2010-10-16 7:55 ` Nick Piggin
2010-10-16 16:28 ` Christoph Hellwig
2010-10-01 5:48 ` Christoph Hellwig
2010-09-29 12:18 ` [PATCH 02/17] fs: icache lock s_inodes list Dave Chinner
2010-10-01 5:49 ` Christoph Hellwig
2010-10-16 7:54 ` Nick Piggin
2010-10-16 16:12 ` Christoph Hellwig
2010-10-16 17:09 ` Nick Piggin
2010-10-17 0:42 ` Christoph Hellwig
2010-10-17 2:03 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 03/17] fs: icache lock inode hash Dave Chinner
2010-09-30 4:52 ` Andrew Morton
2010-09-30 6:13 ` Dave Chinner
2010-10-01 6:06 ` Christoph Hellwig
2010-10-16 7:57 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 04/17] fs: icache lock i_state Dave Chinner
2010-10-01 5:54 ` Christoph Hellwig
2010-10-16 7:54 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 05/17] fs: icache lock i_count Dave Chinner
2010-09-30 4:52 ` Andrew Morton
2010-10-01 5:55 ` Christoph Hellwig
2010-10-01 6:04 ` Andrew Morton
2010-10-01 6:16 ` Christoph Hellwig
2010-10-01 6:23 ` Andrew Morton
2010-09-29 12:18 ` [PATCH 06/17] fs: icache lock lru/writeback lists Dave Chinner
2010-09-30 4:52 ` Andrew Morton
2010-09-30 6:16 ` Dave Chinner
2010-10-16 7:55 ` Nick Piggin
2010-10-01 6:01 ` Christoph Hellwig
2010-10-05 22:30 ` Dave Chinner
2010-09-29 12:18 ` [PATCH 07/17] fs: icache atomic inodes_stat Dave Chinner
2010-09-30 4:52 ` Andrew Morton
2010-09-30 6:20 ` Dave Chinner
2010-09-30 6:37 ` Andrew Morton
2010-10-16 7:56 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 08/17] fs: icache protect inode state Dave Chinner
2010-10-01 6:02 ` Christoph Hellwig
2010-10-16 7:54 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 09/17] fs: Make last_ino, iunique independent of inode_lock Dave Chinner
2010-09-30 4:53 ` Andrew Morton
2010-10-01 6:08 ` Christoph Hellwig
2010-10-16 7:54 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 10/17] fs: icache remove inode_lock Dave Chinner
2010-09-29 12:18 ` [PATCH 11/17] fs: Factor inode hash operations into functions Dave Chinner
2010-10-01 6:06 ` Christoph Hellwig
2010-10-16 7:54 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 12/17] fs: Introduce per-bucket inode hash locks Dave Chinner
2010-09-30 1:52 ` Christoph Hellwig
2010-09-30 2:43 ` Dave Chinner
2010-10-16 7:55 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 13/17] fs: Implement lazy LRU updates for inodes Dave Chinner
2010-09-30 2:05 ` Christoph Hellwig
2010-10-16 7:54 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 14/17] fs: Inode counters do not need to be atomic Dave Chinner
2010-09-29 12:18 ` [PATCH 15/17] fs: inode per-cpu last_ino allocator Dave Chinner
2010-09-30 2:07 ` Christoph Hellwig
2010-10-06 6:29 ` Dave Chinner
2010-10-06 8:51 ` Christoph Hellwig
2010-09-30 4:53 ` Andrew Morton
2010-09-30 5:36 ` Eric Dumazet
2010-09-30 7:53 ` Eric Dumazet
2010-09-30 7:53 ` Eric Dumazet
2010-09-30 8:14 ` Andrew Morton
2010-09-30 10:22 ` [PATCH] " Eric Dumazet
2010-09-30 16:45 ` Andrew Morton
2010-09-30 17:28 ` Eric Dumazet
2010-09-30 17:28 ` Eric Dumazet
2010-09-30 17:39 ` Andrew Morton
2010-09-30 18:05 ` Eric Dumazet
2010-10-01 6:12 ` Christoph Hellwig
2010-10-01 6:45 ` Eric Dumazet
2010-10-01 6:45 ` Eric Dumazet
2010-10-16 6:36 ` Nick Piggin
2010-10-16 6:40 ` Nick Piggin
2010-09-29 12:18 ` [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter Dave Chinner
2010-09-30 2:12 ` Christoph Hellwig
2010-09-30 4:53 ` Andrew Morton
2010-09-30 6:10 ` Dave Chinner
2010-10-16 7:55 ` Nick Piggin
2010-10-16 8:29 ` Eric Dumazet
2010-10-16 8:29 ` Eric Dumazet
2010-10-16 9:07 ` Andrew Morton
2010-10-16 9:31 ` Eric Dumazet
2010-10-16 9:31 ` Eric Dumazet
2010-10-16 14:19 ` [PATCH] percpu_counter : add percpu_counter_add_fast() Eric Dumazet
2010-10-18 15:24 ` Christoph Lameter
2010-10-18 15:39 ` Eric Dumazet
2010-10-18 15:39 ` Eric Dumazet
2010-10-18 16:12 ` Christoph Lameter
2010-10-21 22:37 ` Andrew Morton
2010-10-21 23:10 ` Christoph Lameter
2010-10-22 0:45 ` Andrew Morton
2010-10-22 1:55 ` Andrew Morton
2010-10-22 1:55 ` Andrew Morton
2010-10-22 1:58 ` Nick Piggin
2010-10-22 2:14 ` Andrew Morton
2010-10-22 4:12 ` Eric Dumazet
2010-10-22 4:12 ` Eric Dumazet
2010-10-21 22:43 ` Andrew Morton
2010-10-21 22:58 ` Eric Dumazet
2010-10-21 23:18 ` Andrew Morton
2010-10-21 23:22 ` Eric Dumazet
2010-10-21 23:22 ` Eric Dumazet
2010-10-21 22:31 ` [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter Andrew Morton
2010-10-21 22:58 ` Eric Dumazet
2010-10-02 16:02 ` Christoph Hellwig
2010-09-29 12:18 ` [PATCH 17/17] fs: Clean up inode reference counting Dave Chinner
2010-09-30 2:15 ` Christoph Hellwig
2010-10-16 7:55 ` Nick Piggin
2010-10-16 16:14 ` Christoph Hellwig
2010-10-16 17:09 ` Nick Piggin
2010-09-30 4:53 ` Andrew Morton
2010-09-29 23:57 ` [PATCH 0/17] fs: Inode cache scalability Christoph Hellwig
2010-09-30 0:24 ` Dave Chinner
2010-09-30 2:21 ` Christoph Hellwig
2010-10-02 23:10 ` Carlos Carvalho
2010-10-04 7:22 ` Dave Chinner [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20101004072213.GI4681@dastard \
--to=david@fromorbit.com \
--cc=carlos@fisica.ufpr.br \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.