From: "J. Bruce Fields" <bfields@fieldses.org>
To: Mel Gorman <mgorman@suse.de>
Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, bmarson@redhat.com
Subject: spinning in isolate_migratepages_range on busy nfs server
Date: Thu, 25 Oct 2012 12:47:22 -0400 [thread overview]
Message-ID: <20121025164722.GE6846@fieldses.org> (raw)
We're seeing an nfs server on a 3.6-ish kernel lock up after running
specfs for a while.
Looking at the logs, there are some hung task warnings showing nfsd
threads stuck on directory i_mutexes trying to do lookups.
A sysrq-t dump showed there were also lots of threads holding those
i_mutexes while trying to allocate xfs inodes:
nfsd R running task 0 6517 2 0x00000080
ffff880f925074c0 0000000000000046 ffff880fe4718000 ffff880f92507fd8
ffff880f92507fd8 ffff880f92507fd8 ffff880fd7920000 ffff880fe4718000
0000000000000000 ffff880f92506000 ffff88102ffd96c0 ffff88102ffd9b40
Call Trace:
[<ffffffff81091aaa>] __cond_resched+0x2a/0x40
[<ffffffff815d3750>] _cond_resched+0x30/0x40
[<ffffffff81150e92>] isolate_migratepages_range+0xb2/0x550
[<ffffffff811507c0>] ? compact_checklock_irqsave.isra.17+0xe0/0xe0
[<ffffffff81151536>] compact_zone+0x146/0x3f0
[<ffffffff81151a92>] compact_zone_order+0x82/0xc0
[<ffffffff81151bb1>] try_to_compact_pages+0xe1/0x110
[<ffffffff815c99e2>] __alloc_pages_direct_compact+0xaa/0x190
[<ffffffff81138317>] __alloc_pages_nodemask+0x517/0x980
[<ffffffff81088a00>] ? __synchronize_srcu+0xf0/0x110
[<ffffffff81171e30>] alloc_pages_current+0xb0/0x120
[<ffffffff8117b015>] new_slab+0x265/0x310
[<ffffffff815caefc>] __slab_alloc+0x358/0x525
[<ffffffffa05625a7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[<ffffffff81088c72>] ? up+0x32/0x50
[<ffffffffa05625a7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[<ffffffff8117b4ef>] kmem_cache_alloc+0xff/0x130
[<ffffffffa05625a7>] kmem_zone_alloc+0x67/0xf0 [xfs]
[<ffffffffa0552f49>] xfs_inode_alloc+0x29/0x270 [xfs]
[<ffffffffa0553801>] xfs_iget+0x231/0x6c0 [xfs]
[<ffffffffa0560687>] xfs_lookup+0xe7/0x110 [xfs]
[<ffffffffa05583e1>] xfs_vn_lookup+0x51/0x90 [xfs]
[<ffffffff81193e9d>] lookup_real+0x1d/0x60
[<ffffffff811940b8>] __lookup_hash+0x38/0x50
[<ffffffff81197e26>] lookup_one_len+0xd6/0x110
[<ffffffffa034667b>] nfsd_lookup_dentry+0x12b/0x4a0 [nfsd]
[<ffffffffa0346a69>] nfsd_lookup+0x79/0x140 [nfsd]
[<ffffffffa034fb5f>] nfsd3_proc_lookup+0xef/0x1c0 [nfsd]
[<ffffffffa0341bbb>] nfsd_dispatch+0xeb/0x230 [nfsd]
[<ffffffffa02ee3a8>] svc_process_common+0x328/0x6d0 [sunrpc]
[<ffffffffa02eeaa2>] svc_process+0x102/0x150 [sunrpc]
[<ffffffffa0341115>] nfsd+0xb5/0x1a0 [nfsd]
[<ffffffffa0341060>] ? nfsd_get_default_max_blksize+0x60/0x60 [nfsd]
[<ffffffff81082613>] kthread+0x93/0xa0
[<ffffffff815ddc34>] kernel_thread_helper+0x4/0x10
[<ffffffff81082580>] ? kthread_freezable_should_stop+0x70/0x70
[<ffffffff815ddc30>] ? gs_change+0x13/0x13
And perf --call-graph also shows we're spending all our time in the same
place, spinning on a lock (zone->lru_lock, I assume):
- 92.65% nfsd [kernel.kallsyms] [k] _raw_spin_lock_irqsave
- _raw_spin_lock_irqsave
- 99.86% isolate_migratepages_range
Just grepping through logs, I ran across 2a1402aa04 "mm: compaction:
acquire the zone->lru_lock as late as possible", in v3.7-rc1, which
looks relevant:
Richard Davies and Shaohua Li have both reported lock contention
problems in compaction on the zone and LRU locks as well as
significant amounts of time being spent in compaction. This
series aims to reduce lock contention and scanning rates to
reduce that CPU usage. Richard reported at
https://lkml.org/lkml/2012/9/21/91 that this series made a big
different to a problem he reported in August:
http://marc.info/?l=kvm&m=134511507015614&w=2
So we're trying that. Is there anything else we should try?
--b.
next reply other threads:[~2012-10-25 16:47 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-25 16:47 J. Bruce Fields [this message]
2012-10-26 18:48 ` spinning in isolate_migratepages_range on busy nfs server J. Bruce Fields
2012-10-30 13:56 ` Mel Gorman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121025164722.GE6846@fieldses.org \
--to=bfields@fieldses.org \
--cc=bmarson@redhat.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).